Oct 30 06:53:56.862308 ip-10-0-128-125 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 30 06:53:56.862319 ip-10-0-128-125 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 30 06:53:56.862326 ip-10-0-128-125 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 30 06:53:56.862551 ip-10-0-128-125 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 30 06:54:06.889418 ip-10-0-128-125 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 30 06:54:06.889435 ip-10-0-128-125 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 2b64aa7f6a0d404d87d7a2a59206956f --
Oct 30 06:56:09.714065 ip-10-0-128-125 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 06:56:10.083060 ip-10-0-128-125 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 06:56:10.083060 ip-10-0-128-125 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 06:56:10.083060 ip-10-0-128-125 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 06:56:10.083060 ip-10-0-128-125 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 06:56:10.086182 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086103    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 06:56:10.086182 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086181    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 06:56:10.086182 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086186    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 06:56:10.086276 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086190    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 06:56:10.086276 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086193    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 06:56:10.086276 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086196    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 06:56:10.086276 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086199    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 06:56:10.086276 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086202    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 06:56:10.086276 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086205    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 06:56:10.086276 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086208    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 06:56:10.086276 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086211    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 06:56:10.086276 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086213    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 06:56:10.086276 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086216    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 06:56:10.086276 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086219    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 06:56:10.086276 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086221    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 06:56:10.086276 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086224    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 06:56:10.086276 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086227    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 06:56:10.086276 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086230    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 06:56:10.086276 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086232    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 06:56:10.086276 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086235    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 06:56:10.086276 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086238    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 06:56:10.086276 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086240    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 06:56:10.086276 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086243    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 06:56:10.086740 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086245    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 06:56:10.086740 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086248    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 06:56:10.086740 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086250    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 06:56:10.086740 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086253    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 06:56:10.086740 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086256    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 06:56:10.086740 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086266    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 06:56:10.086740 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086270    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 06:56:10.086740 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086272    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 06:56:10.086740 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086275    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 06:56:10.086740 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086278    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 06:56:10.086740 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086281    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 06:56:10.086740 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086284    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 06:56:10.086740 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086287    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 06:56:10.086740 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086290    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 06:56:10.086740 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086292    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 06:56:10.086740 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086295    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 06:56:10.086740 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086298    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 06:56:10.086740 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086300    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 06:56:10.086740 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086303    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 06:56:10.087210 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086306    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 06:56:10.087210 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086308    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 06:56:10.087210 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086311    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 06:56:10.087210 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086313    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 06:56:10.087210 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086316    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 06:56:10.087210 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086319    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 06:56:10.087210 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086327    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 06:56:10.087210 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086330    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 06:56:10.087210 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086332    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 06:56:10.087210 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086335    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 06:56:10.087210 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086338    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 06:56:10.087210 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086340    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 06:56:10.087210 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086343    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 06:56:10.087210 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086346    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 06:56:10.087210 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086349    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 06:56:10.087210 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086352    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 06:56:10.087210 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086355    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 06:56:10.087210 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086358    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 06:56:10.087210 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086361    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 06:56:10.087655 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086364    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 06:56:10.087655 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086366    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 06:56:10.087655 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086369    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 06:56:10.087655 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086372    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 06:56:10.087655 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086374    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 06:56:10.087655 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086377    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 06:56:10.087655 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086380    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 06:56:10.087655 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086382    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 06:56:10.087655 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086385    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 06:56:10.087655 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086387    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 06:56:10.087655 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086390    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 06:56:10.087655 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086393    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 06:56:10.087655 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086395    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 06:56:10.087655 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086397    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 06:56:10.087655 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086400    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 06:56:10.087655 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086404    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 06:56:10.087655 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086409    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 06:56:10.087655 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086413    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 06:56:10.087655 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086416    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 06:56:10.087655 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086418    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 06:56:10.088138 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086421    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 06:56:10.088138 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086423    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 06:56:10.088138 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086426    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 06:56:10.088138 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086429    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 06:56:10.088138 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086431    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 06:56:10.088138 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086434    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 06:56:10.088138 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086439    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 06:56:10.088138 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086443    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 06:56:10.088138 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086446    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 06:56:10.088138 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086449    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 06:56:10.088138 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086452    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 06:56:10.088138 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086455    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 06:56:10.088138 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086457    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 06:56:10.088138 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086460    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 06:56:10.088138 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086463    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 06:56:10.088138 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086465    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 06:56:10.088138 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086468    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 06:56:10.088138 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086470    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 06:56:10.088138 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086473    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 06:56:10.088619 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086476    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 06:56:10.088619 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086478    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 06:56:10.088619 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086481    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 06:56:10.088619 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086483    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 06:56:10.088619 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086486    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 06:56:10.088619 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.086489    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 06:56:10.088619 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087268    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 06:56:10.088619 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087276    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 06:56:10.088619 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087279    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 06:56:10.088619 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087282    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 06:56:10.088619 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087285    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 06:56:10.088619 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087289    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 06:56:10.088619 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087293    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 06:56:10.088619 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087297    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 06:56:10.088619 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087300    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 06:56:10.088619 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087303    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 06:56:10.088619 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087306    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 06:56:10.088619 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087309    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 06:56:10.088619 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087311    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 06:56:10.089112 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087315    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 06:56:10.089112 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087318    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 06:56:10.089112 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087321    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 06:56:10.089112 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087324    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 06:56:10.089112 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087326    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 06:56:10.089112 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087329    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 06:56:10.089112 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087332    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 06:56:10.089112 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087334    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 06:56:10.089112 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087337    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 06:56:10.089112 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087340    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 06:56:10.089112 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087343    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 06:56:10.089112 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087345    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 06:56:10.089112 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087348    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 06:56:10.089112 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087351    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 06:56:10.089112 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087353    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 06:56:10.089112 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087356    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 06:56:10.089112 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087358    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 06:56:10.089112 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087361    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 06:56:10.089112 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087364    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 06:56:10.089112 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087366    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 06:56:10.089592 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087369    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 06:56:10.089592 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087371    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 06:56:10.089592 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087374    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 06:56:10.089592 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087376    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 06:56:10.089592 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087379    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 06:56:10.089592 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087383    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 06:56:10.089592 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087385    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 06:56:10.089592 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087388    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 06:56:10.089592 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087391    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 06:56:10.089592 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087394    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 06:56:10.089592 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087396    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 06:56:10.089592 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087399    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 06:56:10.089592 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087402    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 06:56:10.089592 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087405    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 06:56:10.089592 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087408    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 06:56:10.089592 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087411    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 06:56:10.089592 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087414    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 06:56:10.089592 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087417    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 06:56:10.089592 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087420    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 06:56:10.089592 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087422    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 06:56:10.090086 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087425    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 06:56:10.090086 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087427    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 06:56:10.090086 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087430    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 06:56:10.090086 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087433    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 06:56:10.090086 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087435    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 06:56:10.090086 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087438    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 06:56:10.090086 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087440    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 06:56:10.090086 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087443    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 06:56:10.090086 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087445    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 06:56:10.090086 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087448    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 06:56:10.090086 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087451    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 06:56:10.090086 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087454    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 06:56:10.090086 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087456    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 06:56:10.090086 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087459    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 06:56:10.090086 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087462    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 06:56:10.090086 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087465    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 06:56:10.090086 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087467    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 06:56:10.090086 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087470    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 06:56:10.090086 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087474    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 06:56:10.090531 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087476    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 06:56:10.090531 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087479    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 06:56:10.090531 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087482    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 06:56:10.090531 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087485    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 06:56:10.090531 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087487    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 06:56:10.090531 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087490    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 06:56:10.090531 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087494    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 06:56:10.090531 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087497    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 06:56:10.090531 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087500    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 06:56:10.090531 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087502    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 06:56:10.090531 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087505    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 06:56:10.090531 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087507    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 06:56:10.090531 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087512    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 06:56:10.090531 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087516    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 06:56:10.090531 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087519    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 06:56:10.090531 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087521    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 06:56:10.090531 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087524    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 06:56:10.090531 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087527    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 06:56:10.090531 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087529    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 06:56:10.090531 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087532    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 06:56:10.091028 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087535    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 06:56:10.091028 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087537    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 06:56:10.091028 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087540    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 06:56:10.091028 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087542    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 06:56:10.091028 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087545    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 06:56:10.091028 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087548    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 06:56:10.091028 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087550    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 06:56:10.091028 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087553    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 06:56:10.091028 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087556    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 06:56:10.091028 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087559    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 06:56:10.091028 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087561    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 06:56:10.091028 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087564    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 06:56:10.091028 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087568    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 06:56:10.091028 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.087571    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 06:56:10.091028 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088088    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 06:56:10.091028 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088095    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 06:56:10.091028 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088098    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 06:56:10.091028 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088100    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 06:56:10.091028 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088104    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 06:56:10.091485 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088106    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 06:56:10.091485 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088111    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 06:56:10.091485 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088115    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 06:56:10.091485 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088119    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 06:56:10.091485 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088122    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 06:56:10.091485 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088125    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 06:56:10.091485 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088128    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 06:56:10.091485 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088130    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 06:56:10.091485 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088133    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 06:56:10.091485 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088136    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 06:56:10.091485 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088139    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 06:56:10.091485 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088142    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 06:56:10.091485 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088145    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 06:56:10.091485 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088148    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 06:56:10.091485 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088151    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 06:56:10.091485 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088154    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 06:56:10.091485 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088157    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 06:56:10.091485 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088159    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 06:56:10.091485 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088162    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 06:56:10.091953 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088165    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 06:56:10.091953 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088167    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 06:56:10.091953 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088170    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 06:56:10.091953 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088173    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 06:56:10.091953 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088175    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 06:56:10.091953 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088178    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 06:56:10.091953 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088181    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 06:56:10.091953 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088184    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 06:56:10.091953 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088186    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 06:56:10.091953 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088189    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 06:56:10.091953 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088191    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 06:56:10.091953 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088194    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 06:56:10.091953 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088197    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 06:56:10.091953 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088200    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 06:56:10.091953 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088202    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 06:56:10.091953 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088205    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 06:56:10.091953 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088207    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 06:56:10.091953 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088210    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 06:56:10.091953 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088212    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 06:56:10.091953 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088215    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 06:56:10.092426 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088217    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 06:56:10.092426 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088220    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 06:56:10.092426 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088223    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 06:56:10.092426 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088226    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 06:56:10.092426 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088228    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 06:56:10.092426 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088231    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 06:56:10.092426 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088234    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 06:56:10.092426 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088236    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 06:56:10.092426 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088239    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 06:56:10.092426 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088241    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 06:56:10.092426 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088244    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 06:56:10.092426 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088246    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 06:56:10.092426 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088249    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 06:56:10.092426 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088252    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 06:56:10.092426 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088255    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 06:56:10.092426 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088257    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 06:56:10.092426 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088260    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 06:56:10.092426 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088262    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 06:56:10.092426 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088265    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 06:56:10.092426 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088268    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 06:56:10.092949 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088270    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 06:56:10.092949 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088273    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 06:56:10.092949 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088275    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 06:56:10.092949 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088278    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 06:56:10.092949 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088281    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 06:56:10.092949 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088283    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 06:56:10.092949 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088286    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 06:56:10.092949 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088288    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 06:56:10.092949 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088291    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 06:56:10.092949 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088293    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 06:56:10.092949 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088296    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 06:56:10.092949 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088299    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 06:56:10.092949 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088301    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 06:56:10.092949 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088304    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 06:56:10.092949 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088306    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 06:56:10.092949 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088309    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 06:56:10.092949 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088311    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 06:56:10.092949 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088314    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 06:56:10.092949 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088317    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 06:56:10.092949 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088320    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 06:56:10.093423 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088322    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 06:56:10.093423 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088325    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 06:56:10.093423 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088328    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 06:56:10.093423 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088330    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 06:56:10.093423 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088333    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 06:56:10.093423 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088335    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 06:56:10.093423 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088337    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 06:56:10.093423 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088340    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 06:56:10.093423 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088344    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 06:56:10.093423 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088348    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 06:56:10.093423 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088351    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 06:56:10.093423 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088353    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 06:56:10.093423 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088356    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 06:56:10.093423 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088359    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 06:56:10.093423 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088362    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 06:56:10.093423 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088364    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 06:56:10.093423 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088367    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 06:56:10.093423 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088370    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 06:56:10.093423 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088372    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 06:56:10.093894 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088376    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 06:56:10.093894 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088379    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 06:56:10.093894 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.088382    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 06:56:10.093894 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089030    2580 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 06:56:10.093894 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089042    2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 06:56:10.093894 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089048    2580 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 06:56:10.093894 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089053    2580 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 06:56:10.093894 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089058    2580 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 06:56:10.093894 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089061    2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 06:56:10.093894 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089066    2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 06:56:10.093894 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089070    2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 06:56:10.093894 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089073    2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 06:56:10.093894 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089076    2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 06:56:10.093894 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089080    2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 06:56:10.093894 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089083    2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 06:56:10.093894 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089086    2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 06:56:10.093894 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089089    2580 flags.go:64] FLAG: --cgroup-root=""
Oct 30 06:56:10.093894 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089092    2580 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 06:56:10.093894 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089095    2580 flags.go:64] FLAG: --client-ca-file=""
Oct 30 06:56:10.093894 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089101    2580 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 06:56:10.093894 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089104    2580 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 06:56:10.093894 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089109    2580 flags.go:64] FLAG: --cluster-domain=""
Oct 30 06:56:10.093894 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089112    2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 06:56:10.093894 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089115    2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 06:56:10.094456 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089119    2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 06:56:10.094456 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089122    2580 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 06:56:10.094456 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089130    2580 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 06:56:10.094456 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089134    2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 06:56:10.094456 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089137    2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 06:56:10.094456 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089140    2580 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 06:56:10.094456 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089143    2580 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 06:56:10.094456 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089146    2580 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 06:56:10.094456 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089149    2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 06:56:10.094456 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089152    2580 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 06:56:10.094456 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089156    2580 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 06:56:10.094456 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089160    2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 06:56:10.094456 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089163    2580 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 06:56:10.094456 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089166    2580 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 06:56:10.094456 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089169    2580 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 06:56:10.094456 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089172    2580 flags.go:64] FLAG: --enable-server="true"
Oct 30 06:56:10.094456 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089175    2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 06:56:10.094456 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089179    2580 flags.go:64] FLAG: --event-burst="100"
Oct 30 06:56:10.094456 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089183    2580 flags.go:64] FLAG: --event-qps="50"
Oct 30 06:56:10.094456 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089186    2580 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 06:56:10.094456 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089189    2580 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 06:56:10.094456 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089192    2580 flags.go:64] FLAG: --eviction-hard=""
Oct 30 06:56:10.094456 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089196    2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 06:56:10.094456 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089199    2580 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 06:56:10.095097 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089203    2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 06:56:10.095097 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089206    2580 flags.go:64] FLAG: --eviction-soft=""
Oct 30 06:56:10.095097 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089209    2580 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 06:56:10.095097 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089213    2580 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 06:56:10.095097 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089216    2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 06:56:10.095097 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089219    2580 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 06:56:10.095097 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089222    2580 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 06:56:10.095097 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089225    2580 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 06:56:10.095097 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089228    2580 flags.go:64] FLAG: --feature-gates=""
Oct 30 06:56:10.095097 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089232    2580 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 06:56:10.095097 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089235    2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 06:56:10.095097 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089239    2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 06:56:10.095097 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089242    2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 06:56:10.095097 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089245    2580 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 06:56:10.095097 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089248    2580 flags.go:64] FLAG: --help="false"
Oct 30 06:56:10.095097 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089251    2580 flags.go:64] FLAG: --hostname-override="ip-10-0-128-125.ec2.internal"
Oct 30 06:56:10.095097 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089255    2580 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 06:56:10.095097 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089258    2580 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 06:56:10.095097 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089262    2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 06:56:10.095097 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089265    2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 06:56:10.095097 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089269    2580 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 06:56:10.095097 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089272    2580 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 06:56:10.095097 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089275    2580 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 06:56:10.096013 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089278    2580 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 06:56:10.096013 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089281    2580 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 06:56:10.096013 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089284    2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 06:56:10.096013 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089287    2580 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 06:56:10.096013 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089290    2580 flags.go:64] FLAG: --kube-reserved=""
Oct 30 06:56:10.096013 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089293    2580 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 06:56:10.096013 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089296    2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 06:56:10.096013 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089299    2580 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 06:56:10.096013 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089302    2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 06:56:10.096013 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089305    2580 flags.go:64] FLAG: --lock-file=""
Oct 30 06:56:10.096013 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089308    2580 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 06:56:10.096013 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089311    2580 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 06:56:10.096013 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089315    2580 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 06:56:10.096013 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089321    2580 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 06:56:10.096013 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089324    2580 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 06:56:10.096013 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089328    2580 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 06:56:10.096013 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089331    2580 flags.go:64] FLAG: --logging-format="text"
Oct 30 06:56:10.096013 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089334    2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 06:56:10.096013 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089337    2580 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 06:56:10.096013 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089340    2580 flags.go:64] FLAG: --manifest-url=""
Oct 30 06:56:10.096013 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089343    2580 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 06:56:10.096013 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089348    2580 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 06:56:10.096013 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089351    2580 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 06:56:10.096013 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089355    2580 flags.go:64] FLAG: --max-pods="110"
Oct 30 06:56:10.096013 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089359    2580 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 06:56:10.097079 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089362    2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 06:56:10.097079 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089365    2580 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 06:56:10.097079 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089369    2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 06:56:10.097079 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089372    2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 06:56:10.097079 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089375    2580 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 06:56:10.097079 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089378    2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 06:56:10.097079 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089385    2580 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 06:56:10.097079 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089389    2580 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 06:56:10.097079 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089392    2580 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 06:56:10.097079 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089395    2580 flags.go:64] FLAG: --pod-cidr=""
Oct 30 06:56:10.097079 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089398    2580 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 06:56:10.097079 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089401    2580 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 06:56:10.097079 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089404    2580 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 06:56:10.097079 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089407    2580 flags.go:64] FLAG: --port="10250"
Oct 30 06:56:10.097079 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089410    2580 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 06:56:10.097079 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089413    2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d46400f776fab97e"
Oct 30 06:56:10.097079 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089416    2580 flags.go:64] FLAG: --qos-reserved=""
Oct 30 06:56:10.097079 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089419    2580 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 06:56:10.097079 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089422    2580 flags.go:64] FLAG: --register-node="true"
Oct 30 06:56:10.097079 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089426    2580 flags.go:64] FLAG: --register-with-taints=""
Oct 30 06:56:10.097079 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089431    2580 flags.go:64] FLAG: --registry-burst="10"
Oct 30 06:56:10.097079 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089434    2580 flags.go:64] FLAG: --registry-qps="5"
Oct 30 06:56:10.097079 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089437    2580 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 06:56:10.097079 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089440    2580 flags.go:64] FLAG: --reserved-memory=""
Oct 30 06:56:10.097079 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089444    2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 06:56:10.097079 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089447    2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 06:56:10.098252 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089450    2580 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 06:56:10.098252 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089453    2580 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 06:56:10.098252 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089456    2580 flags.go:64] FLAG: --runonce="false"
Oct 30 06:56:10.098252 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089460    2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 06:56:10.098252 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089463    2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 06:56:10.098252 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089466    2580 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 06:56:10.098252 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089469    2580 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 06:56:10.098252 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089472    2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 06:56:10.098252 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089477    2580 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 06:56:10.098252 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089480    2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 06:56:10.098252 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089483    2580 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 06:56:10.098252 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089486    2580 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 06:56:10.098252 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089489    2580 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 06:56:10.098252 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089492    2580 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 06:56:10.098252 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089495    2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 06:56:10.098252 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089498    2580 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 06:56:10.098252 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089501    2580 flags.go:64] FLAG: --system-cgroups=""
Oct 30 06:56:10.098252 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089504    2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 06:56:10.098252 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089509    2580 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 06:56:10.098252 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089512    2580 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 06:56:10.098252 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089515    2580 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 06:56:10.098252 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089519    2580 flags.go:64] FLAG: --tls-min-version=""
Oct 30 06:56:10.098252 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089522    2580 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 06:56:10.098252 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089525    2580 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 06:56:10.098252 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089528    2580 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 06:56:10.099469 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089533    2580 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 06:56:10.099469 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089536    2580 flags.go:64] FLAG: --v="2"
Oct 30 06:56:10.099469 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089540    2580 flags.go:64] FLAG: --version="false"
Oct 30 06:56:10.099469 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089545    2580 flags.go:64] FLAG: --vmodule=""
Oct 30 06:56:10.099469 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089549    2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 06:56:10.099469 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.089552    2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 06:56:10.099469 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089724    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 06:56:10.099469 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089730    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 06:56:10.099469 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089733    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 06:56:10.099469 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089736    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 06:56:10.099469 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089739    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 06:56:10.099469 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089742    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 06:56:10.099469 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089745    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 06:56:10.099469 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089748    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 06:56:10.099469 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089751    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 06:56:10.099469 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089755    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 06:56:10.099469 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089758    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 06:56:10.099469 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089761    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 06:56:10.099469 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089765    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 06:56:10.099469 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089768    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 06:56:10.099469 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089771    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 06:56:10.100467 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089774    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 06:56:10.100467 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089777    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 06:56:10.100467 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089779    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 06:56:10.100467 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089782    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 06:56:10.100467 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089784    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 06:56:10.100467 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089787    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 06:56:10.100467 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089790    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 06:56:10.100467 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089792    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 06:56:10.100467 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089795    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 06:56:10.100467 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089798    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 06:56:10.100467 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089801    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 06:56:10.100467 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089817    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 06:56:10.100467 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089820    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 06:56:10.100467 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089822    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 06:56:10.100467 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089825    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 06:56:10.100467 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089828    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 06:56:10.100467 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089831    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 06:56:10.100467 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089833    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 06:56:10.100467 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089836    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 06:56:10.101031 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089839    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 06:56:10.101031 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089841    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 06:56:10.101031 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089844    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 06:56:10.101031 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089846    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 06:56:10.101031 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089849    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 06:56:10.101031 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089852    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 06:56:10.101031 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089856    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 06:56:10.101031 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089862    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 06:56:10.101031 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089866    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 06:56:10.101031 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089869    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 06:56:10.101031 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089872    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 06:56:10.101031 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089875    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 06:56:10.101031 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089877    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 06:56:10.101031 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089880    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 06:56:10.101031 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089883    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 06:56:10.101031 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089886    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 06:56:10.101031 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089888    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 06:56:10.101031 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089891    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 06:56:10.101031 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089894    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 06:56:10.101820 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089897    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 06:56:10.101820 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089899    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 06:56:10.101820 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089902    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 06:56:10.101820 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089904    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 06:56:10.101820 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089907    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 06:56:10.101820 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089911    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 06:56:10.101820 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089914    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 06:56:10.101820 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089916    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 06:56:10.101820 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089919    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 06:56:10.101820 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089922    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 06:56:10.101820 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089924    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 06:56:10.101820 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089927    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 06:56:10.101820 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089929    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 06:56:10.101820 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089932    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 06:56:10.101820 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089934    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 06:56:10.101820 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089937    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 06:56:10.101820 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089940    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 06:56:10.101820 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089942    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 06:56:10.101820 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089950    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 06:56:10.101820 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089953    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 06:56:10.102629 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089958    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 06:56:10.102629 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089960    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 06:56:10.102629 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089963    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 06:56:10.102629 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089966    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 06:56:10.102629 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089969    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 06:56:10.102629 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089971    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 06:56:10.102629 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089974    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 06:56:10.102629 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089976    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 06:56:10.102629 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089979    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 06:56:10.102629 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089981    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 06:56:10.102629 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089984    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 06:56:10.102629 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089987    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 06:56:10.102629 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089989    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 06:56:10.102629 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089992    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 06:56:10.102629 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089995    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 06:56:10.102629 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.089997    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 06:56:10.102629 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090000    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 06:56:10.102629 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090004    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 06:56:10.102629 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090006    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 06:56:10.103277 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090009    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 06:56:10.103277 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090012    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 06:56:10.103277 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090014    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 06:56:10.103277 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090017    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 06:56:10.103277 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090019    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 06:56:10.103277 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090022    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 06:56:10.103277 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090024    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 06:56:10.103277 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090027    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 06:56:10.103277 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090029    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 06:56:10.103277 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090032    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 06:56:10.103277 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090035    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 06:56:10.103277 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090037    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 06:56:10.103277 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090046    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 06:56:10.103277 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090050    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 06:56:10.103277 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.090076    2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 06:56:10.103277 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090117    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 06:56:10.103675 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090123    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 06:56:10.103675 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090126    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 06:56:10.103675 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090129    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 06:56:10.103675 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090132    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 06:56:10.103675 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090134    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 06:56:10.103675 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090137    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 06:56:10.103675 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090140    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 06:56:10.103675 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090143    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 06:56:10.103675 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090145    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 06:56:10.103675 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090148    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 06:56:10.103675 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090150    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 06:56:10.103675 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090153    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 06:56:10.103675 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090155    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 06:56:10.103675 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090158    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 06:56:10.103675 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090162    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 06:56:10.103675 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090165    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 06:56:10.103675 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090167    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 06:56:10.103675 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090170    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 06:56:10.103675 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090173    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 06:56:10.103675 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090175    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 06:56:10.104232 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090178    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 06:56:10.104232 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090181    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 06:56:10.104232 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090184    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 06:56:10.104232 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090186    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 06:56:10.104232 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090189    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 06:56:10.104232 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090191    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 06:56:10.104232 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090194    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 06:56:10.104232 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090196    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 06:56:10.104232 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090199    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 06:56:10.104232 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090202    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 06:56:10.104232 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090206    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 06:56:10.104232 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090209    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 06:56:10.104232 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090212    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 06:56:10.104232 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090215    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 06:56:10.104232 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090217    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 06:56:10.104232 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090220    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 06:56:10.104232 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090222    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 06:56:10.104232 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090225    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 06:56:10.104232 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090228    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 06:56:10.104680 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090230    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 06:56:10.104680 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090233    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 06:56:10.104680 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090236    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 06:56:10.104680 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090238    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 06:56:10.104680 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090241    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 06:56:10.104680 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090243    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 06:56:10.104680 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090246    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 06:56:10.104680 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090250    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 06:56:10.104680 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090253    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 06:56:10.104680 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090255    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 06:56:10.104680 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090258    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 06:56:10.104680 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090261    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 06:56:10.104680 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090263    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 06:56:10.104680 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090266    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 06:56:10.104680 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090269    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 06:56:10.104680 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090272    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 06:56:10.104680 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090274    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 06:56:10.104680 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090277    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 06:56:10.104680 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090280    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 06:56:10.104680 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090282    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 06:56:10.105323 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090285    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 06:56:10.105323 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090288    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 06:56:10.105323 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090292    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 06:56:10.105323 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090298    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 06:56:10.105323 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090302    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 06:56:10.105323 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090305    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 06:56:10.105323 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090307    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 06:56:10.105323 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090310    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 06:56:10.105323 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090313    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 06:56:10.105323 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090315    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 06:56:10.105323 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090318    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 06:56:10.105323 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090320    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 06:56:10.105323 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090323    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 06:56:10.105323 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090326    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 06:56:10.105323 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090328    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 06:56:10.105323 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090331    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 06:56:10.105323 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090333    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 06:56:10.105323 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090336    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 06:56:10.105323 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090339    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 06:56:10.105801 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090343    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 06:56:10.105801 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090346    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 06:56:10.105801 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090349    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 06:56:10.105801 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090352    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 06:56:10.105801 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090354    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 06:56:10.105801 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090357    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 06:56:10.105801 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090360    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 06:56:10.105801 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090362    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 06:56:10.105801 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090365    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 06:56:10.105801 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090367    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 06:56:10.105801 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090370    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 06:56:10.105801 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090373    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 06:56:10.105801 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090376    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 06:56:10.105801 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090380    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 06:56:10.105801 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090383    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 06:56:10.105801 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090386    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 06:56:10.105801 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090390    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 06:56:10.105801 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090393    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 06:56:10.105801 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090396    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 06:56:10.106267 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090398    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 06:56:10.106267 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090401    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 06:56:10.106267 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090404    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 06:56:10.106267 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090407    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 06:56:10.106267 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090410    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 06:56:10.106267 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090412    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 06:56:10.106267 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090415    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 06:56:10.106267 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.090418    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 06:56:10.106267 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.098244    2580 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 06:56:10.106267 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.098263    2580 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 06:56:10.106267 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.098277    2580 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 06:56:10.106267 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.098284    2580 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 06:56:10.106267 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098381    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 06:56:10.106267 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098389    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 06:56:10.106267 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098397    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 06:56:10.106267 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098404    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 06:56:10.106267 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098409    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 06:56:10.106267 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098414    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 06:56:10.106267 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098419    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 06:56:10.106942 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098424    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 06:56:10.106942 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098429    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 06:56:10.106942 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098434    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 06:56:10.106942 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098438    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 06:56:10.106942 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098442    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 06:56:10.106942 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098447    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 06:56:10.106942 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098451    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 06:56:10.106942 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098456    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 06:56:10.106942 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098460    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 06:56:10.106942 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098464    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 06:56:10.106942 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098469    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 06:56:10.106942 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098473    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 06:56:10.106942 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098477    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 06:56:10.106942 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098482    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 06:56:10.106942 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098487    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 06:56:10.106942 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098491    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 06:56:10.106942 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098495    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 06:56:10.106942 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098500    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 06:56:10.106942 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098504    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 06:56:10.107697 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098509    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 06:56:10.107697 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098514    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 06:56:10.107697 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098518    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 06:56:10.107697 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098523    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 06:56:10.107697 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098528    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 06:56:10.107697 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098532    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 06:56:10.107697 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098536    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 06:56:10.107697 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098541    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 06:56:10.107697 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098545    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 06:56:10.107697 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098550    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 06:56:10.107697 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098554    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 06:56:10.107697 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098558    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 06:56:10.107697 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098562    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 06:56:10.107697 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098567    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 06:56:10.107697 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098571    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 06:56:10.107697 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098575    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 06:56:10.107697 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098579    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 06:56:10.107697 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098583    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 06:56:10.107697 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098587    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 06:56:10.107697 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098591    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 06:56:10.108524 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098596    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 06:56:10.108524 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098600    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 06:56:10.108524 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098604    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 06:56:10.108524 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098609    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 06:56:10.108524 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098613    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 06:56:10.108524 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098618    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 06:56:10.108524 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098622    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 06:56:10.108524 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098626    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 06:56:10.108524 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098630    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 06:56:10.108524 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098635    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 06:56:10.108524 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098640    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 06:56:10.108524 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098644    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 06:56:10.108524 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098649    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 06:56:10.108524 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098653    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 06:56:10.108524 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098658    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 06:56:10.108524 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098662    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 06:56:10.108524 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098667    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 06:56:10.108524 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098672    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 06:56:10.108524 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098676    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 06:56:10.108524 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098681    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 06:56:10.109058 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098685    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 06:56:10.109058 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098690    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 06:56:10.109058 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098695    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 06:56:10.109058 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098699    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 06:56:10.109058 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098704    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 06:56:10.109058 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098708    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 06:56:10.109058 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098712    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 06:56:10.109058 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098719    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 06:56:10.109058 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098727    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 06:56:10.109058 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098732    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 06:56:10.109058 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098737    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 06:56:10.109058 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098741    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 06:56:10.109058 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098745    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 06:56:10.109058 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098750    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 06:56:10.109058 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098754    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 06:56:10.109058 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098758    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 06:56:10.109058 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098763    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 06:56:10.109058 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098767    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 06:56:10.109058 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098772    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 06:56:10.109504 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098776    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 06:56:10.109504 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098780    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 06:56:10.109504 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098785    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 06:56:10.109504 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098789    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 06:56:10.109504 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098793    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 06:56:10.109504 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098798    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 06:56:10.109504 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098819    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 06:56:10.109504 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098824    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 06:56:10.109504 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098828    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 06:56:10.109504 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098832    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 06:56:10.109504 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098837    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 06:56:10.109504 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098841    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 06:56:10.109504 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098844    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 06:56:10.109504 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098848    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 06:56:10.109504 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098852    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 06:56:10.109504 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098856    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 06:56:10.109504 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098860    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 06:56:10.109504 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098864    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 06:56:10.109504 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098868    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 06:56:10.109504 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098872    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 06:56:10.109985 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.098876    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 06:56:10.109985 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.098920    2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 06:56:10.109985 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099207    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 06:56:10.109985 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099215    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 06:56:10.109985 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099220    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 06:56:10.109985 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099225    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 06:56:10.109985 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099230    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 06:56:10.109985 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099234    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 06:56:10.109985 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099239    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 06:56:10.109985 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099243    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 06:56:10.109985 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099247    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 06:56:10.109985 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099251    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 06:56:10.109985 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099256    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 06:56:10.109985 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099260    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 06:56:10.109985 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099264    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 06:56:10.109985 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099268    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 06:56:10.110360 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099273    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 06:56:10.110360 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099279    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 06:56:10.110360 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099285    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 06:56:10.110360 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099291    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 06:56:10.110360 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099296    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 06:56:10.110360 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099301    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 06:56:10.110360 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099306    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 06:56:10.110360 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099318    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 06:56:10.110360 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099323    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 06:56:10.110360 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099328    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 06:56:10.110360 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099332    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 06:56:10.110360 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099336    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 06:56:10.110360 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099340    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 06:56:10.110360 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099344    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 06:56:10.110360 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099349    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 06:56:10.110360 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099353    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 06:56:10.110360 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099357    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 06:56:10.110360 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099361    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 06:56:10.110360 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099365    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 06:56:10.110841 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099370    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 06:56:10.110841 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099374    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 06:56:10.110841 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099378    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 06:56:10.110841 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099382    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 06:56:10.110841 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099386    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 06:56:10.110841 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099390    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 06:56:10.110841 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099394    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 06:56:10.110841 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099399    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 06:56:10.110841 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099403    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 06:56:10.110841 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099407    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 06:56:10.110841 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099411    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 06:56:10.110841 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099416    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 06:56:10.110841 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099420    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 06:56:10.110841 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099424    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 06:56:10.110841 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099431    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 06:56:10.110841 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099438    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 06:56:10.110841 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099443    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 06:56:10.110841 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099448    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 06:56:10.111418 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099452    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 06:56:10.111418 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099456    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 06:56:10.111418 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099460    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 06:56:10.111418 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099472    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 06:56:10.111418 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099477    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 06:56:10.111418 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099481    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 06:56:10.111418 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099485    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 06:56:10.111418 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099488    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 06:56:10.111418 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099493    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 06:56:10.111418 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099497    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 06:56:10.111418 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099502    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 06:56:10.111418 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099506    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 06:56:10.111418 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099511    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 06:56:10.111418 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099515    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 06:56:10.111418 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099519    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 06:56:10.111418 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099524    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 06:56:10.111418 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099528    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 06:56:10.111418 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099533    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 06:56:10.111418 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099545    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 06:56:10.111418 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099550    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 06:56:10.111942 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099554    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 06:56:10.111942 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099558    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 06:56:10.111942 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099562    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 06:56:10.111942 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099567    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 06:56:10.111942 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099571    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 06:56:10.111942 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099576    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 06:56:10.111942 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099580    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 06:56:10.111942 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099584    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 06:56:10.111942 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099588    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 06:56:10.111942 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099592    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 06:56:10.111942 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099597    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 06:56:10.111942 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099602    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 06:56:10.111942 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099606    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 06:56:10.111942 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099610    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 06:56:10.111942 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099614    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 06:56:10.111942 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099618    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 06:56:10.111942 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099630    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 06:56:10.111942 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099635    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 06:56:10.111942 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099639    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 06:56:10.111942 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099643    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 06:56:10.112649 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099647    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 06:56:10.112649 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099652    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 06:56:10.112649 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099656    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 06:56:10.112649 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099660    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 06:56:10.112649 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099664    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 06:56:10.112649 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099668    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 06:56:10.112649 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099673    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 06:56:10.112649 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099677    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 06:56:10.112649 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099681    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 06:56:10.112649 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099686    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 06:56:10.112649 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099690    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 06:56:10.112649 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099694    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 06:56:10.112649 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099699    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 06:56:10.112649 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099703    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 06:56:10.112649 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099707    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 06:56:10.112649 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.099746    2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 06:56:10.113136 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099834    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 06:56:10.113136 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099841    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 06:56:10.113136 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099845    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 06:56:10.113136 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099849    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 06:56:10.113136 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099853    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 06:56:10.113136 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099857    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 06:56:10.113136 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099861    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 06:56:10.113136 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099867    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 06:56:10.113136 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099873    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 06:56:10.113136 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099877    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 06:56:10.113136 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099881    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 06:56:10.113136 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099884    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 06:56:10.113136 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099888    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 06:56:10.113136 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099901    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 06:56:10.113136 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099905    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 06:56:10.113136 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099910    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 06:56:10.113136 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099914    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 06:56:10.113136 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099918    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 06:56:10.113136 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099922    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 06:56:10.113616 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099926    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 06:56:10.113616 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099930    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 06:56:10.113616 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099934    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 06:56:10.113616 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099938    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 06:56:10.113616 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099942    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 06:56:10.113616 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099946    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 06:56:10.113616 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099951    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 06:56:10.113616 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099956    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 06:56:10.113616 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099960    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 06:56:10.113616 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099964    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 06:56:10.113616 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099972    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 06:56:10.113616 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099976    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 06:56:10.113616 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099981    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 06:56:10.113616 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099985    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 06:56:10.113616 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099989    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 06:56:10.113616 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099993    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 06:56:10.113616 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.099998    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 06:56:10.113616 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100002    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 06:56:10.113616 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100006    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 06:56:10.113616 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100010    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 06:56:10.114178 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100014    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 06:56:10.114178 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100018    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 06:56:10.114178 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100023    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 06:56:10.114178 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100027    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 06:56:10.114178 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100031    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 06:56:10.114178 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100036    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 06:56:10.114178 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100040    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 06:56:10.114178 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100052    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 06:56:10.114178 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100057    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 06:56:10.114178 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100061    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 06:56:10.114178 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100065    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 06:56:10.114178 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100069    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 06:56:10.114178 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100074    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 06:56:10.114178 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100078    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 06:56:10.114178 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100082    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 06:56:10.114178 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100086    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 06:56:10.114178 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100090    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 06:56:10.114178 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100095    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 06:56:10.114178 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100099    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 06:56:10.114178 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100103    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 06:56:10.114862 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100107    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 06:56:10.114862 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100112    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 06:56:10.114862 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100121    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 06:56:10.114862 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100125    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 06:56:10.114862 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100132    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 06:56:10.114862 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100136    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 06:56:10.114862 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100140    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 06:56:10.114862 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100145    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 06:56:10.114862 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100149    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 06:56:10.114862 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100154    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 06:56:10.114862 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100158    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 06:56:10.114862 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100163    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 06:56:10.114862 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100167    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 06:56:10.114862 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100171    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 06:56:10.114862 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100176    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 06:56:10.114862 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100180    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 06:56:10.114862 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100184    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 06:56:10.114862 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100188    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 06:56:10.114862 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100192    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 06:56:10.115358 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100197    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 06:56:10.115358 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100209    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 06:56:10.115358 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100213    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 06:56:10.115358 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100220    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 06:56:10.115358 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100226    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 06:56:10.115358 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100231    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 06:56:10.115358 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100236    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 06:56:10.115358 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100240    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 06:56:10.115358 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100244    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 06:56:10.115358 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100248    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 06:56:10.115358 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100253    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 06:56:10.115358 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100257    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 06:56:10.115358 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100261    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 06:56:10.115358 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100265    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 06:56:10.115358 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100269    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 06:56:10.115358 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100275    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 06:56:10.115358 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100279    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 06:56:10.115358 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100285    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 06:56:10.115358 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100289    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 06:56:10.115785 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100294    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 06:56:10.115785 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100298    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 06:56:10.115785 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100302    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 06:56:10.115785 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100306    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 06:56:10.115785 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100310    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 06:56:10.115785 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100314    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 06:56:10.115785 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100318    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 06:56:10.115785 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100323    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 06:56:10.115785 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:10.100329    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 06:56:10.115785 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.100450    2580 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 06:56:10.115785 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.103030    2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 06:56:10.115785 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.103968    2580 server.go:1014] "Starting client certificate rotation"
Oct 30 06:56:10.115785 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.104079    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 06:56:10.115785 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.104128    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 06:56:10.115785 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.107270    2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 06:56:10.115785 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.113575    2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 06:56:10.126230 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.126211    2580 log.go:25] "Validated CRI v1 runtime API"
Oct 30 06:56:10.129147 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.129132    2580 log.go:25] "Validated CRI v1 image API"
Oct 30 06:56:10.129944 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.129918    2580 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 06:56:10.134388 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.134358    2580 fs.go:135] Filesystem UUIDs: map[404a11c7-7d36-440d-a1a4-22b3ef3b102f:/dev/nvme0n1p3 46e964aa-6867-435b-949d-a90b55fe4c93:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Oct 30 06:56:10.134493 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.134385    2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 06:56:10.139944 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.139826    2580 manager.go:217] Machine: {Timestamp:2026-10-30 06:56:10.138689998 +0000 UTC m=+0.326003763 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3125418 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2661e942f7794e68faf61846d0e2d2 SystemUUID:ec2661e9-42f7-794e-68fa-f61846d0e2d2 BootID:2b64aa7f-6a0d-404d-87d7-a2a59206956f Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ec:79:d4:b7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ec:79:d4:b7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:96:28:50:21:8e:78 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 06:56:10.143145 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.143135    2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 06:56:10.143254 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.143241    2580 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 06:56:10.145722 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.145698    2580 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 06:56:10.145864 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.145724    2580 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-125.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 06:56:10.145906 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.145868    2580 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 06:56:10.145906 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.145876    2580 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 06:56:10.145906 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.145885    2580 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 06:56:10.146615 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.146605    2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 06:56:10.147321 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.147312    2580 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 06:56:10.150940 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.150922    2580 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 06:56:10.151726 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.151717    2580 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 06:56:10.152376 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.152366    2580 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 06:56:10.152406 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.152382    2580 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 06:56:10.152406 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.152399    2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 06:56:10.152468 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.152412    2580 kubelet.go:405] "Adding apiserver pod source"
Oct 30 06:56:10.152468 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.152423    2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 06:56:10.152849 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.152835    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 06:56:10.155388 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.155373    2580 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 06:56:10.161382 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.161370    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 06:56:10.162213 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.162202    2580 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 06:56:10.162248 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.162231    2580 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 06:56:10.162374 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.162365    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 06:56:10.162408 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.162379    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 06:56:10.162408 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.162385    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 06:56:10.162408 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.162391    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 06:56:10.162408 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.162397    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 06:56:10.162408 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.162403    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 06:56:10.162861 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.162852    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 06:56:10.162897 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.162861    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 06:56:10.162897 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.162868    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 06:56:10.162897 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.162874    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 06:56:10.162897 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.162885    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 06:56:10.162897 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.162894    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 06:56:10.163626 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.163615    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 06:56:10.163658 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.163629    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 06:56:10.167150 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.167136    2580 server.go:1263] "Started kubelet"
Oct 30 06:56:10.167266 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.167220    2580 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 06:56:10.167318 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.167260    2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 06:56:10.167350 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.167328    2580 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 06:56:10.167787 ip-10-0-128-125 systemd[1]: Started Kubernetes Kubelet.
Oct 30 06:56:10.170004 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.169978    2580 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 06:56:10.172607 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.172588    2580 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 06:56:10.175961 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:10.175942    2580 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 06:56:10.179157 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.179138    2580 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 06:56:10.179255 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.179190    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 06:56:10.179255 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:10.179225    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-125.ec2.internal\" not found"
Oct 30 06:56:10.179663 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.179644    2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 06:56:10.179663 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.179645    2580 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 06:56:10.179817 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.179669    2580 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 06:56:10.179817 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.179758    2580 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 06:56:10.179817 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.179769    2580 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 06:56:10.180312 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.180294    2580 factory.go:153] Registering CRI-O factory
Oct 30 06:56:10.180312 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.180314    2580 factory.go:223] Registration of the crio container factory successfully
Oct 30 06:56:10.180450 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.180397    2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 06:56:10.180450 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.180405    2580 factory.go:55] Registering systemd factory
Oct 30 06:56:10.180450 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.180413    2580 factory.go:223] Registration of the systemd container factory successfully
Oct 30 06:56:10.180450 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.180428    2580 factory.go:103] Registering Raw factory
Oct 30 06:56:10.180450 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.180437    2580 manager.go:1199] Started watching for new ooms in manager
Oct 30 06:56:10.181331 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.181309    2580 manager.go:319] Starting recovery of all containers
Oct 30 06:56:10.192093 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.192076    2580 manager.go:324] Recovery completed
Oct 30 06:56:10.196480 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.196468    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 06:56:10.200132 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.200114    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-125.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 06:56:10.200212 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.200146    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-125.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 06:56:10.200212 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.200161    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-125.ec2.internal" event="NodeHasSufficientPID"
Oct 30 06:56:10.200699 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:10.200676    2580 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-125.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 06:56:10.200772 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.200684    2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-125.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 06:56:10.202131 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:10.200685    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-125.ec2.internal.18e33c4b2b1c9a81  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-125.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-125.ec2.internal,},FirstTimestamp:2026-10-30 06:56:10.167097985 +0000 UTC m=+0.354411744,LastTimestamp:2026-10-30 06:56:10.167097985 +0000 UTC m=+0.354411744,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-125.ec2.internal,}"
Oct 30 06:56:10.202378 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.202360    2580 cpu_manager.go:226] "Starting" policy="none"
Oct 30 06:56:10.202475 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.202382    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 06:56:10.202576 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.202486    2580 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 06:56:10.202657 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.202648    2580 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 06:56:10.204886 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.204869    2580 policy_none.go:50] "Start"
Oct 30 06:56:10.204960 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.204890    2580 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 06:56:10.204960 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.204906    2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 06:56:10.207020 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.207006    2580 policy_none.go:44] "Start"
Oct 30 06:56:10.207131 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.207112    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-125.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 06:56:10.207224 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.207146    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-125.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 06:56:10.207224 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.207161    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-125.ec2.internal" event="NodeHasSufficientPID"
Oct 30 06:56:10.211094 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.211075    2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 06:56:10.211214 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.211119    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kxv5v"
Oct 30 06:56:10.227194 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:10.227119    2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 06:56:10.227267 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:10.227235    2580 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 06:56:10.227267 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:10.227253    2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-125.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 06:56:10.227373 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:10.227309    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-125.ec2.internal.18e33c4b2d14a4bf  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-125.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-125.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-125.ec2.internal,},FirstTimestamp:2026-10-30 06:56:10.200130751 +0000 UTC m=+0.387444527,LastTimestamp:2026-10-30 06:56:10.200130751 +0000 UTC m=+0.387444527,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-125.ec2.internal,}"
Oct 30 06:56:10.238389 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.238361    2580 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 06:56:10.240859 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.240844    2580 manager.go:341] "Starting Device Plugin manager"
Oct 30 06:56:10.240953 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:10.240880    2580 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 06:56:10.240953 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.240891    2580 server.go:87] "Starting device plugin registration server"
Oct 30 06:56:10.241166 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.241154    2580 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 06:56:10.241231 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.241167    2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 06:56:10.241279 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.241249    2580 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 06:56:10.241328 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.241322    2580 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 06:56:10.241381 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.241330    2580 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 06:56:10.241920 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:10.241900    2580 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 06:56:10.241997 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:10.241948    2580 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-125.ec2.internal\" not found"
Oct 30 06:56:10.342077 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.342027    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 06:56:10.342898 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.342882    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-125.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 06:56:10.343012 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.342912    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-125.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 06:56:10.343012 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.342926    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-125.ec2.internal" event="NodeHasSufficientPID"
Oct 30 06:56:10.343012 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.342972    2580 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-125.ec2.internal"
Oct 30 06:56:10.362745 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.362718    2580 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-125.ec2.internal"
Oct 30 06:56:10.362745 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:10.362743    2580 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-125.ec2.internal\": node \"ip-10-0-128-125.ec2.internal\" not found"
Oct 30 06:56:10.386758 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.386737    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 06:56:10.387855 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.387838    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 06:56:10.387855 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.387853    2580 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 06:56:10.387985 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.387871    2580 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 06:56:10.387985 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:10.387905    2580 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 06:56:10.404540 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.404524    2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 06:56:10.411796 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:10.411780    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-125.ec2.internal\" not found"
Oct 30 06:56:10.488652 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.488633    2580 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-125.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-125.ec2.internal"]
Oct 30 06:56:10.488732 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.488689    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 06:56:10.489482 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.489468    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-125.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 06:56:10.489557 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.489495    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-125.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 06:56:10.489557 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.489511    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-125.ec2.internal" event="NodeHasSufficientPID"
Oct 30 06:56:10.491517 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.491506    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 06:56:10.491654 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.491641    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-125.ec2.internal"
Oct 30 06:56:10.491713 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.491669    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 06:56:10.492219 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.492203    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-125.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 06:56:10.492270 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.492233    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-125.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 06:56:10.492270 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.492245    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-125.ec2.internal" event="NodeHasSufficientPID"
Oct 30 06:56:10.492348 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.492205    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-125.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 06:56:10.492348 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.492311    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-125.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 06:56:10.492348 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.492327    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-125.ec2.internal" event="NodeHasSufficientPID"
Oct 30 06:56:10.494296 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.494277    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-125.ec2.internal"
Oct 30 06:56:10.494296 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.494300    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 06:56:10.495021 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.495003    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-125.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 06:56:10.495109 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.495031    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-125.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 06:56:10.495109 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.495042    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-125.ec2.internal" event="NodeHasSufficientPID"
Oct 30 06:56:10.512174 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:10.512154    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-125.ec2.internal\" not found"
Oct 30 06:56:10.520196 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:10.520180    2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-125.ec2.internal\" not found" node="ip-10-0-128-125.ec2.internal"
Oct 30 06:56:10.522951 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:10.522929    2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-125.ec2.internal\" not found" node="ip-10-0-128-125.ec2.internal"
Oct 30 06:56:10.582003 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.581983    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4ba2b683794afdaba9ca4ed884bfac10-config\") pod \"kube-apiserver-proxy-ip-10-0-128-125.ec2.internal\" (UID: \"4ba2b683794afdaba9ca4ed884bfac10\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-125.ec2.internal"
Oct 30 06:56:10.582083 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.582008    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0aefa49a1bd7275a0d110a8cd8b72657-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-125.ec2.internal\" (UID: \"0aefa49a1bd7275a0d110a8cd8b72657\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-125.ec2.internal"
Oct 30 06:56:10.582083 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.582025    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0aefa49a1bd7275a0d110a8cd8b72657-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-125.ec2.internal\" (UID: \"0aefa49a1bd7275a0d110a8cd8b72657\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-125.ec2.internal"
Oct 30 06:56:10.613023 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:10.612980    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-125.ec2.internal\" not found"
Oct 30 06:56:10.682333 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.682315    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0aefa49a1bd7275a0d110a8cd8b72657-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-125.ec2.internal\" (UID: \"0aefa49a1bd7275a0d110a8cd8b72657\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-125.ec2.internal"
Oct 30 06:56:10.682416 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.682349    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4ba2b683794afdaba9ca4ed884bfac10-config\") pod \"kube-apiserver-proxy-ip-10-0-128-125.ec2.internal\" (UID: \"4ba2b683794afdaba9ca4ed884bfac10\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-125.ec2.internal"
Oct 30 06:56:10.682416 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.682367    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0aefa49a1bd7275a0d110a8cd8b72657-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-125.ec2.internal\" (UID: \"0aefa49a1bd7275a0d110a8cd8b72657\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-125.ec2.internal"
Oct 30 06:56:10.682479 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.682429    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0aefa49a1bd7275a0d110a8cd8b72657-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-125.ec2.internal\" (UID: \"0aefa49a1bd7275a0d110a8cd8b72657\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-125.ec2.internal"
Oct 30 06:56:10.682511 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.682484    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0aefa49a1bd7275a0d110a8cd8b72657-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-125.ec2.internal\" (UID: \"0aefa49a1bd7275a0d110a8cd8b72657\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-125.ec2.internal"
Oct 30 06:56:10.682570 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.682518    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4ba2b683794afdaba9ca4ed884bfac10-config\") pod \"kube-apiserver-proxy-ip-10-0-128-125.ec2.internal\" (UID: \"4ba2b683794afdaba9ca4ed884bfac10\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-125.ec2.internal"
Oct 30 06:56:10.713580 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:10.713556    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-125.ec2.internal\" not found"
Oct 30 06:56:10.814042 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:10.814011    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-125.ec2.internal\" not found"
Oct 30 06:56:10.823199 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.823178    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-125.ec2.internal"
Oct 30 06:56:10.826292 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:10.826275    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-125.ec2.internal"
Oct 30 06:56:10.915338 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:10.915105    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-125.ec2.internal\" not found"
Oct 30 06:56:11.015566 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:11.015538    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-125.ec2.internal\" not found"
Oct 30 06:56:11.104039 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:11.104007    2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 06:56:11.104453 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:11.104148    2580 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 06:56:11.116308 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:11.116289    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-125.ec2.internal\" not found"
Oct 30 06:56:11.216032 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:11.215955    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 06:51:11 +0000 UTC" deadline="2028-07-16 19:21:32.20143724 +0000 UTC"
Oct 30 06:56:11.216032 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:11.215991    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15012h25m20.985449371s"
Oct 30 06:56:11.217064 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:11.217046    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-125.ec2.internal\" not found"
Oct 30 06:56:11.286072 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:11.286047    2580 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 06:56:11.317813 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:11.317778    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-125.ec2.internal\" not found"
Oct 30 06:56:11.349432 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:11.349387    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aefa49a1bd7275a0d110a8cd8b72657.slice/crio-3420c9754d38d1b6f1bf2af5f6dd65d8bc93f12c95d4a4f1e9bdfdfd8a76bd7f WatchSource:0}: Error finding container 3420c9754d38d1b6f1bf2af5f6dd65d8bc93f12c95d4a4f1e9bdfdfd8a76bd7f: Status 404 returned error can't find the container with id 3420c9754d38d1b6f1bf2af5f6dd65d8bc93f12c95d4a4f1e9bdfdfd8a76bd7f
Oct 30 06:56:11.349844 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:11.349814    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ba2b683794afdaba9ca4ed884bfac10.slice/crio-cab1d1b7478b01d62e7af6636d06b4f3300b2fe9d55994cd23840f04d88290c3 WatchSource:0}: Error finding container cab1d1b7478b01d62e7af6636d06b4f3300b2fe9d55994cd23840f04d88290c3: Status 404 returned error can't find the container with id cab1d1b7478b01d62e7af6636d06b4f3300b2fe9d55994cd23840f04d88290c3
Oct 30 06:56:11.356679 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:11.356663    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 06:56:11.389011 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:11.388983    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 06:56:11.390202 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:11.390179    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-125.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 06:56:11.390316 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:11.390222    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-125.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 06:56:11.390316 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:11.390238    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-125.ec2.internal" event="NodeHasSufficientPID"
Oct 30 06:56:11.391002 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:11.390963    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-125.ec2.internal" event={"ID":"4ba2b683794afdaba9ca4ed884bfac10","Type":"ContainerStarted","Data":"cab1d1b7478b01d62e7af6636d06b4f3300b2fe9d55994cd23840f04d88290c3"}
Oct 30 06:56:11.392111 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:11.392088    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-125.ec2.internal" event={"ID":"0aefa49a1bd7275a0d110a8cd8b72657","Type":"ContainerStarted","Data":"3420c9754d38d1b6f1bf2af5f6dd65d8bc93f12c95d4a4f1e9bdfdfd8a76bd7f"}
Oct 30 06:56:11.418198 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:11.418167    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-125.ec2.internal\" not found"
Oct 30 06:56:11.520356 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:11.520289    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-125.ec2.internal\" not found"
Oct 30 06:56:11.621162 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:11.621131    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-125.ec2.internal\" not found"
Oct 30 06:56:11.684158 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:11.684134    2580 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 06:56:11.717308 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:11.717284    2580 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 06:56:11.779778 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:11.779708    2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-125.ec2.internal"
Oct 30 06:56:11.803910 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:11.803888    2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 06:56:11.805225 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:11.805008    2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-125.ec2.internal"
Oct 30 06:56:11.825606 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:11.825580    2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 06:56:12.154549 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.154466    2580 apiserver.go:52] "Watching apiserver"
Oct 30 06:56:12.181068 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.181036    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 06:56:12.184147 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.184123    2580 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 06:56:12.184630 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.184604    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-5p67n","openshift-network-operator/iptables-alerter-tq7pf","openshift-ovn-kubernetes/ovnkube-node-r9jjx","openshift-image-registry/node-ca-l9mfn","openshift-multus/multus-9gqrp","openshift-network-diagnostics/network-check-target-c6trh","kube-system/konnectivity-agent-wv8tb","kube-system/global-pull-secret-syncer-qktbg","openshift-cluster-node-tuning-operator/tuned-4kfdw","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-125.ec2.internal","openshift-multus/multus-additional-cni-plugins-4js5r","openshift-multus/network-metrics-daemon-r4d72","kube-system/kube-apiserver-proxy-ip-10-0-128-125.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8kcn"]
Oct 30 06:56:12.187391 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.187363    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wv8tb"
Oct 30 06:56:12.190398 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.189921    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4js5r"
Oct 30 06:56:12.192467 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.192440    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:12.196018 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.195996    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-l9mfn"
Oct 30 06:56:12.198432 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.198386    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9gqrp"
Oct 30 06:56:12.201015 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.200714    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6trh"
Oct 30 06:56:12.201015 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:12.200789    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6trh" podUID="6d493f94-3aef-445e-9b60-32a10c184eac"
Oct 30 06:56:12.202250 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.202207    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 06:56:12.202250 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.202241    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 06:56:12.202464 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.202424    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-f2h44\""
Oct 30 06:56:12.204051 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.203469    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:56:12.206826 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.206436    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8kcn"
Oct 30 06:56:12.207122 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.207069    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8vngf\""
Oct 30 06:56:12.207122 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.207090    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6h5pm\""
Oct 30 06:56:12.207122 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.207113    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 06:56:12.207322 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.207201    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 06:56:12.207322 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.207308    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 06:56:12.207436 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.207372    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 06:56:12.207770 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.207748    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 06:56:12.207770 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.207764    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 06:56:12.207927 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.207833    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 06:56:12.207927 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.207911    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 06:56:12.208041 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.207971    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 06:56:12.208041 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.207973    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 06:56:12.208110 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.208072    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 06:56:12.209213 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.209197    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tq7pf"
Oct 30 06:56:12.210044 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.210013    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 06:56:12.211360 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.211339    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qktbg"
Oct 30 06:56:12.211480 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:12.211444    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qktbg" podUID="9092e2e8-ad23-426c-b6f1-14e1a2b9221e"
Oct 30 06:56:12.212676 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.212252    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-44sjc\""
Oct 30 06:56:12.212676 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.212336    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 06:56:12.212676 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.212551    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 06:56:12.213509 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.213490    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 06:56:12.213637 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.213620    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-s8tns\""
Oct 30 06:56:12.215121 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.215096    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5p67n"
Oct 30 06:56:12.218186 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.218158    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4d72"
Oct 30 06:56:12.218290 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:12.218242    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4d72" podUID="6f8b75af-cc86-49c6-a66c-1798309f9806"
Oct 30 06:56:12.230700 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.230678    2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 06:56:12.230818 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.230749    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-hnsc4"
Oct 30 06:56:12.244894 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.244824    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-hnsc4"
Oct 30 06:56:12.281403 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.281375    2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 06:56:12.290512 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.290487    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-host-var-lib-cni-bin\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:56:12.290663 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.290520    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe292d5d-6d46-4c34-ab24-7991571cd680-var-lib-kubelet\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:56:12.290663 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.290546    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe292d5d-6d46-4c34-ab24-7991571cd680-tmp\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:56:12.290663 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.290616    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvdd8\" (UniqueName: \"kubernetes.io/projected/851fb6ff-4a58-431b-834f-db3f87b67aa9-kube-api-access-hvdd8\") pod \"iptables-alerter-tq7pf\" (UID: \"851fb6ff-4a58-431b-834f-db3f87b67aa9\") " pod="openshift-network-operator/iptables-alerter-tq7pf"
Oct 30 06:56:12.290663 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.290645    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-log-socket\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:12.290809 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.290671    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/675dc1eb-ff05-40c7-bf0b-fb3161d3cc8c-tmp-dir\") pod \"node-resolver-5p67n\" (UID: \"675dc1eb-ff05-40c7-bf0b-fb3161d3cc8c\") " pod="openshift-dns/node-resolver-5p67n"
Oct 30 06:56:12.290809 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.290715    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-host-run-multus-certs\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:56:12.290809 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.290766    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ca422300-6212-4a01-b1e2-5ba04af79ce6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4js5r\" (UID: \"ca422300-6212-4a01-b1e2-5ba04af79ce6\") " pod="openshift-multus/multus-additional-cni-plugins-4js5r"
Oct 30 06:56:12.290949 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.290805    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-systemd-units\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:12.290949 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.290854    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-host-cni-netd\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:12.290949 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.290897    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-multus-conf-dir\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:56:12.290949 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.290923    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fe292d5d-6d46-4c34-ab24-7991571cd680-etc-modprobe-d\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:56:12.291130 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.290976    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-run-systemd\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:12.291130 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.291006    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d87089b-8b1b-43ea-be1f-72bfb56bf311-host\") pod \"node-ca-l9mfn\" (UID: \"5d87089b-8b1b-43ea-be1f-72bfb56bf311\") " pod="openshift-image-registry/node-ca-l9mfn"
Oct 30 06:56:12.291130 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.291033    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlfr5\" (UniqueName: \"kubernetes.io/projected/5d87089b-8b1b-43ea-be1f-72bfb56bf311-kube-api-access-tlfr5\") pod \"node-ca-l9mfn\" (UID: \"5d87089b-8b1b-43ea-be1f-72bfb56bf311\") " pod="openshift-image-registry/node-ca-l9mfn"
Oct 30 06:56:12.291130 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.291058    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-etc-kubernetes\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:56:12.291130 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.291104    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fe292d5d-6d46-4c34-ab24-7991571cd680-etc-systemd\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:56:12.291376 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.291140    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/851fb6ff-4a58-431b-834f-db3f87b67aa9-iptables-alerter-script\") pod \"iptables-alerter-tq7pf\" (UID: \"851fb6ff-4a58-431b-834f-db3f87b67aa9\") " pod="openshift-network-operator/iptables-alerter-tq7pf"
Oct 30 06:56:12.291376 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.291169    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/851fb6ff-4a58-431b-834f-db3f87b67aa9-host-slash\") pod \"iptables-alerter-tq7pf\" (UID: \"851fb6ff-4a58-431b-834f-db3f87b67aa9\") " pod="openshift-network-operator/iptables-alerter-tq7pf"
Oct 30 06:56:12.291376 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.291197    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:12.291376 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.291225    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-multus-cni-dir\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:56:12.291376 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.291249    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fe292d5d-6d46-4c34-ab24-7991571cd680-sys\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:56:12.291376 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.291288    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmzfj\" (UniqueName: \"kubernetes.io/projected/ca422300-6212-4a01-b1e2-5ba04af79ce6-kube-api-access-lmzfj\") pod \"multus-additional-cni-plugins-4js5r\" (UID: \"ca422300-6212-4a01-b1e2-5ba04af79ce6\") " pod="openshift-multus/multus-additional-cni-plugins-4js5r"
Oct 30 06:56:12.291376 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.291315    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a6968152-3c44-4205-a078-f857c413ad91-registration-dir\") pod \"aws-ebs-csi-driver-node-b8kcn\" (UID: \"a6968152-3c44-4205-a078-f857c413ad91\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8kcn"
Oct 30 06:56:12.291376 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.291340    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a6968152-3c44-4205-a078-f857c413ad91-etc-selinux\") pod \"aws-ebs-csi-driver-node-b8kcn\" (UID: \"a6968152-3c44-4205-a078-f857c413ad91\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8kcn"
Oct 30 06:56:12.291732 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.291376    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-host-var-lib-cni-multus\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:56:12.291732 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.291471    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fe292d5d-6d46-4c34-ab24-7991571cd680-etc-sysconfig\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:56:12.291732 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.291515    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-run-openvswitch\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:12.291732 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.291559    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b321b279-a102-48b9-bd9b-8ed938eed0f7-env-overrides\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:12.291732 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.291596    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/31519c22-050e-435c-aec3-7be0b68ff313-agent-certs\") pod \"konnectivity-agent-wv8tb\" (UID: \"31519c22-050e-435c-aec3-7be0b68ff313\") " pod="kube-system/konnectivity-agent-wv8tb"
Oct 30 06:56:12.291732 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.291613    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ca422300-6212-4a01-b1e2-5ba04af79ce6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4js5r\" (UID: \"ca422300-6212-4a01-b1e2-5ba04af79ce6\") " pod="openshift-multus/multus-additional-cni-plugins-4js5r"
Oct 30 06:56:12.291732 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.291667    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-cnibin\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:56:12.291732 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.291696    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-multus-socket-dir-parent\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:56:12.291732 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.291721    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fe292d5d-6d46-4c34-ab24-7991571cd680-run\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:56:12.292062 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.291757    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmkz2\" (UniqueName: \"kubernetes.io/projected/fe292d5d-6d46-4c34-ab24-7991571cd680-kube-api-access-bmkz2\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:56:12.292062 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.291799    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9092e2e8-ad23-426c-b6f1-14e1a2b9221e-kubelet-config\") pod \"global-pull-secret-syncer-qktbg\" (UID: \"9092e2e8-ad23-426c-b6f1-14e1a2b9221e\") " pod="kube-system/global-pull-secret-syncer-qktbg"
Oct 30 06:56:12.292062 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.291858    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-run-ovn\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:12.292062 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.291889    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-node-log\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:12.292062 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.291924    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b321b279-a102-48b9-bd9b-8ed938eed0f7-ovnkube-config\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:12.292062 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.291967    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ca422300-6212-4a01-b1e2-5ba04af79ce6-cni-binary-copy\") pod \"multus-additional-cni-plugins-4js5r\" (UID: \"ca422300-6212-4a01-b1e2-5ba04af79ce6\") " pod="openshift-multus/multus-additional-cni-plugins-4js5r"
Oct 30 06:56:12.292062 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.291993    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-host-slash\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:12.292062 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.292031    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-host-run-netns\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:12.292062 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.292051    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cwrb\" (UniqueName: \"kubernetes.io/projected/6d493f94-3aef-445e-9b60-32a10c184eac-kube-api-access-2cwrb\") pod \"network-check-target-c6trh\" (UID: \"6d493f94-3aef-445e-9b60-32a10c184eac\") " pod="openshift-network-diagnostics/network-check-target-c6trh"
Oct 30 06:56:12.292397 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.292075    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkb8j\" (UniqueName: \"kubernetes.io/projected/a6968152-3c44-4205-a078-f857c413ad91-kube-api-access-rkb8j\") pod \"aws-ebs-csi-driver-node-b8kcn\" (UID: \"a6968152-3c44-4205-a078-f857c413ad91\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8kcn"
Oct 30 06:56:12.292397 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.292101    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b74sr\" (UniqueName: \"kubernetes.io/projected/675dc1eb-ff05-40c7-bf0b-fb3161d3cc8c-kube-api-access-b74sr\") pod \"node-resolver-5p67n\" (UID: \"675dc1eb-ff05-40c7-bf0b-fb3161d3cc8c\") " pod="openshift-dns/node-resolver-5p67n"
Oct 30 06:56:12.292397 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.292120    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-host-run-netns\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:56:12.292397 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.292167    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fe292d5d-6d46-4c34-ab24-7991571cd680-etc-sysctl-d\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:56:12.292397 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.292192    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b321b279-a102-48b9-bd9b-8ed938eed0f7-ovn-node-metrics-cert\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:12.292397 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.292210    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kds7\" (UniqueName: \"kubernetes.io/projected/b321b279-a102-48b9-bd9b-8ed938eed0f7-kube-api-access-8kds7\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:12.292397 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.292235    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/31519c22-050e-435c-aec3-7be0b68ff313-konnectivity-ca\") pod \"konnectivity-agent-wv8tb\" (UID: \"31519c22-050e-435c-aec3-7be0b68ff313\") " pod="kube-system/konnectivity-agent-wv8tb"
Oct 30 06:56:12.292397 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.292270    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fe292d5d-6d46-4c34-ab24-7991571cd680-etc-sysctl-conf\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:56:12.292397 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.292287    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-hostroot\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:56:12.292397 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.292304    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9092e2e8-ad23-426c-b6f1-14e1a2b9221e-dbus\") pod \"global-pull-secret-syncer-qktbg\" (UID: \"9092e2e8-ad23-426c-b6f1-14e1a2b9221e\") " pod="kube-system/global-pull-secret-syncer-qktbg"
Oct 30 06:56:12.292397 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.292324    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-host-cni-bin\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:12.292397 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.292357    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a6968152-3c44-4205-a078-f857c413ad91-socket-dir\") pod \"aws-ebs-csi-driver-node-b8kcn\" (UID: \"a6968152-3c44-4205-a078-f857c413ad91\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8kcn"
Oct 30 06:56:12.292821 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.292418    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmzmm\" (UniqueName: \"kubernetes.io/projected/6f8b75af-cc86-49c6-a66c-1798309f9806-kube-api-access-lmzmm\") pod \"network-metrics-daemon-r4d72\" (UID: \"6f8b75af-cc86-49c6-a66c-1798309f9806\") " pod="openshift-multus/network-metrics-daemon-r4d72"
Oct 30 06:56:12.292821 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.292439    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-system-cni-dir\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:56:12.292821 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.292459    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-host-run-k8s-cni-cncf-io\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:56:12.292821 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.292480    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a6968152-3c44-4205-a078-f857c413ad91-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b8kcn\" (UID: \"a6968152-3c44-4205-a078-f857c413ad91\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8kcn"
Oct 30 06:56:12.292821 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.292504    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5d87089b-8b1b-43ea-be1f-72bfb56bf311-serviceca\") pod \"node-ca-l9mfn\" (UID: \"5d87089b-8b1b-43ea-be1f-72bfb56bf311\") " pod="openshift-image-registry/node-ca-l9mfn"
Oct 30 06:56:12.292821 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.292528    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-host-var-lib-kubelet\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:56:12.292821 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.292551    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-host-run-ovn-kubernetes\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:12.292821 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.292577    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-etc-openvswitch\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:12.292821 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.292600    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2368101c-6719-4650-8393-ef048500a624-multus-daemon-config\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:56:12.292821 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.292624    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dklkl\" (UniqueName: \"kubernetes.io/projected/2368101c-6719-4650-8393-ef048500a624-kube-api-access-dklkl\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:56:12.292821 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.292640    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe292d5d-6d46-4c34-ab24-7991571cd680-etc-kubernetes\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:56:12.292821 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.292654    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fe292d5d-6d46-4c34-ab24-7991571cd680-lib-modules\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:56:12.292821 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.292669    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fe292d5d-6d46-4c34-ab24-7991571cd680-etc-tuned\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:56:12.292821 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.292691    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ca422300-6212-4a01-b1e2-5ba04af79ce6-cnibin\") pod \"multus-additional-cni-plugins-4js5r\" (UID: \"ca422300-6212-4a01-b1e2-5ba04af79ce6\") " pod="openshift-multus/multus-additional-cni-plugins-4js5r"
Oct 30 06:56:12.292821 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.292716    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f8b75af-cc86-49c6-a66c-1798309f9806-metrics-certs\") pod \"network-metrics-daemon-r4d72\" (UID: \"6f8b75af-cc86-49c6-a66c-1798309f9806\") " pod="openshift-multus/network-metrics-daemon-r4d72"
Oct 30 06:56:12.292821 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.292736    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9092e2e8-ad23-426c-b6f1-14e1a2b9221e-original-pull-secret\") pod \"global-pull-secret-syncer-qktbg\" (UID: \"9092e2e8-ad23-426c-b6f1-14e1a2b9221e\") " pod="kube-system/global-pull-secret-syncer-qktbg"
Oct 30 06:56:12.293538 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.292758    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ca422300-6212-4a01-b1e2-5ba04af79ce6-system-cni-dir\") pod \"multus-additional-cni-plugins-4js5r\" (UID: \"ca422300-6212-4a01-b1e2-5ba04af79ce6\") " pod="openshift-multus/multus-additional-cni-plugins-4js5r"
Oct 30 06:56:12.293538 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.292786    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2368101c-6719-4650-8393-ef048500a624-cni-binary-copy\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:56:12.293538 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.292867    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a6968152-3c44-4205-a078-f857c413ad91-device-dir\") pod \"aws-ebs-csi-driver-node-b8kcn\" (UID: \"a6968152-3c44-4205-a078-f857c413ad91\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8kcn"
Oct 30 06:56:12.293538 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.292907    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a6968152-3c44-4205-a078-f857c413ad91-sys-fs\") pod \"aws-ebs-csi-driver-node-b8kcn\" (UID: \"a6968152-3c44-4205-a078-f857c413ad91\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8kcn"
Oct 30 06:56:12.293538 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.292939    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fe292d5d-6d46-4c34-ab24-7991571cd680-host\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:56:12.293538 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.292986    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ca422300-6212-4a01-b1e2-5ba04af79ce6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4js5r\" (UID: \"ca422300-6212-4a01-b1e2-5ba04af79ce6\") " pod="openshift-multus/multus-additional-cni-plugins-4js5r"
Oct 30 06:56:12.293538 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.293036    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-host-kubelet\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:12.293538 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.293090    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-var-lib-openvswitch\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:12.293538 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.293131    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b321b279-a102-48b9-bd9b-8ed938eed0f7-ovnkube-script-lib\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:12.293538 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.293159    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/675dc1eb-ff05-40c7-bf0b-fb3161d3cc8c-hosts-file\") pod \"node-resolver-5p67n\" (UID: \"675dc1eb-ff05-40c7-bf0b-fb3161d3cc8c\") " pod="openshift-dns/node-resolver-5p67n"
Oct 30 06:56:12.311391 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.311355    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 06:56:12.311534 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.311393    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 06:56:12.311600 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.311556    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 06:56:12.311600 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.311567    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 06:56:12.311600 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.311567    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 06:56:12.311766 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.311613    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-q4jsx\""
Oct 30 06:56:12.311766 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.311638    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 06:56:12.311766 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.311666    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 06:56:12.311766 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.311695    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-th56t\""
Oct 30 06:56:12.311766 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.311697    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 06:56:12.311766 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.311720    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 06:56:12.311766 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.311756    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2tccl\""
Oct 30 06:56:12.312081 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.311788    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 06:56:12.312081 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.311877    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7szgh\""
Oct 30 06:56:12.378395 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.378364    2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 06:56:12.394281 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.394245    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-host-run-ovn-kubernetes\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:12.394464 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.394290    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-etc-openvswitch\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:12.394464 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.394324    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2368101c-6719-4650-8393-ef048500a624-multus-daemon-config\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:56:12.394464 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.394384    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-host-run-ovn-kubernetes\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:12.394464 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.394385    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-etc-openvswitch\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:12.394464 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.394433    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dklkl\" (UniqueName: \"kubernetes.io/projected/2368101c-6719-4650-8393-ef048500a624-kube-api-access-dklkl\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:56:12.394464 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.394463    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe292d5d-6d46-4c34-ab24-7991571cd680-etc-kubernetes\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:56:12.394687 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.394485    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fe292d5d-6d46-4c34-ab24-7991571cd680-lib-modules\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:56:12.394687 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.394507    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fe292d5d-6d46-4c34-ab24-7991571cd680-etc-tuned\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:56:12.394687 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.394532    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ca422300-6212-4a01-b1e2-5ba04af79ce6-cnibin\") pod \"multus-additional-cni-plugins-4js5r\" (UID: \"ca422300-6212-4a01-b1e2-5ba04af79ce6\") " pod="openshift-multus/multus-additional-cni-plugins-4js5r"
Oct 30 06:56:12.394687 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.394570    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f8b75af-cc86-49c6-a66c-1798309f9806-metrics-certs\") pod \"network-metrics-daemon-r4d72\" (UID: \"6f8b75af-cc86-49c6-a66c-1798309f9806\") " pod="openshift-multus/network-metrics-daemon-r4d72"
Oct 30 06:56:12.394687 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.394596    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9092e2e8-ad23-426c-b6f1-14e1a2b9221e-original-pull-secret\") pod \"global-pull-secret-syncer-qktbg\" (UID: \"9092e2e8-ad23-426c-b6f1-14e1a2b9221e\") " pod="kube-system/global-pull-secret-syncer-qktbg"
Oct 30 06:56:12.394687 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.394624    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ca422300-6212-4a01-b1e2-5ba04af79ce6-system-cni-dir\") pod \"multus-additional-cni-plugins-4js5r\" (UID: \"ca422300-6212-4a01-b1e2-5ba04af79ce6\") " pod="openshift-multus/multus-additional-cni-plugins-4js5r"
Oct 30 06:56:12.394687 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.394635    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe292d5d-6d46-4c34-ab24-7991571cd680-etc-kubernetes\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:56:12.394687 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.394664    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2368101c-6719-4650-8393-ef048500a624-cni-binary-copy\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:56:12.395074 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.394693    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a6968152-3c44-4205-a078-f857c413ad91-device-dir\") pod \"aws-ebs-csi-driver-node-b8kcn\" (UID: \"a6968152-3c44-4205-a078-f857c413ad91\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8kcn"
Oct 30 06:56:12.395074 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.394726    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a6968152-3c44-4205-a078-f857c413ad91-sys-fs\") pod \"aws-ebs-csi-driver-node-b8kcn\" (UID: \"a6968152-3c44-4205-a078-f857c413ad91\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8kcn"
Oct 30 06:56:12.395074 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.394751    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fe292d5d-6d46-4c34-ab24-7991571cd680-host\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:56:12.395074 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.394783    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ca422300-6212-4a01-b1e2-5ba04af79ce6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4js5r\" (UID: \"ca422300-6212-4a01-b1e2-5ba04af79ce6\") " pod="openshift-multus/multus-additional-cni-plugins-4js5r"
Oct 30 06:56:12.395074 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.394813    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-host-kubelet\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:12.395074 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.394839    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-var-lib-openvswitch\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:12.395074 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.394887    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b321b279-a102-48b9-bd9b-8ed938eed0f7-ovnkube-script-lib\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:12.395074 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.394914    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/675dc1eb-ff05-40c7-bf0b-fb3161d3cc8c-hosts-file\") pod \"node-resolver-5p67n\" (UID: \"675dc1eb-ff05-40c7-bf0b-fb3161d3cc8c\") " pod="openshift-dns/node-resolver-5p67n"
Oct 30 06:56:12.395074 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.394939    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-host-var-lib-cni-bin\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:56:12.395074 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.394965    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe292d5d-6d46-4c34-ab24-7991571cd680-var-lib-kubelet\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:56:12.395074 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.395001    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe292d5d-6d46-4c34-ab24-7991571cd680-tmp\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:56:12.395074 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.395026    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a6968152-3c44-4205-a078-f857c413ad91-device-dir\") pod \"aws-ebs-csi-driver-node-b8kcn\" (UID: \"a6968152-3c44-4205-a078-f857c413ad91\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8kcn"
Oct 30 06:56:12.395074 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.395027    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvdd8\" (UniqueName: \"kubernetes.io/projected/851fb6ff-4a58-431b-834f-db3f87b67aa9-kube-api-access-hvdd8\") pod \"iptables-alerter-tq7pf\" (UID: \"851fb6ff-4a58-431b-834f-db3f87b67aa9\") " pod="openshift-network-operator/iptables-alerter-tq7pf"
Oct 30 06:56:12.395074 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.395046    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2368101c-6719-4650-8393-ef048500a624-multus-daemon-config\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:56:12.395074 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.395058    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-log-socket\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:12.395074 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.395081    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/675dc1eb-ff05-40c7-bf0b-fb3161d3cc8c-tmp-dir\") pod \"node-resolver-5p67n\" (UID: \"675dc1eb-ff05-40c7-bf0b-fb3161d3cc8c\") " pod="openshift-dns/node-resolver-5p67n"
Oct 30 06:56:12.395580 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.395108    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-host-run-multus-certs\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:56:12.395580 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.395120    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fe292d5d-6d46-4c34-ab24-7991571cd680-lib-modules\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:56:12.395580 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.395134    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ca422300-6212-4a01-b1e2-5ba04af79ce6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4js5r\" (UID: \"ca422300-6212-4a01-b1e2-5ba04af79ce6\") " pod="openshift-multus/multus-additional-cni-plugins-4js5r"
Oct 30 06:56:12.395580 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.395165    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-systemd-units\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:12.395580 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:12.395186    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 06:56:12.395580 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.395218    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-host-cni-netd\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:12.395580 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:12.395240    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 06:56:12.395580 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.395250    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-multus-conf-dir\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:56:12.395580 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:12.395269    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f8b75af-cc86-49c6-a66c-1798309f9806-metrics-certs podName:6f8b75af-cc86-49c6-a66c-1798309f9806 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:12.895247332 +0000 UTC m=+3.082561105 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6f8b75af-cc86-49c6-a66c-1798309f9806-metrics-certs") pod "network-metrics-daemon-r4d72" (UID: "6f8b75af-cc86-49c6-a66c-1798309f9806") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 06:56:12.395580 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:12.395306    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9092e2e8-ad23-426c-b6f1-14e1a2b9221e-original-pull-secret podName:9092e2e8-ad23-426c-b6f1-14e1a2b9221e nodeName:}" failed. No retries permitted until 2026-10-30 06:56:12.895289872 +0000 UTC m=+3.082603642 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9092e2e8-ad23-426c-b6f1-14e1a2b9221e-original-pull-secret") pod "global-pull-secret-syncer-qktbg" (UID: "9092e2e8-ad23-426c-b6f1-14e1a2b9221e") : object "kube-system"/"original-pull-secret" not registered
Oct 30 06:56:12.395580 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.395324    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fe292d5d-6d46-4c34-ab24-7991571cd680-etc-modprobe-d\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:56:12.395580 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.395351    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a6968152-3c44-4205-a078-f857c413ad91-sys-fs\") pod \"aws-ebs-csi-driver-node-b8kcn\" (UID: \"a6968152-3c44-4205-a078-f857c413ad91\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8kcn"
Oct 30 06:56:12.395580 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.395366    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-run-systemd\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:12.395580 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.395395    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d87089b-8b1b-43ea-be1f-72bfb56bf311-host\") pod \"node-ca-l9mfn\" (UID: \"5d87089b-8b1b-43ea-be1f-72bfb56bf311\") " pod="openshift-image-registry/node-ca-l9mfn"
Oct 30 06:56:12.395580 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.395411    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fe292d5d-6d46-4c34-ab24-7991571cd680-host\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:56:12.395580 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.395428    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlfr5\" (UniqueName: \"kubernetes.io/projected/5d87089b-8b1b-43ea-be1f-72bfb56bf311-kube-api-access-tlfr5\") pod \"node-ca-l9mfn\" (UID: \"5d87089b-8b1b-43ea-be1f-72bfb56bf311\") " pod="openshift-image-registry/node-ca-l9mfn"
Oct 30 06:56:12.395580 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.395450    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-etc-kubernetes\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:56:12.396186 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.395466    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fe292d5d-6d46-4c34-ab24-7991571cd680-etc-systemd\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:56:12.396186 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.395482    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/851fb6ff-4a58-431b-834f-db3f87b67aa9-iptables-alerter-script\") pod \"iptables-alerter-tq7pf\" (UID: \"851fb6ff-4a58-431b-834f-db3f87b67aa9\") " pod="openshift-network-operator/iptables-alerter-tq7pf"
Oct 30 06:56:12.396186 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.395498    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/851fb6ff-4a58-431b-834f-db3f87b67aa9-host-slash\") pod \"iptables-alerter-tq7pf\" (UID: \"851fb6ff-4a58-431b-834f-db3f87b67aa9\") " pod="openshift-network-operator/iptables-alerter-tq7pf"
Oct 30 06:56:12.396186 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.395518    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:12.396186 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.395547    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-multus-cni-dir\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:56:12.396186 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.395549    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ca422300-6212-4a01-b1e2-5ba04af79ce6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4js5r\" (UID: \"ca422300-6212-4a01-b1e2-5ba04af79ce6\") " pod="openshift-multus/multus-additional-cni-plugins-4js5r"
Oct 30 06:56:12.396186 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.395564    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fe292d5d-6d46-4c34-ab24-7991571cd680-sys\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:56:12.396186 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.395581    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmzfj\" (UniqueName: \"kubernetes.io/projected/ca422300-6212-4a01-b1e2-5ba04af79ce6-kube-api-access-lmzfj\") pod \"multus-additional-cni-plugins-4js5r\" (UID: \"ca422300-6212-4a01-b1e2-5ba04af79ce6\") " pod="openshift-multus/multus-additional-cni-plugins-4js5r"
Oct 30 06:56:12.396186 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.395597    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a6968152-3c44-4205-a078-f857c413ad91-registration-dir\") pod \"aws-ebs-csi-driver-node-b8kcn\" (UID: \"a6968152-3c44-4205-a078-f857c413ad91\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8kcn"
Oct 30 06:56:12.396186 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.395635    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a6968152-3c44-4205-a078-f857c413ad91-etc-selinux\") pod \"aws-ebs-csi-driver-node-b8kcn\" (UID: \"a6968152-3c44-4205-a078-f857c413ad91\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8kcn"
Oct 30 06:56:12.396186 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.395633    2580 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 06:56:12.396186 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.395694    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-host-var-lib-cni-multus\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:56:12.396186 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.395706    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fe292d5d-6d46-4c34-ab24-7991571cd680-etc-systemd\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:56:12.396186 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.395659    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-host-var-lib-cni-multus\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:56:12.396186 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.395659    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-etc-kubernetes\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:56:12.396186 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.395303    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-multus-conf-dir\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:56:12.396186 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.395760    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-run-systemd\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:12.396186 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.395650    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-var-lib-openvswitch\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:12.396911 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.396090    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fe292d5d-6d46-4c34-ab24-7991571cd680-etc-sysconfig\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:56:12.396911 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.395595    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-host-kubelet\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:12.396911 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.396201    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d87089b-8b1b-43ea-be1f-72bfb56bf311-host\") pod \"node-ca-l9mfn\" (UID: \"5d87089b-8b1b-43ea-be1f-72bfb56bf311\") " pod="openshift-image-registry/node-ca-l9mfn"
Oct 30 06:56:12.396911 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.396280    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/851fb6ff-4a58-431b-834f-db3f87b67aa9-iptables-alerter-script\") pod \"iptables-alerter-tq7pf\" (UID: \"851fb6ff-4a58-431b-834f-db3f87b67aa9\") " pod="openshift-network-operator/iptables-alerter-tq7pf"
Oct 30 06:56:12.396911 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.396306    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b321b279-a102-48b9-bd9b-8ed938eed0f7-ovnkube-script-lib\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:12.396911 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.396358    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ca422300-6212-4a01-b1e2-5ba04af79ce6-system-cni-dir\") pod \"multus-additional-cni-plugins-4js5r\" (UID: \"ca422300-6212-4a01-b1e2-5ba04af79ce6\") " pod="openshift-multus/multus-additional-cni-plugins-4js5r"
Oct 30 06:56:12.396911 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.396409    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ca422300-6212-4a01-b1e2-5ba04af79ce6-cnibin\") pod \"multus-additional-cni-plugins-4js5r\" (UID: \"ca422300-6212-4a01-b1e2-5ba04af79ce6\") " pod="openshift-multus/multus-additional-cni-plugins-4js5r"
Oct 30 06:56:12.396911 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.396399    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe292d5d-6d46-4c34-ab24-7991571cd680-var-lib-kubelet\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:56:12.396911 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.396445    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a6968152-3c44-4205-a078-f857c413ad91-registration-dir\") pod \"aws-ebs-csi-driver-node-b8kcn\" (UID: \"a6968152-3c44-4205-a078-f857c413ad91\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8kcn"
Oct 30 06:56:12.396911 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.396451    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fe292d5d-6d46-4c34-ab24-7991571cd680-sys\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:56:12.396911 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.396479    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/851fb6ff-4a58-431b-834f-db3f87b67aa9-host-slash\") pod \"iptables-alerter-tq7pf\" (UID: \"851fb6ff-4a58-431b-834f-db3f87b67aa9\") " pod="openshift-network-operator/iptables-alerter-tq7pf"
Oct 30 06:56:12.396911 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.396502    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:12.396911 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.396576    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2368101c-6719-4650-8393-ef048500a624-cni-binary-copy\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:56:12.396911 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.396588    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-multus-cni-dir\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:56:12.396911 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.396631    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-systemd-units\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:12.396911 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.396632    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-log-socket\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:12.396911 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.396685    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/675dc1eb-ff05-40c7-bf0b-fb3161d3cc8c-hosts-file\") pod \"node-resolver-5p67n\" (UID: \"675dc1eb-ff05-40c7-bf0b-fb3161d3cc8c\") " pod="openshift-dns/node-resolver-5p67n"
Oct 30 06:56:12.396911 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.396710    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-host-var-lib-cni-bin\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:56:12.397674 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.396876    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-host-run-multus-certs\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:56:12.397674 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.396948    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fe292d5d-6d46-4c34-ab24-7991571cd680-etc-sysconfig\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:56:12.397674 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.396966    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/675dc1eb-ff05-40c7-bf0b-fb3161d3cc8c-tmp-dir\") pod \"node-resolver-5p67n\" (UID: \"675dc1eb-ff05-40c7-bf0b-fb3161d3cc8c\") " pod="openshift-dns/node-resolver-5p67n"
Oct 30 06:56:12.397674 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.396979    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-host-cni-netd\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:12.397674 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.397010    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-run-openvswitch\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:12.397674 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.397036    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b321b279-a102-48b9-bd9b-8ed938eed0f7-env-overrides\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:12.397674 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.397085    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fe292d5d-6d46-4c34-ab24-7991571cd680-etc-modprobe-d\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:56:12.397674 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.397130    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-run-openvswitch\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:12.397674 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.397171    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/31519c22-050e-435c-aec3-7be0b68ff313-agent-certs\") pod \"konnectivity-agent-wv8tb\" (UID: \"31519c22-050e-435c-aec3-7be0b68ff313\") " pod="kube-system/konnectivity-agent-wv8tb"
Oct 30 06:56:12.397674 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.397167    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a6968152-3c44-4205-a078-f857c413ad91-etc-selinux\") pod \"aws-ebs-csi-driver-node-b8kcn\" (UID: \"a6968152-3c44-4205-a078-f857c413ad91\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8kcn"
Oct 30 06:56:12.397674 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.397220    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ca422300-6212-4a01-b1e2-5ba04af79ce6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4js5r\" (UID: \"ca422300-6212-4a01-b1e2-5ba04af79ce6\") " pod="openshift-multus/multus-additional-cni-plugins-4js5r"
Oct 30 06:56:12.397674 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.397353    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ca422300-6212-4a01-b1e2-5ba04af79ce6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4js5r\" (UID: \"ca422300-6212-4a01-b1e2-5ba04af79ce6\") " pod="openshift-multus/multus-additional-cni-plugins-4js5r"
Oct 30 06:56:12.397674 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.397410    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-cnibin\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:56:12.397674 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.397455    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-multus-socket-dir-parent\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:56:12.397674 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.397483    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fe292d5d-6d46-4c34-ab24-7991571cd680-run\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:56:12.397674 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.397520    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmkz2\" (UniqueName: \"kubernetes.io/projected/fe292d5d-6d46-4c34-ab24-7991571cd680-kube-api-access-bmkz2\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:56:12.397674 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.397548    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9092e2e8-ad23-426c-b6f1-14e1a2b9221e-kubelet-config\") pod \"global-pull-secret-syncer-qktbg\" (UID: \"9092e2e8-ad23-426c-b6f1-14e1a2b9221e\") " pod="kube-system/global-pull-secret-syncer-qktbg"
Oct 30 06:56:12.398534 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.397577    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-run-ovn\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:12.398534 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.397599    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-cnibin\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:56:12.398534 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.397624    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-node-log\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:12.398534 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.397651    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b321b279-a102-48b9-bd9b-8ed938eed0f7-ovnkube-config\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:12.398534 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.397664    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fe292d5d-6d46-4c34-ab24-7991571cd680-run\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:56:12.398534 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.397851    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ca422300-6212-4a01-b1e2-5ba04af79ce6-cni-binary-copy\") pod \"multus-additional-cni-plugins-4js5r\" (UID: \"ca422300-6212-4a01-b1e2-5ba04af79ce6\") " pod="openshift-multus/multus-additional-cni-plugins-4js5r"
Oct 30 06:56:12.398534 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.397889    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-host-slash\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:12.398534 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.397969    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-host-run-netns\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:12.398534 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.398017    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b321b279-a102-48b9-bd9b-8ed938eed0f7-env-overrides\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:12.398534 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.398026    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-host-run-netns\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:12.398534 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.398058    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-host-slash\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:12.398534 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.398089    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cwrb\" (UniqueName: \"kubernetes.io/projected/6d493f94-3aef-445e-9b60-32a10c184eac-kube-api-access-2cwrb\") pod \"network-check-target-c6trh\" (UID: \"6d493f94-3aef-445e-9b60-32a10c184eac\") " pod="openshift-network-diagnostics/network-check-target-c6trh"
Oct 30 06:56:12.398534 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.398121    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-node-log\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:12.398534 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.398136    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkb8j\" (UniqueName: \"kubernetes.io/projected/a6968152-3c44-4205-a078-f857c413ad91-kube-api-access-rkb8j\") pod \"aws-ebs-csi-driver-node-b8kcn\" (UID: \"a6968152-3c44-4205-a078-f857c413ad91\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8kcn"
Oct 30 06:56:12.398534 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.398237    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9092e2e8-ad23-426c-b6f1-14e1a2b9221e-kubelet-config\") pod \"global-pull-secret-syncer-qktbg\" (UID: \"9092e2e8-ad23-426c-b6f1-14e1a2b9221e\") " pod="kube-system/global-pull-secret-syncer-qktbg"
Oct 30 06:56:12.398534 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.398257    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-run-ovn\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:12.398534 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.398279    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b74sr\" (UniqueName: \"kubernetes.io/projected/675dc1eb-ff05-40c7-bf0b-fb3161d3cc8c-kube-api-access-b74sr\") pod \"node-resolver-5p67n\" (UID: \"675dc1eb-ff05-40c7-bf0b-fb3161d3cc8c\") " pod="openshift-dns/node-resolver-5p67n"
Oct 30 06:56:12.398534 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.398333    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-host-run-netns\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:56:12.399449 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.398411    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-host-run-netns\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:56:12.399449 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.398455    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fe292d5d-6d46-4c34-ab24-7991571cd680-etc-sysctl-d\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:56:12.399449 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.398480    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b321b279-a102-48b9-bd9b-8ed938eed0f7-ovnkube-config\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:12.399449 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.398491    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b321b279-a102-48b9-bd9b-8ed938eed0f7-ovn-node-metrics-cert\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:12.399449 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.398566    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kds7\" (UniqueName: \"kubernetes.io/projected/b321b279-a102-48b9-bd9b-8ed938eed0f7-kube-api-access-8kds7\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:12.399449 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.398581    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fe292d5d-6d46-4c34-ab24-7991571cd680-etc-sysctl-d\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:56:12.399449 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.398687    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/31519c22-050e-435c-aec3-7be0b68ff313-konnectivity-ca\") pod \"konnectivity-agent-wv8tb\" (UID: \"31519c22-050e-435c-aec3-7be0b68ff313\") " pod="kube-system/konnectivity-agent-wv8tb"
Oct 30 06:56:12.399449 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.398798    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ca422300-6212-4a01-b1e2-5ba04af79ce6-cni-binary-copy\") pod \"multus-additional-cni-plugins-4js5r\" (UID: \"ca422300-6212-4a01-b1e2-5ba04af79ce6\") " pod="openshift-multus/multus-additional-cni-plugins-4js5r"
Oct 30 06:56:12.399449 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.398855    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fe292d5d-6d46-4c34-ab24-7991571cd680-etc-sysctl-conf\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:56:12.399449 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.398902    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-hostroot\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:56:12.399449 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.398919    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ca422300-6212-4a01-b1e2-5ba04af79ce6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4js5r\" (UID: \"ca422300-6212-4a01-b1e2-5ba04af79ce6\") " pod="openshift-multus/multus-additional-cni-plugins-4js5r"
Oct 30 06:56:12.399449 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.398974    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-hostroot\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:56:12.399449 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.398972    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9092e2e8-ad23-426c-b6f1-14e1a2b9221e-dbus\") pod \"global-pull-secret-syncer-qktbg\" (UID: \"9092e2e8-ad23-426c-b6f1-14e1a2b9221e\") " pod="kube-system/global-pull-secret-syncer-qktbg"
Oct 30 06:56:12.399449 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.399017    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fe292d5d-6d46-4c34-ab24-7991571cd680-etc-sysctl-conf\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:56:12.399449 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.399027    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-host-cni-bin\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:12.399449 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.399034    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-multus-socket-dir-parent\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:56:12.399449 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.399040    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9092e2e8-ad23-426c-b6f1-14e1a2b9221e-dbus\") pod \"global-pull-secret-syncer-qktbg\" (UID: \"9092e2e8-ad23-426c-b6f1-14e1a2b9221e\") " pod="kube-system/global-pull-secret-syncer-qktbg"
Oct 30 06:56:12.399449 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.399069    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a6968152-3c44-4205-a078-f857c413ad91-socket-dir\") pod \"aws-ebs-csi-driver-node-b8kcn\" (UID: \"a6968152-3c44-4205-a078-f857c413ad91\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8kcn"
Oct 30 06:56:12.400334 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.399076    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-host-cni-bin\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:12.400334 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.399102    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmzmm\" (UniqueName: \"kubernetes.io/projected/6f8b75af-cc86-49c6-a66c-1798309f9806-kube-api-access-lmzmm\") pod \"network-metrics-daemon-r4d72\" (UID: \"6f8b75af-cc86-49c6-a66c-1798309f9806\") " pod="openshift-multus/network-metrics-daemon-r4d72"
Oct 30 06:56:12.400334 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.399139    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-system-cni-dir\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:56:12.400334 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.399247    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-host-run-k8s-cni-cncf-io\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:56:12.400334 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.399277    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a6968152-3c44-4205-a078-f857c413ad91-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b8kcn\" (UID: \"a6968152-3c44-4205-a078-f857c413ad91\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8kcn"
Oct 30 06:56:12.400334 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.399319    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5d87089b-8b1b-43ea-be1f-72bfb56bf311-serviceca\") pod \"node-ca-l9mfn\" (UID: \"5d87089b-8b1b-43ea-be1f-72bfb56bf311\") " pod="openshift-image-registry/node-ca-l9mfn"
Oct 30 06:56:12.400334 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.399345    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-host-var-lib-kubelet\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:56:12.400334 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.399359    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-system-cni-dir\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:56:12.400334 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.399435    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a6968152-3c44-4205-a078-f857c413ad91-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b8kcn\" (UID: \"a6968152-3c44-4205-a078-f857c413ad91\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8kcn"
Oct 30 06:56:12.400334 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.399471    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-host-var-lib-kubelet\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:56:12.400334 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.399512    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-host-run-k8s-cni-cncf-io\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:56:12.400334 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.400183    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a6968152-3c44-4205-a078-f857c413ad91-socket-dir\") pod \"aws-ebs-csi-driver-node-b8kcn\" (UID: \"a6968152-3c44-4205-a078-f857c413ad91\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8kcn"
Oct 30 06:56:12.400946 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.400481    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/31519c22-050e-435c-aec3-7be0b68ff313-konnectivity-ca\") pod \"konnectivity-agent-wv8tb\" (UID: \"31519c22-050e-435c-aec3-7be0b68ff313\") " pod="kube-system/konnectivity-agent-wv8tb"
Oct 30 06:56:12.401082 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.401058    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5d87089b-8b1b-43ea-be1f-72bfb56bf311-serviceca\") pod \"node-ca-l9mfn\" (UID: \"5d87089b-8b1b-43ea-be1f-72bfb56bf311\") " pod="openshift-image-registry/node-ca-l9mfn"
Oct 30 06:56:12.401164 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.401142    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/31519c22-050e-435c-aec3-7be0b68ff313-agent-certs\") pod \"konnectivity-agent-wv8tb\" (UID: \"31519c22-050e-435c-aec3-7be0b68ff313\") " pod="kube-system/konnectivity-agent-wv8tb"
Oct 30 06:56:12.401888 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.401862    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b321b279-a102-48b9-bd9b-8ed938eed0f7-ovn-node-metrics-cert\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:12.402068 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.402045    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fe292d5d-6d46-4c34-ab24-7991571cd680-etc-tuned\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:56:12.403204 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.403184    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe292d5d-6d46-4c34-ab24-7991571cd680-tmp\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:56:12.418211 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.418100    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dklkl\" (UniqueName: \"kubernetes.io/projected/2368101c-6719-4650-8393-ef048500a624-kube-api-access-dklkl\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:56:12.423271 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.423241    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlfr5\" (UniqueName: \"kubernetes.io/projected/5d87089b-8b1b-43ea-be1f-72bfb56bf311-kube-api-access-tlfr5\") pod \"node-ca-l9mfn\" (UID: \"5d87089b-8b1b-43ea-be1f-72bfb56bf311\") " pod="openshift-image-registry/node-ca-l9mfn"
Oct 30 06:56:12.424513 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.424482    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kds7\" (UniqueName: \"kubernetes.io/projected/b321b279-a102-48b9-bd9b-8ed938eed0f7-kube-api-access-8kds7\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:12.425440 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.425419    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvdd8\" (UniqueName: \"kubernetes.io/projected/851fb6ff-4a58-431b-834f-db3f87b67aa9-kube-api-access-hvdd8\") pod \"iptables-alerter-tq7pf\" (UID: \"851fb6ff-4a58-431b-834f-db3f87b67aa9\") " pod="openshift-network-operator/iptables-alerter-tq7pf"
Oct 30 06:56:12.430222 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.429407    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b74sr\" (UniqueName: \"kubernetes.io/projected/675dc1eb-ff05-40c7-bf0b-fb3161d3cc8c-kube-api-access-b74sr\") pod \"node-resolver-5p67n\" (UID: \"675dc1eb-ff05-40c7-bf0b-fb3161d3cc8c\") " pod="openshift-dns/node-resolver-5p67n"
Oct 30 06:56:12.430222 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.429502    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkb8j\" (UniqueName: \"kubernetes.io/projected/a6968152-3c44-4205-a078-f857c413ad91-kube-api-access-rkb8j\") pod \"aws-ebs-csi-driver-node-b8kcn\" (UID: \"a6968152-3c44-4205-a078-f857c413ad91\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8kcn"
Oct 30 06:56:12.430222 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:12.429639    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 06:56:12.430222 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:12.429684    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 06:56:12.430222 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:12.430193    2580 projected.go:196] Error preparing data for projected volume kube-api-access-2cwrb for pod openshift-network-diagnostics/network-check-target-c6trh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 06:56:12.430516 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:12.430300    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d493f94-3aef-445e-9b60-32a10c184eac-kube-api-access-2cwrb podName:6d493f94-3aef-445e-9b60-32a10c184eac nodeName:}" failed. No retries permitted until 2026-10-30 06:56:12.930270058 +0000 UTC m=+3.117583825 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2cwrb" (UniqueName: "kubernetes.io/projected/6d493f94-3aef-445e-9b60-32a10c184eac-kube-api-access-2cwrb") pod "network-check-target-c6trh" (UID: "6d493f94-3aef-445e-9b60-32a10c184eac") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 06:56:12.434394 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.434343    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmzfj\" (UniqueName: \"kubernetes.io/projected/ca422300-6212-4a01-b1e2-5ba04af79ce6-kube-api-access-lmzfj\") pod \"multus-additional-cni-plugins-4js5r\" (UID: \"ca422300-6212-4a01-b1e2-5ba04af79ce6\") " pod="openshift-multus/multus-additional-cni-plugins-4js5r"
Oct 30 06:56:12.434949 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.434887    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmkz2\" (UniqueName: \"kubernetes.io/projected/fe292d5d-6d46-4c34-ab24-7991571cd680-kube-api-access-bmkz2\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:56:12.435060 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.435038    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmzmm\" (UniqueName: \"kubernetes.io/projected/6f8b75af-cc86-49c6-a66c-1798309f9806-kube-api-access-lmzmm\") pod \"network-metrics-daemon-r4d72\" (UID: \"6f8b75af-cc86-49c6-a66c-1798309f9806\") " pod="openshift-multus/network-metrics-daemon-r4d72"
Oct 30 06:56:12.503844 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.503800    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wv8tb"
Oct 30 06:56:12.517853 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:12.517806    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31519c22_050e_435c_aec3_7be0b68ff313.slice/crio-b2f0d0628cf42a429a4f71ec6f40317fa67e7594d24d33bbfc74fd27438d50a9 WatchSource:0}: Error finding container b2f0d0628cf42a429a4f71ec6f40317fa67e7594d24d33bbfc74fd27438d50a9: Status 404 returned error can't find the container with id b2f0d0628cf42a429a4f71ec6f40317fa67e7594d24d33bbfc74fd27438d50a9
Oct 30 06:56:12.536549 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.536522    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-l9mfn"
Oct 30 06:56:12.542919 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.542890    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4js5r"
Oct 30 06:56:12.544039 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.544017    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9gqrp"
Oct 30 06:56:12.546925 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:12.546899    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d87089b_8b1b_43ea_be1f_72bfb56bf311.slice/crio-fa9ac36aa63c36f432ecb122126c429701322d5758b168a1026b9c3b6353db26 WatchSource:0}: Error finding container fa9ac36aa63c36f432ecb122126c429701322d5758b168a1026b9c3b6353db26: Status 404 returned error can't find the container with id fa9ac36aa63c36f432ecb122126c429701322d5758b168a1026b9c3b6353db26
Oct 30 06:56:12.551094 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.551071    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:56:12.553583 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:12.553552    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca422300_6212_4a01_b1e2_5ba04af79ce6.slice/crio-80a771c73af370f46a40ec02354dc73456551de82c38c22b9f57bbfd09a7c86a WatchSource:0}: Error finding container 80a771c73af370f46a40ec02354dc73456551de82c38c22b9f57bbfd09a7c86a: Status 404 returned error can't find the container with id 80a771c73af370f46a40ec02354dc73456551de82c38c22b9f57bbfd09a7c86a
Oct 30 06:56:12.559922 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:12.559895    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2368101c_6719_4650_8393_ef048500a624.slice/crio-e04753844ffdbcd2c1f467e3e188f856361230266cc9e241bb967283f893901c WatchSource:0}: Error finding container e04753844ffdbcd2c1f467e3e188f856361230266cc9e241bb967283f893901c: Status 404 returned error can't find the container with id e04753844ffdbcd2c1f467e3e188f856361230266cc9e241bb967283f893901c
Oct 30 06:56:12.563088 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.563065    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8kcn"
Oct 30 06:56:12.563358 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:12.563324    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe292d5d_6d46_4c34_ab24_7991571cd680.slice/crio-bc65bdcd581cfc4ddc866c5bf733ea06d05843df92b765587c18ee82df411d78 WatchSource:0}: Error finding container bc65bdcd581cfc4ddc866c5bf733ea06d05843df92b765587c18ee82df411d78: Status 404 returned error can't find the container with id bc65bdcd581cfc4ddc866c5bf733ea06d05843df92b765587c18ee82df411d78
Oct 30 06:56:12.565358 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.565339    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:12.567495 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.567473    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tq7pf"
Oct 30 06:56:12.570847 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.570826    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5p67n"
Oct 30 06:56:12.578517 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:12.578472    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6968152_3c44_4205_a078_f857c413ad91.slice/crio-943ee05bb25ae7fe4215fdaec4395add68669b2a4643aa0d476543979b4c28a6 WatchSource:0}: Error finding container 943ee05bb25ae7fe4215fdaec4395add68669b2a4643aa0d476543979b4c28a6: Status 404 returned error can't find the container with id 943ee05bb25ae7fe4215fdaec4395add68669b2a4643aa0d476543979b4c28a6
Oct 30 06:56:12.584858 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:12.584828    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb321b279_a102_48b9_bd9b_8ed938eed0f7.slice/crio-3a3410f94fd4c007d13abc13d7e3d4f70224c98d53ea499adc6ae46982b9eebf WatchSource:0}: Error finding container 3a3410f94fd4c007d13abc13d7e3d4f70224c98d53ea499adc6ae46982b9eebf: Status 404 returned error can't find the container with id 3a3410f94fd4c007d13abc13d7e3d4f70224c98d53ea499adc6ae46982b9eebf
Oct 30 06:56:12.589713 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:12.589686    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod851fb6ff_4a58_431b_834f_db3f87b67aa9.slice/crio-88c5fae3e58af8696d8b0aa5eba674d92d35383ce719079cc013d2034376d11a WatchSource:0}: Error finding container 88c5fae3e58af8696d8b0aa5eba674d92d35383ce719079cc013d2034376d11a: Status 404 returned error can't find the container with id 88c5fae3e58af8696d8b0aa5eba674d92d35383ce719079cc013d2034376d11a
Oct 30 06:56:12.593960 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:12.593922    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod675dc1eb_ff05_40c7_bf0b_fb3161d3cc8c.slice/crio-06224bc406875c768834b53973ff4ff559b88f05bd498fbe25fa9a80007bec57 WatchSource:0}: Error finding container 06224bc406875c768834b53973ff4ff559b88f05bd498fbe25fa9a80007bec57: Status 404 returned error can't find the container with id 06224bc406875c768834b53973ff4ff559b88f05bd498fbe25fa9a80007bec57
Oct 30 06:56:12.904736 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.904707    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f8b75af-cc86-49c6-a66c-1798309f9806-metrics-certs\") pod \"network-metrics-daemon-r4d72\" (UID: \"6f8b75af-cc86-49c6-a66c-1798309f9806\") " pod="openshift-multus/network-metrics-daemon-r4d72"
Oct 30 06:56:12.904853 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:12.904747    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9092e2e8-ad23-426c-b6f1-14e1a2b9221e-original-pull-secret\") pod \"global-pull-secret-syncer-qktbg\" (UID: \"9092e2e8-ad23-426c-b6f1-14e1a2b9221e\") " pod="kube-system/global-pull-secret-syncer-qktbg"
Oct 30 06:56:12.904944 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:12.904871    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 06:56:12.904944 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:12.904876    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 06:56:12.904944 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:12.904938    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f8b75af-cc86-49c6-a66c-1798309f9806-metrics-certs podName:6f8b75af-cc86-49c6-a66c-1798309f9806 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:13.904922448 +0000 UTC m=+4.092236212 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6f8b75af-cc86-49c6-a66c-1798309f9806-metrics-certs") pod "network-metrics-daemon-r4d72" (UID: "6f8b75af-cc86-49c6-a66c-1798309f9806") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 06:56:12.905059 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:12.904954    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9092e2e8-ad23-426c-b6f1-14e1a2b9221e-original-pull-secret podName:9092e2e8-ad23-426c-b6f1-14e1a2b9221e nodeName:}" failed. No retries permitted until 2026-10-30 06:56:13.904947462 +0000 UTC m=+4.092261210 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9092e2e8-ad23-426c-b6f1-14e1a2b9221e-original-pull-secret") pod "global-pull-secret-syncer-qktbg" (UID: "9092e2e8-ad23-426c-b6f1-14e1a2b9221e") : object "kube-system"/"original-pull-secret" not registered
Oct 30 06:56:13.006267 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:13.006205    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cwrb\" (UniqueName: \"kubernetes.io/projected/6d493f94-3aef-445e-9b60-32a10c184eac-kube-api-access-2cwrb\") pod \"network-check-target-c6trh\" (UID: \"6d493f94-3aef-445e-9b60-32a10c184eac\") " pod="openshift-network-diagnostics/network-check-target-c6trh"
Oct 30 06:56:13.006476 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:13.006378    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 06:56:13.006476 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:13.006401    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 06:56:13.006476 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:13.006416    2580 projected.go:196] Error preparing data for projected volume kube-api-access-2cwrb for pod openshift-network-diagnostics/network-check-target-c6trh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 06:56:13.006476 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:13.006476    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d493f94-3aef-445e-9b60-32a10c184eac-kube-api-access-2cwrb podName:6d493f94-3aef-445e-9b60-32a10c184eac nodeName:}" failed. No retries permitted until 2026-10-30 06:56:14.006456304 +0000 UTC m=+4.193770072 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2cwrb" (UniqueName: "kubernetes.io/projected/6d493f94-3aef-445e-9b60-32a10c184eac-kube-api-access-2cwrb") pod "network-check-target-c6trh" (UID: "6d493f94-3aef-445e-9b60-32a10c184eac") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 06:56:13.248252 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:13.247863    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 06:51:12 +0000 UTC" deadline="2028-07-15 16:06:22.758922672 +0000 UTC"
Oct 30 06:56:13.248252 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:13.248134    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14985h10m9.510794335s"
Oct 30 06:56:13.402828 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:13.402769    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tq7pf" event={"ID":"851fb6ff-4a58-431b-834f-db3f87b67aa9","Type":"ContainerStarted","Data":"88c5fae3e58af8696d8b0aa5eba674d92d35383ce719079cc013d2034376d11a"}
Oct 30 06:56:13.419176 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:13.419140    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9gqrp" event={"ID":"2368101c-6719-4650-8393-ef048500a624","Type":"ContainerStarted","Data":"e04753844ffdbcd2c1f467e3e188f856361230266cc9e241bb967283f893901c"}
Oct 30 06:56:13.426994 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:13.426963    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4js5r" event={"ID":"ca422300-6212-4a01-b1e2-5ba04af79ce6","Type":"ContainerStarted","Data":"80a771c73af370f46a40ec02354dc73456551de82c38c22b9f57bbfd09a7c86a"}
Oct 30 06:56:13.439691 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:13.439622    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wv8tb" event={"ID":"31519c22-050e-435c-aec3-7be0b68ff313","Type":"ContainerStarted","Data":"b2f0d0628cf42a429a4f71ec6f40317fa67e7594d24d33bbfc74fd27438d50a9"}
Oct 30 06:56:13.445595 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:13.445567    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx" event={"ID":"b321b279-a102-48b9-bd9b-8ed938eed0f7","Type":"ContainerStarted","Data":"3a3410f94fd4c007d13abc13d7e3d4f70224c98d53ea499adc6ae46982b9eebf"}
Oct 30 06:56:13.456028 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:13.452950    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8kcn" event={"ID":"a6968152-3c44-4205-a078-f857c413ad91","Type":"ContainerStarted","Data":"943ee05bb25ae7fe4215fdaec4395add68669b2a4643aa0d476543979b4c28a6"}
Oct 30 06:56:13.458865 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:13.458793    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4kfdw" event={"ID":"fe292d5d-6d46-4c34-ab24-7991571cd680","Type":"ContainerStarted","Data":"bc65bdcd581cfc4ddc866c5bf733ea06d05843df92b765587c18ee82df411d78"}
Oct 30 06:56:13.463143 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:13.463080    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-l9mfn" event={"ID":"5d87089b-8b1b-43ea-be1f-72bfb56bf311","Type":"ContainerStarted","Data":"fa9ac36aa63c36f432ecb122126c429701322d5758b168a1026b9c3b6353db26"}
Oct 30 06:56:13.475144 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:13.475114    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-125.ec2.internal" event={"ID":"4ba2b683794afdaba9ca4ed884bfac10","Type":"ContainerStarted","Data":"46906adf2a26b7edee73da7546eae9e7e092b885fd2165608c50a391c952c662"}
Oct 30 06:56:13.494966 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:13.494937    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5p67n" event={"ID":"675dc1eb-ff05-40c7-bf0b-fb3161d3cc8c","Type":"ContainerStarted","Data":"06224bc406875c768834b53973ff4ff559b88f05bd498fbe25fa9a80007bec57"}
Oct 30 06:56:13.916662 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:13.915832    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f8b75af-cc86-49c6-a66c-1798309f9806-metrics-certs\") pod \"network-metrics-daemon-r4d72\" (UID: \"6f8b75af-cc86-49c6-a66c-1798309f9806\") " pod="openshift-multus/network-metrics-daemon-r4d72"
Oct 30 06:56:13.916662 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:13.915881    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9092e2e8-ad23-426c-b6f1-14e1a2b9221e-original-pull-secret\") pod \"global-pull-secret-syncer-qktbg\" (UID: \"9092e2e8-ad23-426c-b6f1-14e1a2b9221e\") " pod="kube-system/global-pull-secret-syncer-qktbg"
Oct 30 06:56:13.916662 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:13.916041    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 06:56:13.916662 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:13.916098    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9092e2e8-ad23-426c-b6f1-14e1a2b9221e-original-pull-secret podName:9092e2e8-ad23-426c-b6f1-14e1a2b9221e nodeName:}" failed. No retries permitted until 2026-10-30 06:56:15.916080305 +0000 UTC m=+6.103394069 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9092e2e8-ad23-426c-b6f1-14e1a2b9221e-original-pull-secret") pod "global-pull-secret-syncer-qktbg" (UID: "9092e2e8-ad23-426c-b6f1-14e1a2b9221e") : object "kube-system"/"original-pull-secret" not registered
Oct 30 06:56:13.916662 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:13.916496    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 06:56:13.916662 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:13.916542    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f8b75af-cc86-49c6-a66c-1798309f9806-metrics-certs podName:6f8b75af-cc86-49c6-a66c-1798309f9806 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:15.916526329 +0000 UTC m=+6.103840083 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6f8b75af-cc86-49c6-a66c-1798309f9806-metrics-certs") pod "network-metrics-daemon-r4d72" (UID: "6f8b75af-cc86-49c6-a66c-1798309f9806") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 06:56:14.017810 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:14.017163    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cwrb\" (UniqueName: \"kubernetes.io/projected/6d493f94-3aef-445e-9b60-32a10c184eac-kube-api-access-2cwrb\") pod \"network-check-target-c6trh\" (UID: \"6d493f94-3aef-445e-9b60-32a10c184eac\") " pod="openshift-network-diagnostics/network-check-target-c6trh"
Oct 30 06:56:14.017810 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:14.017382    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 06:56:14.017810 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:14.017401    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 06:56:14.017810 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:14.017415    2580 projected.go:196] Error preparing data for projected volume kube-api-access-2cwrb for pod openshift-network-diagnostics/network-check-target-c6trh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 06:56:14.017810 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:14.017473    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d493f94-3aef-445e-9b60-32a10c184eac-kube-api-access-2cwrb podName:6d493f94-3aef-445e-9b60-32a10c184eac nodeName:}" failed. No retries permitted until 2026-10-30 06:56:16.017454184 +0000 UTC m=+6.204767951 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2cwrb" (UniqueName: "kubernetes.io/projected/6d493f94-3aef-445e-9b60-32a10c184eac-kube-api-access-2cwrb") pod "network-check-target-c6trh" (UID: "6d493f94-3aef-445e-9b60-32a10c184eac") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 06:56:14.249070 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:14.248900    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 06:51:12 +0000 UTC" deadline="2028-06-26 20:01:33.522675878 +0000 UTC"
Oct 30 06:56:14.249070 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:14.248963    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14533h5m19.273717043s"
Oct 30 06:56:14.393580 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:14.392934    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qktbg"
Oct 30 06:56:14.393580 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:14.393074    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qktbg" podUID="9092e2e8-ad23-426c-b6f1-14e1a2b9221e"
Oct 30 06:56:14.393580 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:14.393298    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6trh"
Oct 30 06:56:14.393580 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:14.393405    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6trh" podUID="6d493f94-3aef-445e-9b60-32a10c184eac"
Oct 30 06:56:14.393580 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:14.393478    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4d72"
Oct 30 06:56:14.393580 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:14.393560    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4d72" podUID="6f8b75af-cc86-49c6-a66c-1798309f9806"
Oct 30 06:56:14.511851 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:14.511696    2580 generic.go:356] "Generic (PLEG): container finished" podID="0aefa49a1bd7275a0d110a8cd8b72657" containerID="18f085935a5cc80d1a21521a5383aa41052013a0eb2dbaeabb981d4f3977f462" exitCode=0
Oct 30 06:56:14.512148 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:14.512073    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-125.ec2.internal" event={"ID":"0aefa49a1bd7275a0d110a8cd8b72657","Type":"ContainerDied","Data":"18f085935a5cc80d1a21521a5383aa41052013a0eb2dbaeabb981d4f3977f462"}
Oct 30 06:56:14.551408 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:14.551341    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-125.ec2.internal" podStartSLOduration=2.551321298 podStartE2EDuration="2.551321298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 06:56:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:13.513775412 +0000 UTC m=+3.701089186" watchObservedRunningTime="2026-10-30 06:56:14.551321298 +0000 UTC m=+4.738635072"
Oct 30 06:56:15.527414 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:15.526645    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-125.ec2.internal" event={"ID":"0aefa49a1bd7275a0d110a8cd8b72657","Type":"ContainerStarted","Data":"407f2fdca3e939433b7fdbd96d800d7d4463a0536d04c3711b1bc838e0a189ee"}
Oct 30 06:56:15.938451 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:15.938373    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f8b75af-cc86-49c6-a66c-1798309f9806-metrics-certs\") pod \"network-metrics-daemon-r4d72\" (UID: \"6f8b75af-cc86-49c6-a66c-1798309f9806\") " pod="openshift-multus/network-metrics-daemon-r4d72"
Oct 30 06:56:15.938451 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:15.938422    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9092e2e8-ad23-426c-b6f1-14e1a2b9221e-original-pull-secret\") pod \"global-pull-secret-syncer-qktbg\" (UID: \"9092e2e8-ad23-426c-b6f1-14e1a2b9221e\") " pod="kube-system/global-pull-secret-syncer-qktbg"
Oct 30 06:56:15.938676 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:15.938548    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 06:56:15.938735 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:15.938707    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f8b75af-cc86-49c6-a66c-1798309f9806-metrics-certs podName:6f8b75af-cc86-49c6-a66c-1798309f9806 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:19.938633266 +0000 UTC m=+10.125947031 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6f8b75af-cc86-49c6-a66c-1798309f9806-metrics-certs") pod "network-metrics-daemon-r4d72" (UID: "6f8b75af-cc86-49c6-a66c-1798309f9806") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 06:56:15.939270 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:15.938559    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 06:56:15.939270 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:15.939175    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9092e2e8-ad23-426c-b6f1-14e1a2b9221e-original-pull-secret podName:9092e2e8-ad23-426c-b6f1-14e1a2b9221e nodeName:}" failed. No retries permitted until 2026-10-30 06:56:19.939158324 +0000 UTC m=+10.126472074 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9092e2e8-ad23-426c-b6f1-14e1a2b9221e-original-pull-secret") pod "global-pull-secret-syncer-qktbg" (UID: "9092e2e8-ad23-426c-b6f1-14e1a2b9221e") : object "kube-system"/"original-pull-secret" not registered
Oct 30 06:56:16.039752 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:16.039673    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cwrb\" (UniqueName: \"kubernetes.io/projected/6d493f94-3aef-445e-9b60-32a10c184eac-kube-api-access-2cwrb\") pod \"network-check-target-c6trh\" (UID: \"6d493f94-3aef-445e-9b60-32a10c184eac\") " pod="openshift-network-diagnostics/network-check-target-c6trh"
Oct 30 06:56:16.039966 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:16.039907    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 06:56:16.039966 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:16.039944    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 06:56:16.039966 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:16.039958    2580 projected.go:196] Error preparing data for projected volume kube-api-access-2cwrb for pod openshift-network-diagnostics/network-check-target-c6trh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 06:56:16.040138 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:16.040021    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d493f94-3aef-445e-9b60-32a10c184eac-kube-api-access-2cwrb podName:6d493f94-3aef-445e-9b60-32a10c184eac nodeName:}" failed. No retries permitted until 2026-10-30 06:56:20.040001091 +0000 UTC m=+10.227314862 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2cwrb" (UniqueName: "kubernetes.io/projected/6d493f94-3aef-445e-9b60-32a10c184eac-kube-api-access-2cwrb") pod "network-check-target-c6trh" (UID: "6d493f94-3aef-445e-9b60-32a10c184eac") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 06:56:16.391757 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:16.391679    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4d72"
Oct 30 06:56:16.391921 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:16.391842    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4d72" podUID="6f8b75af-cc86-49c6-a66c-1798309f9806"
Oct 30 06:56:16.392284 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:16.392247    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6trh"
Oct 30 06:56:16.392413 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:16.392348    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6trh" podUID="6d493f94-3aef-445e-9b60-32a10c184eac"
Oct 30 06:56:16.392413 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:16.392397    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qktbg"
Oct 30 06:56:16.392565 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:16.392476    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qktbg" podUID="9092e2e8-ad23-426c-b6f1-14e1a2b9221e"
Oct 30 06:56:16.538830 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:16.538785    2580 generic.go:356] "Generic (PLEG): container finished" podID="ca422300-6212-4a01-b1e2-5ba04af79ce6" containerID="baa0a62281350432aff794f00c6f91cfd26c0bf9f27c2a7d51e8b9c2db32aa91" exitCode=0
Oct 30 06:56:16.539464 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:16.539437    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4js5r" event={"ID":"ca422300-6212-4a01-b1e2-5ba04af79ce6","Type":"ContainerDied","Data":"baa0a62281350432aff794f00c6f91cfd26c0bf9f27c2a7d51e8b9c2db32aa91"}
Oct 30 06:56:16.610323 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:16.610263    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-125.ec2.internal" podStartSLOduration=4.610243624 podStartE2EDuration="4.610243624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 06:56:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:15.567397583 +0000 UTC m=+5.754711358" watchObservedRunningTime="2026-10-30 06:56:16.610243624 +0000 UTC m=+6.797557397"
Oct 30 06:56:17.570800 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:17.568561    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wv8tb" event={"ID":"31519c22-050e-435c-aec3-7be0b68ff313","Type":"ContainerStarted","Data":"25967274f1648b40dd91a7be5984231809890113671a684408d535675c9a8bfb"}
Oct 30 06:56:18.391447 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:18.391398    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qktbg"
Oct 30 06:56:18.391630 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:18.391525    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qktbg" podUID="9092e2e8-ad23-426c-b6f1-14e1a2b9221e"
Oct 30 06:56:18.391993 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:18.391947    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6trh"
Oct 30 06:56:18.392094 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:18.392056    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6trh" podUID="6d493f94-3aef-445e-9b60-32a10c184eac"
Oct 30 06:56:18.392134 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:18.392124    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4d72"
Oct 30 06:56:18.392235 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:18.392212    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4d72" podUID="6f8b75af-cc86-49c6-a66c-1798309f9806"
Oct 30 06:56:18.578089 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:18.576189    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-l9mfn" event={"ID":"5d87089b-8b1b-43ea-be1f-72bfb56bf311","Type":"ContainerStarted","Data":"0ed0b75c1fce60de44a48995e48b88f6ab0780d2e16e900454efe7c38829f16b"}
Oct 30 06:56:18.614407 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:18.614350    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-l9mfn" podStartSLOduration=2.599552545 podStartE2EDuration="7.614328282s" totalImagesPullingTime="5.014775737s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 06:56:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:18.614188533 +0000 UTC m=+8.801502307" watchObservedRunningTime="2026-10-30 06:56:18.614328282 +0000 UTC m=+8.801642052"
Oct 30 06:56:18.614663 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:18.614623    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-wv8tb" podStartSLOduration=3.557602986 podStartE2EDuration="7.614612576s" totalImagesPullingTime="4.05700959s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:56:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:17.608989432 +0000 UTC m=+7.796303205" watchObservedRunningTime="2026-10-30 06:56:18.614612576 +0000 UTC m=+8.801926349"
Oct 30 06:56:19.581584 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:19.581540    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8kcn" event={"ID":"a6968152-3c44-4205-a078-f857c413ad91","Type":"ContainerStarted","Data":"b9c575388cbc363457f62cc62890200e3217aeb4c6ca60e85274800017463e88"}
Oct 30 06:56:19.977405 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:19.977323    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f8b75af-cc86-49c6-a66c-1798309f9806-metrics-certs\") pod \"network-metrics-daemon-r4d72\" (UID: \"6f8b75af-cc86-49c6-a66c-1798309f9806\") " pod="openshift-multus/network-metrics-daemon-r4d72"
Oct 30 06:56:19.977405 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:19.977367    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9092e2e8-ad23-426c-b6f1-14e1a2b9221e-original-pull-secret\") pod \"global-pull-secret-syncer-qktbg\" (UID: \"9092e2e8-ad23-426c-b6f1-14e1a2b9221e\") " pod="kube-system/global-pull-secret-syncer-qktbg"
Oct 30 06:56:19.977635 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:19.977469    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 06:56:19.977635 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:19.977527    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9092e2e8-ad23-426c-b6f1-14e1a2b9221e-original-pull-secret podName:9092e2e8-ad23-426c-b6f1-14e1a2b9221e nodeName:}" failed. No retries permitted until 2026-10-30 06:56:27.977507159 +0000 UTC m=+18.164820922 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9092e2e8-ad23-426c-b6f1-14e1a2b9221e-original-pull-secret") pod "global-pull-secret-syncer-qktbg" (UID: "9092e2e8-ad23-426c-b6f1-14e1a2b9221e") : object "kube-system"/"original-pull-secret" not registered
Oct 30 06:56:19.978027 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:19.978002    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 06:56:19.978090 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:19.978068    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f8b75af-cc86-49c6-a66c-1798309f9806-metrics-certs podName:6f8b75af-cc86-49c6-a66c-1798309f9806 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:27.978054639 +0000 UTC m=+18.165368402 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6f8b75af-cc86-49c6-a66c-1798309f9806-metrics-certs") pod "network-metrics-daemon-r4d72" (UID: "6f8b75af-cc86-49c6-a66c-1798309f9806") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 06:56:20.078222 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:20.078178    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cwrb\" (UniqueName: \"kubernetes.io/projected/6d493f94-3aef-445e-9b60-32a10c184eac-kube-api-access-2cwrb\") pod \"network-check-target-c6trh\" (UID: \"6d493f94-3aef-445e-9b60-32a10c184eac\") " pod="openshift-network-diagnostics/network-check-target-c6trh"
Oct 30 06:56:20.078439 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:20.078419    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 06:56:20.078517 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:20.078448    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 06:56:20.078517 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:20.078463    2580 projected.go:196] Error preparing data for projected volume kube-api-access-2cwrb for pod openshift-network-diagnostics/network-check-target-c6trh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 06:56:20.078627 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:20.078531    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d493f94-3aef-445e-9b60-32a10c184eac-kube-api-access-2cwrb podName:6d493f94-3aef-445e-9b60-32a10c184eac nodeName:}" failed. No retries permitted until 2026-10-30 06:56:28.078510529 +0000 UTC m=+18.265824299 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2cwrb" (UniqueName: "kubernetes.io/projected/6d493f94-3aef-445e-9b60-32a10c184eac-kube-api-access-2cwrb") pod "network-check-target-c6trh" (UID: "6d493f94-3aef-445e-9b60-32a10c184eac") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 06:56:20.389944 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:20.389864    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qktbg"
Oct 30 06:56:20.390111 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:20.389994    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qktbg" podUID="9092e2e8-ad23-426c-b6f1-14e1a2b9221e"
Oct 30 06:56:20.390111 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:20.390043    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4d72"
Oct 30 06:56:20.390218 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:20.390189    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4d72" podUID="6f8b75af-cc86-49c6-a66c-1798309f9806"
Oct 30 06:56:20.390269 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:20.390254    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6trh"
Oct 30 06:56:20.390377 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:20.390342    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6trh" podUID="6d493f94-3aef-445e-9b60-32a10c184eac"
Oct 30 06:56:20.750730 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:20.750702    2580 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 06:56:21.247166 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:21.247023    2580 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T06:56:20.750727895Z","UUID":"d70426b0-3153-4ab5-b586-507b6a39e526","Handler":null,"Name":"","Endpoint":""}
Oct 30 06:56:21.248746 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:21.248723    2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 06:56:21.248855 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:21.248752    2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 06:56:21.589577 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:21.589363    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5p67n" event={"ID":"675dc1eb-ff05-40c7-bf0b-fb3161d3cc8c","Type":"ContainerStarted","Data":"fa4eb2d0c3c0d47bcdd86f506cc69f0eda9830bf19eefc3a7c86bbe56b40d4cf"}
Oct 30 06:56:21.591470 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:21.591436    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9gqrp" event={"ID":"2368101c-6719-4650-8393-ef048500a624","Type":"ContainerStarted","Data":"60d594dec54df859d3d5337956c5577bdced7837ac3bb0ee8aa9136bcf02a350"}
Oct 30 06:56:21.593871 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:21.593841    2580 generic.go:356] "Generic (PLEG): container finished" podID="ca422300-6212-4a01-b1e2-5ba04af79ce6" containerID="b0f4b3989ce0cc9774d660e0f4ec409d7bc78150a1cd72d6b842fdcd65c2303e" exitCode=0
Oct 30 06:56:21.593973 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:21.593940    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4js5r" event={"ID":"ca422300-6212-4a01-b1e2-5ba04af79ce6","Type":"ContainerDied","Data":"b0f4b3989ce0cc9774d660e0f4ec409d7bc78150a1cd72d6b842fdcd65c2303e"}
Oct 30 06:56:21.598169 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:21.598136    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8kcn" event={"ID":"a6968152-3c44-4205-a078-f857c413ad91","Type":"ContainerStarted","Data":"b8ea389592619a4117327588f114d85b88185991fcfee44c215310b9705d493c"}
Oct 30 06:56:21.628196 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:21.628150    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-5p67n" podStartSLOduration=2.396802342 podStartE2EDuration="10.628135278s" totalImagesPullingTime="8.231332936s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 06:56:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:21.627753067 +0000 UTC m=+11.815066840" watchObservedRunningTime="2026-10-30 06:56:21.628135278 +0000 UTC m=+11.815449053"
Oct 30 06:56:21.715648 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:21.715592    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-9gqrp" podStartSLOduration=2.610456654 podStartE2EDuration="10.715577876s" totalImagesPullingTime="8.105121222s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 06:56:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:21.715006497 +0000 UTC m=+11.902320270" watchObservedRunningTime="2026-10-30 06:56:21.715577876 +0000 UTC m=+11.902891648"
Oct 30 06:56:22.258740 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:22.258710    2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-wv8tb"
Oct 30 06:56:22.261990 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:22.261967    2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-wv8tb"
Oct 30 06:56:22.388832 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:22.388789    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6trh"
Oct 30 06:56:22.388955 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:22.388919    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6trh" podUID="6d493f94-3aef-445e-9b60-32a10c184eac"
Oct 30 06:56:22.389026 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:22.389011    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4d72"
Oct 30 06:56:22.389133 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:22.389112    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4d72" podUID="6f8b75af-cc86-49c6-a66c-1798309f9806"
Oct 30 06:56:22.389286 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:22.389261    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qktbg"
Oct 30 06:56:22.389354 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:22.389338    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qktbg" podUID="9092e2e8-ad23-426c-b6f1-14e1a2b9221e"
Oct 30 06:56:22.602194 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:22.602116    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tq7pf" event={"ID":"851fb6ff-4a58-431b-834f-db3f87b67aa9","Type":"ContainerStarted","Data":"3bc88cc2051df843e9553d111130b679512732d7507e88f924e542bbcff4bb1f"}
Oct 30 06:56:22.604897 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:22.604857    2580 generic.go:356] "Generic (PLEG): container finished" podID="ca422300-6212-4a01-b1e2-5ba04af79ce6" containerID="1a939e120e007404225424b099320dded90571a77f93e2ba4aeffa40d4869883" exitCode=0
Oct 30 06:56:22.605009 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:22.604952    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4js5r" event={"ID":"ca422300-6212-4a01-b1e2-5ba04af79ce6","Type":"ContainerDied","Data":"1a939e120e007404225424b099320dded90571a77f93e2ba4aeffa40d4869883"}
Oct 30 06:56:22.608130 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:22.608099    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8kcn" event={"ID":"a6968152-3c44-4205-a078-f857c413ad91","Type":"ContainerStarted","Data":"84a25734f5ded411d292ec4ae17bef40ef4be1eda49c33370e887b9a4b67e92b"}
Oct 30 06:56:22.682076 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:22.681802    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-tq7pf" podStartSLOduration=3.815911108 podStartE2EDuration="11.68178539s" totalImagesPullingTime="7.865874282s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 06:56:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:22.639195068 +0000 UTC m=+12.826508838" watchObservedRunningTime="2026-10-30 06:56:22.68178539 +0000 UTC m=+12.869099160"
Oct 30 06:56:22.721285 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:22.721238    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8kcn" podStartSLOduration=2.998697561 podStartE2EDuration="11.72122078s" totalImagesPullingTime="8.722523219s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 06:56:11 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:22.720918605 +0000 UTC m=+12.908232374" watchObservedRunningTime="2026-10-30 06:56:22.72122078 +0000 UTC m=+12.908534554"
Oct 30 06:56:23.612074 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:23.612037    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4kfdw" event={"ID":"fe292d5d-6d46-4c34-ab24-7991571cd680","Type":"ContainerStarted","Data":"99a0f1a998993d1787f986aa1f2901e7ed256b983ee377a03a09852f6cd651ff"}
Oct 30 06:56:24.389046 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:24.389012    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6trh"
Oct 30 06:56:24.389236 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:24.389066    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4d72"
Oct 30 06:56:24.389236 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:24.389013    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qktbg"
Oct 30 06:56:24.389236 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:24.389153    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6trh" podUID="6d493f94-3aef-445e-9b60-32a10c184eac"
Oct 30 06:56:24.389405 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:24.389243    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qktbg" podUID="9092e2e8-ad23-426c-b6f1-14e1a2b9221e"
Oct 30 06:56:24.389405 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:24.389317    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4d72" podUID="6f8b75af-cc86-49c6-a66c-1798309f9806"
Oct 30 06:56:24.617280 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:24.616730    2580 generic.go:356] "Generic (PLEG): container finished" podID="ca422300-6212-4a01-b1e2-5ba04af79ce6" containerID="96d297ea28178d5d113805bce6db022c0f37866299edf48ccbcf20e18076d280" exitCode=0
Oct 30 06:56:24.617280 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:24.616779    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4js5r" event={"ID":"ca422300-6212-4a01-b1e2-5ba04af79ce6","Type":"ContainerDied","Data":"96d297ea28178d5d113805bce6db022c0f37866299edf48ccbcf20e18076d280"}
Oct 30 06:56:24.681432 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:24.681329    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-4kfdw" podStartSLOduration=2.875819654 podStartE2EDuration="13.681309348s" totalImagesPullingTime="10.805489694s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 06:56:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:23.651509631 +0000 UTC m=+13.838823402" watchObservedRunningTime="2026-10-30 06:56:24.681309348 +0000 UTC m=+14.868623111"
Oct 30 06:56:26.388421 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:26.388178    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6trh"
Oct 30 06:56:26.388915 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:26.388513    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6trh" podUID="6d493f94-3aef-445e-9b60-32a10c184eac"
Oct 30 06:56:26.388915 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:26.388248    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4d72"
Oct 30 06:56:26.388915 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:26.388591    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4d72" podUID="6f8b75af-cc86-49c6-a66c-1798309f9806"
Oct 30 06:56:26.388915 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:26.388178    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qktbg"
Oct 30 06:56:26.388915 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:26.388645    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qktbg" podUID="9092e2e8-ad23-426c-b6f1-14e1a2b9221e"
Oct 30 06:56:26.624846 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:26.624793    2580 generic.go:356] "Generic (PLEG): container finished" podID="ca422300-6212-4a01-b1e2-5ba04af79ce6" containerID="f419063bee1726a21df48a53721f93506cc649b2e339443cecf2ef494b95ffc5" exitCode=0
Oct 30 06:56:26.625000 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:26.624841    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4js5r" event={"ID":"ca422300-6212-4a01-b1e2-5ba04af79ce6","Type":"ContainerDied","Data":"f419063bee1726a21df48a53721f93506cc649b2e339443cecf2ef494b95ffc5"}
Oct 30 06:56:27.629732 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:27.629531    2580 generic.go:356] "Generic (PLEG): container finished" podID="ca422300-6212-4a01-b1e2-5ba04af79ce6" containerID="faa60c8808a69c43b547e9ae5181be751acaee31878affe4d0b60f356a31a544" exitCode=0
Oct 30 06:56:27.630089 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:27.629605    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4js5r" event={"ID":"ca422300-6212-4a01-b1e2-5ba04af79ce6","Type":"ContainerDied","Data":"faa60c8808a69c43b547e9ae5181be751acaee31878affe4d0b60f356a31a544"}
Oct 30 06:56:27.632547 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:27.632528    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx" event={"ID":"b321b279-a102-48b9-bd9b-8ed938eed0f7","Type":"ContainerStarted","Data":"5e4540e43a060c268aff82ebe33225dfe8c2d70414b2f9465ba75f300bcd36fa"}
Oct 30 06:56:27.632613 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:27.632590    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx" event={"ID":"b321b279-a102-48b9-bd9b-8ed938eed0f7","Type":"ContainerStarted","Data":"7bf943733b87a2ec01bb65ec579bc27e90d5c5fbb7b4a1cb449fbb42f09913ee"}
Oct 30 06:56:27.632644 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:27.632620    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx" event={"ID":"b321b279-a102-48b9-bd9b-8ed938eed0f7","Type":"ContainerStarted","Data":"97adb7ab67d7ae964d34383762ac0ab43b71e0f8f0e26f0ea30dc173152a65f1"}
Oct 30 06:56:27.632682 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:27.632655    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx" event={"ID":"b321b279-a102-48b9-bd9b-8ed938eed0f7","Type":"ContainerStarted","Data":"ef26d26f5c524101d8af94a98e0655696d8862cb0f8d61f13af0e1b19b704d19"}
Oct 30 06:56:28.038178 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:28.038146    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f8b75af-cc86-49c6-a66c-1798309f9806-metrics-certs\") pod \"network-metrics-daemon-r4d72\" (UID: \"6f8b75af-cc86-49c6-a66c-1798309f9806\") " pod="openshift-multus/network-metrics-daemon-r4d72"
Oct 30 06:56:28.038178 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:28.038180    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9092e2e8-ad23-426c-b6f1-14e1a2b9221e-original-pull-secret\") pod \"global-pull-secret-syncer-qktbg\" (UID: \"9092e2e8-ad23-426c-b6f1-14e1a2b9221e\") " pod="kube-system/global-pull-secret-syncer-qktbg"
Oct 30 06:56:28.038338 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:28.038293    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 06:56:28.038393 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:28.038345    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9092e2e8-ad23-426c-b6f1-14e1a2b9221e-original-pull-secret podName:9092e2e8-ad23-426c-b6f1-14e1a2b9221e nodeName:}" failed. No retries permitted until 2026-10-30 06:56:44.03833296 +0000 UTC m=+34.225646721 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9092e2e8-ad23-426c-b6f1-14e1a2b9221e-original-pull-secret") pod "global-pull-secret-syncer-qktbg" (UID: "9092e2e8-ad23-426c-b6f1-14e1a2b9221e") : object "kube-system"/"original-pull-secret" not registered
Oct 30 06:56:28.038393 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:28.038293    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 06:56:28.038464 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:28.038412    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f8b75af-cc86-49c6-a66c-1798309f9806-metrics-certs podName:6f8b75af-cc86-49c6-a66c-1798309f9806 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:44.038399619 +0000 UTC m=+34.225713380 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6f8b75af-cc86-49c6-a66c-1798309f9806-metrics-certs") pod "network-metrics-daemon-r4d72" (UID: "6f8b75af-cc86-49c6-a66c-1798309f9806") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 06:56:28.138973 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:28.138946    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cwrb\" (UniqueName: \"kubernetes.io/projected/6d493f94-3aef-445e-9b60-32a10c184eac-kube-api-access-2cwrb\") pod \"network-check-target-c6trh\" (UID: \"6d493f94-3aef-445e-9b60-32a10c184eac\") " pod="openshift-network-diagnostics/network-check-target-c6trh"
Oct 30 06:56:28.139113 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:28.139097    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 06:56:28.139172 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:28.139118    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 06:56:28.139172 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:28.139128    2580 projected.go:196] Error preparing data for projected volume kube-api-access-2cwrb for pod openshift-network-diagnostics/network-check-target-c6trh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 06:56:28.139233 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:28.139181    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d493f94-3aef-445e-9b60-32a10c184eac-kube-api-access-2cwrb podName:6d493f94-3aef-445e-9b60-32a10c184eac nodeName:}" failed. No retries permitted until 2026-10-30 06:56:44.139168038 +0000 UTC m=+34.326481799 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2cwrb" (UniqueName: "kubernetes.io/projected/6d493f94-3aef-445e-9b60-32a10c184eac-kube-api-access-2cwrb") pod "network-check-target-c6trh" (UID: "6d493f94-3aef-445e-9b60-32a10c184eac") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 06:56:28.388826 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:28.388735    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6trh"
Oct 30 06:56:28.388826 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:28.388761    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qktbg"
Oct 30 06:56:28.388980 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:28.388918    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4d72"
Oct 30 06:56:28.388980 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:28.388933    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qktbg" podUID="9092e2e8-ad23-426c-b6f1-14e1a2b9221e"
Oct 30 06:56:28.389077 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:28.388988    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6trh" podUID="6d493f94-3aef-445e-9b60-32a10c184eac"
Oct 30 06:56:28.389108 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:28.389076    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4d72" podUID="6f8b75af-cc86-49c6-a66c-1798309f9806"
Oct 30 06:56:28.637182 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:28.637149    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4js5r" event={"ID":"ca422300-6212-4a01-b1e2-5ba04af79ce6","Type":"ContainerStarted","Data":"9e0bcbc230b480ed1e1c76a51888e75941c28ce534130d3c7b4e6700aec44491"}
Oct 30 06:56:28.640026 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:28.639968    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx" event={"ID":"b321b279-a102-48b9-bd9b-8ed938eed0f7","Type":"ContainerStarted","Data":"07ca5835d5cbcfa8a153f637702a32c8033f467713e074c63baf0a11d7a87f90"}
Oct 30 06:56:28.640127 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:28.640028    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx" event={"ID":"b321b279-a102-48b9-bd9b-8ed938eed0f7","Type":"ContainerStarted","Data":"d217f8ecfffb284465e14c6d076ae256846eaff407f3adbd8cd3762fbc9891ab"}
Oct 30 06:56:28.701451 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:28.701410    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4js5r" podStartSLOduration=7.088287723 podStartE2EDuration="17.701397804s" totalImagesPullingTime="10.613110081s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 06:56:11 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:28.701381081 +0000 UTC m=+18.888694850" watchObservedRunningTime="2026-10-30 06:56:28.701397804 +0000 UTC m=+18.888711573"
Oct 30 06:56:30.388732 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:30.388703    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qktbg"
Oct 30 06:56:30.389210 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:30.388790    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qktbg" podUID="9092e2e8-ad23-426c-b6f1-14e1a2b9221e"
Oct 30 06:56:30.389210 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:30.388860    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4d72"
Oct 30 06:56:30.389210 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:30.388911    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4d72" podUID="6f8b75af-cc86-49c6-a66c-1798309f9806"
Oct 30 06:56:30.389210 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:30.388959    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6trh"
Oct 30 06:56:30.389210 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:30.389053    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6trh" podUID="6d493f94-3aef-445e-9b60-32a10c184eac"
Oct 30 06:56:30.646722 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:30.646652    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx" event={"ID":"b321b279-a102-48b9-bd9b-8ed938eed0f7","Type":"ContainerStarted","Data":"72671e1891893f717cf057155645587f0999138608e5058dd6d7d1eb8325f49b"}
Oct 30 06:56:32.388959 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:32.388756    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4d72"
Oct 30 06:56:32.389431 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:32.388783    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qktbg"
Oct 30 06:56:32.389431 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:32.389044    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4d72" podUID="6f8b75af-cc86-49c6-a66c-1798309f9806"
Oct 30 06:56:32.389431 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:32.389150    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qktbg" podUID="9092e2e8-ad23-426c-b6f1-14e1a2b9221e"
Oct 30 06:56:32.389431 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:32.388851    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6trh"
Oct 30 06:56:32.389431 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:32.389251    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6trh" podUID="6d493f94-3aef-445e-9b60-32a10c184eac"
Oct 30 06:56:32.654178 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:32.654111    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx" event={"ID":"b321b279-a102-48b9-bd9b-8ed938eed0f7","Type":"ContainerStarted","Data":"428ca3ba943ee720d826e89f6566589431c92ef8b26368f38ed1c927e2c78ea1"}
Oct 30 06:56:32.654354 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:32.654335    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:32.669764 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:32.669739    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:32.742343 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:32.742298    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx" podStartSLOduration=6.921939762 podStartE2EDuration="21.742286393s" totalImagesPullingTime="14.820346631s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 06:56:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:32.742059472 +0000 UTC m=+22.929373242" watchObservedRunningTime="2026-10-30 06:56:32.742286393 +0000 UTC m=+22.929600161"
Oct 30 06:56:33.656233 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:33.656211    2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 06:56:33.656635 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:33.656616    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:33.672303 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:33.672282    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:33.980911 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:33.980832    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qktbg"]
Oct 30 06:56:33.981236 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:33.980969    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qktbg"
Oct 30 06:56:33.981236 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:33.981082    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qktbg" podUID="9092e2e8-ad23-426c-b6f1-14e1a2b9221e"
Oct 30 06:56:33.981886 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:33.981862    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-c6trh"]
Oct 30 06:56:33.981994 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:33.981968    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6trh"
Oct 30 06:56:33.982086 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:33.982053    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6trh" podUID="6d493f94-3aef-445e-9b60-32a10c184eac"
Oct 30 06:56:33.983924 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:33.983904    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-r4d72"]
Oct 30 06:56:33.984006 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:33.983984    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4d72"
Oct 30 06:56:33.984093 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:33.984078    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4d72" podUID="6f8b75af-cc86-49c6-a66c-1798309f9806"
Oct 30 06:56:34.657593 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:34.657565    2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 06:56:35.195262 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:35.195236    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5p67n_675dc1eb-ff05-40c7-bf0b-fb3161d3cc8c/dns-node-resolver/0.log"
Oct 30 06:56:35.388250 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:35.388219    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qktbg"
Oct 30 06:56:35.388250 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:35.388242    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6trh"
Oct 30 06:56:35.388413 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:35.388314    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qktbg" podUID="9092e2e8-ad23-426c-b6f1-14e1a2b9221e"
Oct 30 06:56:35.388413 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:35.388373    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6trh" podUID="6d493f94-3aef-445e-9b60-32a10c184eac"
Oct 30 06:56:35.388413 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:35.388396    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4d72"
Oct 30 06:56:35.388525 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:35.388470    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4d72" podUID="6f8b75af-cc86-49c6-a66c-1798309f9806"
Oct 30 06:56:35.659737 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:35.659709    2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 06:56:36.331080 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:36.331054    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-l9mfn_5d87089b-8b1b-43ea-be1f-72bfb56bf311/node-ca/0.log"
Oct 30 06:56:37.388775 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:37.388744    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qktbg"
Oct 30 06:56:37.388775 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:37.388772    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6trh"
Oct 30 06:56:37.389374 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:37.388876    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qktbg" podUID="9092e2e8-ad23-426c-b6f1-14e1a2b9221e"
Oct 30 06:56:37.389374 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:37.388960    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6trh" podUID="6d493f94-3aef-445e-9b60-32a10c184eac"
Oct 30 06:56:37.389374 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:37.389049    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4d72"
Oct 30 06:56:37.389374 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:37.389144    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4d72" podUID="6f8b75af-cc86-49c6-a66c-1798309f9806"
Oct 30 06:56:39.388298 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:39.388268    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qktbg"
Oct 30 06:56:39.388298 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:39.388280    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4d72"
Oct 30 06:56:39.388765 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:39.388270    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6trh"
Oct 30 06:56:39.388765 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:39.388360    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qktbg" podUID="9092e2e8-ad23-426c-b6f1-14e1a2b9221e"
Oct 30 06:56:39.388765 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:39.388458    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6trh" podUID="6d493f94-3aef-445e-9b60-32a10c184eac"
Oct 30 06:56:39.388765 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:39.388543    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4d72" podUID="6f8b75af-cc86-49c6-a66c-1798309f9806"
Oct 30 06:56:40.983675 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:40.983491    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-wv8tb"
Oct 30 06:56:40.984076 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:40.983766    2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 06:56:40.984076 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:40.984054    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wv8tb"
Oct 30 06:56:41.224112 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:41.224083    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:41.224329 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:41.224313    2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 06:56:41.238402 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:41.238341    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:56:41.388317 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:41.388286    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4d72"
Oct 30 06:56:41.388464 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:41.388381    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4d72" podUID="6f8b75af-cc86-49c6-a66c-1798309f9806"
Oct 30 06:56:41.388464 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:41.388294    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qktbg"
Oct 30 06:56:41.388464 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:41.388398    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6trh"
Oct 30 06:56:41.388464 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:41.388451    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qktbg" podUID="9092e2e8-ad23-426c-b6f1-14e1a2b9221e"
Oct 30 06:56:41.388594 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:41.388519    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6trh" podUID="6d493f94-3aef-445e-9b60-32a10c184eac"
Oct 30 06:56:43.388476 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:43.388448    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4d72"
Oct 30 06:56:43.388868 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:43.388525    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6trh"
Oct 30 06:56:43.388868 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:43.388612    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qktbg"
Oct 30 06:56:43.388868 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:43.388625    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4d72" podUID="6f8b75af-cc86-49c6-a66c-1798309f9806"
Oct 30 06:56:43.388868 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:43.388713    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6trh" podUID="6d493f94-3aef-445e-9b60-32a10c184eac"
Oct 30 06:56:43.388868 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:43.388779    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qktbg" podUID="9092e2e8-ad23-426c-b6f1-14e1a2b9221e"
Oct 30 06:56:44.060439 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:44.060406    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f8b75af-cc86-49c6-a66c-1798309f9806-metrics-certs\") pod \"network-metrics-daemon-r4d72\" (UID: \"6f8b75af-cc86-49c6-a66c-1798309f9806\") " pod="openshift-multus/network-metrics-daemon-r4d72"
Oct 30 06:56:44.060439 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:44.060438    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9092e2e8-ad23-426c-b6f1-14e1a2b9221e-original-pull-secret\") pod \"global-pull-secret-syncer-qktbg\" (UID: \"9092e2e8-ad23-426c-b6f1-14e1a2b9221e\") " pod="kube-system/global-pull-secret-syncer-qktbg"
Oct 30 06:56:44.060684 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:44.060569    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 06:56:44.060684 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:44.060606    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 06:56:44.060684 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:44.060639    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f8b75af-cc86-49c6-a66c-1798309f9806-metrics-certs podName:6f8b75af-cc86-49c6-a66c-1798309f9806 nodeName:}" failed. No retries permitted until 2026-10-30 06:57:16.060620197 +0000 UTC m=+66.247933947 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6f8b75af-cc86-49c6-a66c-1798309f9806-metrics-certs") pod "network-metrics-daemon-r4d72" (UID: "6f8b75af-cc86-49c6-a66c-1798309f9806") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 06:56:44.060684 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:44.060659    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9092e2e8-ad23-426c-b6f1-14e1a2b9221e-original-pull-secret podName:9092e2e8-ad23-426c-b6f1-14e1a2b9221e nodeName:}" failed. No retries permitted until 2026-10-30 06:57:16.060649253 +0000 UTC m=+66.247963003 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9092e2e8-ad23-426c-b6f1-14e1a2b9221e-original-pull-secret") pod "global-pull-secret-syncer-qktbg" (UID: "9092e2e8-ad23-426c-b6f1-14e1a2b9221e") : object "kube-system"/"original-pull-secret" not registered
Oct 30 06:56:44.161188 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:44.161132    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cwrb\" (UniqueName: \"kubernetes.io/projected/6d493f94-3aef-445e-9b60-32a10c184eac-kube-api-access-2cwrb\") pod \"network-check-target-c6trh\" (UID: \"6d493f94-3aef-445e-9b60-32a10c184eac\") " pod="openshift-network-diagnostics/network-check-target-c6trh"
Oct 30 06:56:44.161287 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:44.161241    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 06:56:44.161287 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:44.161253    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 06:56:44.161287 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:44.161262    2580 projected.go:196] Error preparing data for projected volume kube-api-access-2cwrb for pod openshift-network-diagnostics/network-check-target-c6trh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 06:56:44.161380 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:44.161299    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d493f94-3aef-445e-9b60-32a10c184eac-kube-api-access-2cwrb podName:6d493f94-3aef-445e-9b60-32a10c184eac nodeName:}" failed. No retries permitted until 2026-10-30 06:57:16.161286835 +0000 UTC m=+66.348600581 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-2cwrb" (UniqueName: "kubernetes.io/projected/6d493f94-3aef-445e-9b60-32a10c184eac-kube-api-access-2cwrb") pod "network-check-target-c6trh" (UID: "6d493f94-3aef-445e-9b60-32a10c184eac") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 06:56:45.388742 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:45.388706    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6trh"
Oct 30 06:56:45.388742 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:45.388740    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qktbg"
Oct 30 06:56:45.389183 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:45.388798    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6trh" podUID="6d493f94-3aef-445e-9b60-32a10c184eac"
Oct 30 06:56:45.389183 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:45.388851    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qktbg" podUID="9092e2e8-ad23-426c-b6f1-14e1a2b9221e"
Oct 30 06:56:45.389183 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:45.388890    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4d72"
Oct 30 06:56:45.389183 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:45.388986    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4d72" podUID="6f8b75af-cc86-49c6-a66c-1798309f9806"
Oct 30 06:56:47.389057 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:47.389022    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4d72"
Oct 30 06:56:47.389465 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:47.389022    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qktbg"
Oct 30 06:56:47.389465 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:47.389128    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4d72" podUID="6f8b75af-cc86-49c6-a66c-1798309f9806"
Oct 30 06:56:47.389465 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:47.389035    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6trh"
Oct 30 06:56:47.389465 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:47.389199    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qktbg" podUID="9092e2e8-ad23-426c-b6f1-14e1a2b9221e"
Oct 30 06:56:47.389465 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:47.389259    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6trh" podUID="6d493f94-3aef-445e-9b60-32a10c184eac"
Oct 30 06:56:49.388205 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:49.388174    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4d72"
Oct 30 06:56:49.388672 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:49.388280    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qktbg"
Oct 30 06:56:49.388672 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:49.388293    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4d72" podUID="6f8b75af-cc86-49c6-a66c-1798309f9806"
Oct 30 06:56:49.388672 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:49.388355    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qktbg" podUID="9092e2e8-ad23-426c-b6f1-14e1a2b9221e"
Oct 30 06:56:49.388672 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:49.388388    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6trh"
Oct 30 06:56:49.388672 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:49.388433    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6trh" podUID="6d493f94-3aef-445e-9b60-32a10c184eac"
Oct 30 06:56:51.388318 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:51.388288    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4d72"
Oct 30 06:56:51.388690 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:51.388364    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6trh"
Oct 30 06:56:51.388690 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:51.388447    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qktbg"
Oct 30 06:56:51.388690 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:51.388461    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4d72" podUID="6f8b75af-cc86-49c6-a66c-1798309f9806"
Oct 30 06:56:51.388690 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:51.388508    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qktbg" podUID="9092e2e8-ad23-426c-b6f1-14e1a2b9221e"
Oct 30 06:56:51.388690 ip-10-0-128-125 kubenswrapper[2580]: E1030 06:56:51.388577    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6trh" podUID="6d493f94-3aef-445e-9b60-32a10c184eac"
Oct 30 06:56:53.127215 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.127193    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-125.ec2.internal" event="NodeReady"
Oct 30 06:56:53.127536 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.127302    2580 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 06:56:53.228424 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.228397    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-ltdxc"]
Oct 30 06:56:53.253524 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.253502    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-dncbb"]
Oct 30 06:56:53.253628 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.253612    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ltdxc"
Oct 30 06:56:53.267290 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.267269    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ltdxc"]
Oct 30 06:56:53.267376 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.267315    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dncbb"]
Oct 30 06:56:53.267413 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.267374    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dncbb"
Oct 30 06:56:53.271015 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.270993    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-df2z5"]
Oct 30 06:56:53.272506 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.272489    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-df2z5"
Oct 30 06:56:53.273771 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.273749    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-x5k2w\""
Oct 30 06:56:53.274031 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.274013    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 06:56:53.275061 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.274372    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 06:56:53.275061 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.274488    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 06:56:53.275061 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.274504    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 06:56:53.295490 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.295471    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-df2z5"]
Oct 30 06:56:53.375365 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.375302    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 06:56:53.375365 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.375326    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 06:56:53.375365 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.375328    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 06:56:53.375521 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.375375    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 06:56:53.375521 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.375427    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 06:56:53.375521 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.375498    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pptc7\""
Oct 30 06:56:53.375607 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.375580    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-dpx9k\""
Oct 30 06:56:53.388604 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.388581    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6trh"
Oct 30 06:56:53.388678 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.388644    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4d72"
Oct 30 06:56:53.388678 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.388663    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qktbg"
Oct 30 06:56:53.430252 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.430224    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dda66b1a-94e1-40dd-a77d-bfe37dd22b55-metrics-tls\") pod \"dns-default-dncbb\" (UID: \"dda66b1a-94e1-40dd-a77d-bfe37dd22b55\") " pod="openshift-dns/dns-default-dncbb"
Oct 30 06:56:53.430361 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.430259    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dda66b1a-94e1-40dd-a77d-bfe37dd22b55-tmp-dir\") pod \"dns-default-dncbb\" (UID: \"dda66b1a-94e1-40dd-a77d-bfe37dd22b55\") " pod="openshift-dns/dns-default-dncbb"
Oct 30 06:56:53.430361 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.430283    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9c5ccc21-fce2-4221-9c19-4a35fef32eff-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ltdxc\" (UID: \"9c5ccc21-fce2-4221-9c19-4a35fef32eff\") " pod="openshift-insights/insights-runtime-extractor-ltdxc"
Oct 30 06:56:53.430449 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.430428    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dda66b1a-94e1-40dd-a77d-bfe37dd22b55-config-volume\") pod \"dns-default-dncbb\" (UID: \"dda66b1a-94e1-40dd-a77d-bfe37dd22b55\") " pod="openshift-dns/dns-default-dncbb"
Oct 30 06:56:53.430485 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.430472    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3062f537-9b22-4e15-97aa-9a68703702f8-cert\") pod \"ingress-canary-df2z5\" (UID: \"3062f537-9b22-4e15-97aa-9a68703702f8\") " pod="openshift-ingress-canary/ingress-canary-df2z5"
Oct 30 06:56:53.430528 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.430493    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9c5ccc21-fce2-4221-9c19-4a35fef32eff-crio-socket\") pod \"insights-runtime-extractor-ltdxc\" (UID: \"9c5ccc21-fce2-4221-9c19-4a35fef32eff\") " pod="openshift-insights/insights-runtime-extractor-ltdxc"
Oct 30 06:56:53.430528 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.430512    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9c5ccc21-fce2-4221-9c19-4a35fef32eff-data-volume\") pod \"insights-runtime-extractor-ltdxc\" (UID: \"9c5ccc21-fce2-4221-9c19-4a35fef32eff\") " pod="openshift-insights/insights-runtime-extractor-ltdxc"
Oct 30 06:56:53.430728 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.430708    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c5ccc21-fce2-4221-9c19-4a35fef32eff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ltdxc\" (UID: \"9c5ccc21-fce2-4221-9c19-4a35fef32eff\") " pod="openshift-insights/insights-runtime-extractor-ltdxc"
Oct 30 06:56:53.430781 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.430749    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px9vv\" (UniqueName: \"kubernetes.io/projected/dda66b1a-94e1-40dd-a77d-bfe37dd22b55-kube-api-access-px9vv\") pod \"dns-default-dncbb\" (UID: \"dda66b1a-94e1-40dd-a77d-bfe37dd22b55\") " pod="openshift-dns/dns-default-dncbb"
Oct 30 06:56:53.430781 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.430775    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnrqt\" (UniqueName: \"kubernetes.io/projected/3062f537-9b22-4e15-97aa-9a68703702f8-kube-api-access-wnrqt\") pod \"ingress-canary-df2z5\" (UID: \"3062f537-9b22-4e15-97aa-9a68703702f8\") " pod="openshift-ingress-canary/ingress-canary-df2z5"
Oct 30 06:56:53.430897 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.430799    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbpk2\" (UniqueName: \"kubernetes.io/projected/9c5ccc21-fce2-4221-9c19-4a35fef32eff-kube-api-access-lbpk2\") pod \"insights-runtime-extractor-ltdxc\" (UID: \"9c5ccc21-fce2-4221-9c19-4a35fef32eff\") " pod="openshift-insights/insights-runtime-extractor-ltdxc"
Oct 30 06:56:53.475680 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.475661    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 06:56:53.475829 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.475720    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 06:56:53.476112 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.476093    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 06:56:53.476112 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.476107    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9hhpl\""
Oct 30 06:56:53.476253 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.476112    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-wlntw\""
Oct 30 06:56:53.476253 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.476142    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hkqzl\""
Oct 30 06:56:53.476253 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.476165    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 06:56:53.531570 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.531545    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dda66b1a-94e1-40dd-a77d-bfe37dd22b55-config-volume\") pod \"dns-default-dncbb\" (UID: \"dda66b1a-94e1-40dd-a77d-bfe37dd22b55\") " pod="openshift-dns/dns-default-dncbb"
Oct 30 06:56:53.531677 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.531575    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3062f537-9b22-4e15-97aa-9a68703702f8-cert\") pod \"ingress-canary-df2z5\" (UID: \"3062f537-9b22-4e15-97aa-9a68703702f8\") " pod="openshift-ingress-canary/ingress-canary-df2z5"
Oct 30 06:56:53.531677 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.531593    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9c5ccc21-fce2-4221-9c19-4a35fef32eff-crio-socket\") pod \"insights-runtime-extractor-ltdxc\" (UID: \"9c5ccc21-fce2-4221-9c19-4a35fef32eff\") " pod="openshift-insights/insights-runtime-extractor-ltdxc"
Oct 30 06:56:53.531677 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.531609    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9c5ccc21-fce2-4221-9c19-4a35fef32eff-data-volume\") pod \"insights-runtime-extractor-ltdxc\" (UID: \"9c5ccc21-fce2-4221-9c19-4a35fef32eff\") " pod="openshift-insights/insights-runtime-extractor-ltdxc"
Oct 30 06:56:53.531848 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.531755    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c5ccc21-fce2-4221-9c19-4a35fef32eff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ltdxc\" (UID: \"9c5ccc21-fce2-4221-9c19-4a35fef32eff\") " pod="openshift-insights/insights-runtime-extractor-ltdxc"
Oct 30 06:56:53.531848 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.531798    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px9vv\" (UniqueName: \"kubernetes.io/projected/dda66b1a-94e1-40dd-a77d-bfe37dd22b55-kube-api-access-px9vv\") pod \"dns-default-dncbb\" (UID: \"dda66b1a-94e1-40dd-a77d-bfe37dd22b55\") " pod="openshift-dns/dns-default-dncbb"
Oct 30 06:56:53.531848 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.531835    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9c5ccc21-fce2-4221-9c19-4a35fef32eff-crio-socket\") pod \"insights-runtime-extractor-ltdxc\" (UID: \"9c5ccc21-fce2-4221-9c19-4a35fef32eff\") " pod="openshift-insights/insights-runtime-extractor-ltdxc"
Oct 30 06:56:53.531981 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.531847    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnrqt\" (UniqueName: \"kubernetes.io/projected/3062f537-9b22-4e15-97aa-9a68703702f8-kube-api-access-wnrqt\") pod \"ingress-canary-df2z5\" (UID: \"3062f537-9b22-4e15-97aa-9a68703702f8\") " pod="openshift-ingress-canary/ingress-canary-df2z5"
Oct 30 06:56:53.531981 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.531893    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbpk2\" (UniqueName: \"kubernetes.io/projected/9c5ccc21-fce2-4221-9c19-4a35fef32eff-kube-api-access-lbpk2\") pod \"insights-runtime-extractor-ltdxc\" (UID: \"9c5ccc21-fce2-4221-9c19-4a35fef32eff\") " pod="openshift-insights/insights-runtime-extractor-ltdxc"
Oct 30 06:56:53.531981 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.531928    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dda66b1a-94e1-40dd-a77d-bfe37dd22b55-metrics-tls\") pod \"dns-default-dncbb\" (UID: \"dda66b1a-94e1-40dd-a77d-bfe37dd22b55\") " pod="openshift-dns/dns-default-dncbb"
Oct 30 06:56:53.531981 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.531958    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dda66b1a-94e1-40dd-a77d-bfe37dd22b55-tmp-dir\") pod \"dns-default-dncbb\" (UID: \"dda66b1a-94e1-40dd-a77d-bfe37dd22b55\") " pod="openshift-dns/dns-default-dncbb"
Oct 30 06:56:53.531981 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.531975    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9c5ccc21-fce2-4221-9c19-4a35fef32eff-data-volume\") pod \"insights-runtime-extractor-ltdxc\" (UID: \"9c5ccc21-fce2-4221-9c19-4a35fef32eff\") " pod="openshift-insights/insights-runtime-extractor-ltdxc"
Oct 30 06:56:53.532269 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.531982    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9c5ccc21-fce2-4221-9c19-4a35fef32eff-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ltdxc\" (UID: \"9c5ccc21-fce2-4221-9c19-4a35fef32eff\") " pod="openshift-insights/insights-runtime-extractor-ltdxc"
Oct 30 06:56:53.532422 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.532394    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dda66b1a-94e1-40dd-a77d-bfe37dd22b55-tmp-dir\") pod \"dns-default-dncbb\" (UID: \"dda66b1a-94e1-40dd-a77d-bfe37dd22b55\") " pod="openshift-dns/dns-default-dncbb"
Oct 30 06:56:53.532544 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.532524    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9c5ccc21-fce2-4221-9c19-4a35fef32eff-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ltdxc\" (UID: \"9c5ccc21-fce2-4221-9c19-4a35fef32eff\") " pod="openshift-insights/insights-runtime-extractor-ltdxc"
Oct 30 06:56:53.532599 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.532529    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dda66b1a-94e1-40dd-a77d-bfe37dd22b55-config-volume\") pod \"dns-default-dncbb\" (UID: \"dda66b1a-94e1-40dd-a77d-bfe37dd22b55\") " pod="openshift-dns/dns-default-dncbb"
Oct 30 06:56:53.535555 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.535535    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dda66b1a-94e1-40dd-a77d-bfe37dd22b55-metrics-tls\") pod \"dns-default-dncbb\" (UID: \"dda66b1a-94e1-40dd-a77d-bfe37dd22b55\") " pod="openshift-dns/dns-default-dncbb"
Oct 30 06:56:53.535654 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.535618    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c5ccc21-fce2-4221-9c19-4a35fef32eff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ltdxc\" (UID: \"9c5ccc21-fce2-4221-9c19-4a35fef32eff\") " pod="openshift-insights/insights-runtime-extractor-ltdxc"
Oct 30 06:56:53.535734 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.535715    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3062f537-9b22-4e15-97aa-9a68703702f8-cert\") pod \"ingress-canary-df2z5\" (UID: \"3062f537-9b22-4e15-97aa-9a68703702f8\") " pod="openshift-ingress-canary/ingress-canary-df2z5"
Oct 30 06:56:53.553457 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.553438    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-px9vv\" (UniqueName: \"kubernetes.io/projected/dda66b1a-94e1-40dd-a77d-bfe37dd22b55-kube-api-access-px9vv\") pod \"dns-default-dncbb\" (UID: \"dda66b1a-94e1-40dd-a77d-bfe37dd22b55\") " pod="openshift-dns/dns-default-dncbb"
Oct 30 06:56:53.553596 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.553580    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnrqt\" (UniqueName: \"kubernetes.io/projected/3062f537-9b22-4e15-97aa-9a68703702f8-kube-api-access-wnrqt\") pod \"ingress-canary-df2z5\" (UID: \"3062f537-9b22-4e15-97aa-9a68703702f8\") " pod="openshift-ingress-canary/ingress-canary-df2z5"
Oct 30 06:56:53.553725 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.553701    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbpk2\" (UniqueName: \"kubernetes.io/projected/9c5ccc21-fce2-4221-9c19-4a35fef32eff-kube-api-access-lbpk2\") pod \"insights-runtime-extractor-ltdxc\" (UID: \"9c5ccc21-fce2-4221-9c19-4a35fef32eff\") " pod="openshift-insights/insights-runtime-extractor-ltdxc"
Oct 30 06:56:53.570564 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.570545    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ltdxc"
Oct 30 06:56:53.581973 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.581954    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dncbb"
Oct 30 06:56:53.585848 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.585825    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-df2z5"
Oct 30 06:56:53.889419 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.889274    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-df2z5"]
Oct 30 06:56:53.894372 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.894349    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ltdxc"]
Oct 30 06:56:53.894839 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:53.894797    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dncbb"]
Oct 30 06:56:53.897954 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:53.897933    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c5ccc21_fce2_4221_9c19_4a35fef32eff.slice/crio-ca2afa0103577be424a378670e76be6f79a1d7d5eccc4adf294a66963c7af7ab WatchSource:0}: Error finding container ca2afa0103577be424a378670e76be6f79a1d7d5eccc4adf294a66963c7af7ab: Status 404 returned error can't find the container with id ca2afa0103577be424a378670e76be6f79a1d7d5eccc4adf294a66963c7af7ab
Oct 30 06:56:53.898728 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:53.898708    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddda66b1a_94e1_40dd_a77d_bfe37dd22b55.slice/crio-93adb19d38e4ac992b2167c0faf831a897937178bfbddca1361abf56ba4cb941 WatchSource:0}: Error finding container 93adb19d38e4ac992b2167c0faf831a897937178bfbddca1361abf56ba4cb941: Status 404 returned error can't find the container with id 93adb19d38e4ac992b2167c0faf831a897937178bfbddca1361abf56ba4cb941
Oct 30 06:56:54.700330 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:54.700175    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ltdxc" event={"ID":"9c5ccc21-fce2-4221-9c19-4a35fef32eff","Type":"ContainerStarted","Data":"3d5da51a13159176074e03cdd1b07f99fd03359ade64019aed0ca0cb44dea01d"}
Oct 30 06:56:54.700330 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:54.700249    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ltdxc" event={"ID":"9c5ccc21-fce2-4221-9c19-4a35fef32eff","Type":"ContainerStarted","Data":"0d1ef848d7a933df59bf7307894fa9fab2c6d1e1f8c64fc6bd0393c3eceba670"}
Oct 30 06:56:54.700330 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:54.700277    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ltdxc" event={"ID":"9c5ccc21-fce2-4221-9c19-4a35fef32eff","Type":"ContainerStarted","Data":"ca2afa0103577be424a378670e76be6f79a1d7d5eccc4adf294a66963c7af7ab"}
Oct 30 06:56:54.702488 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:54.701901    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-df2z5" event={"ID":"3062f537-9b22-4e15-97aa-9a68703702f8","Type":"ContainerStarted","Data":"4cf0039f2b797f9938c56521778329a145c93299deea89534bff76814a1fb9cb"}
Oct 30 06:56:54.705906 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:54.704731    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dncbb" event={"ID":"dda66b1a-94e1-40dd-a77d-bfe37dd22b55","Type":"ContainerStarted","Data":"93adb19d38e4ac992b2167c0faf831a897937178bfbddca1361abf56ba4cb941"}
Oct 30 06:56:55.384293 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:55.384258    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5p67n_675dc1eb-ff05-40c7-bf0b-fb3161d3cc8c/dns-node-resolver/0.log"
Oct 30 06:56:56.286470 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:56.286273    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-xh6wz"]
Oct 30 06:56:56.288439 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:56.288423    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-xh6wz"
Oct 30 06:56:56.322077 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:56.322039    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 30 06:56:56.322234 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:56.322107    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 06:56:56.322307 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:56.322259    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 06:56:56.322381 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:56.322360    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-mxksr\""
Oct 30 06:56:56.332582 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:56.332509    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 06:56:56.333038 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:56.333016    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 30 06:56:56.334174 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:56.334154    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-xh6wz"]
Oct 30 06:56:56.354494 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:56.354462    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf410657-29d1-408b-a956-e7e5ef548a33-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-xh6wz\" (UID: \"cf410657-29d1-408b-a956-e7e5ef548a33\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xh6wz"
Oct 30 06:56:56.354651 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:56.354531    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr6w5\" (UniqueName: \"kubernetes.io/projected/cf410657-29d1-408b-a956-e7e5ef548a33-kube-api-access-gr6w5\") pod \"prometheus-operator-9cd89b74c-xh6wz\" (UID: \"cf410657-29d1-408b-a956-e7e5ef548a33\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xh6wz"
Oct 30 06:56:56.354651 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:56.354561    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cf410657-29d1-408b-a956-e7e5ef548a33-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-xh6wz\" (UID: \"cf410657-29d1-408b-a956-e7e5ef548a33\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xh6wz"
Oct 30 06:56:56.354651 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:56.354601    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cf410657-29d1-408b-a956-e7e5ef548a33-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-xh6wz\" (UID: \"cf410657-29d1-408b-a956-e7e5ef548a33\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xh6wz"
Oct 30 06:56:56.455324 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:56.455292    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr6w5\" (UniqueName: \"kubernetes.io/projected/cf410657-29d1-408b-a956-e7e5ef548a33-kube-api-access-gr6w5\") pod \"prometheus-operator-9cd89b74c-xh6wz\" (UID: \"cf410657-29d1-408b-a956-e7e5ef548a33\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xh6wz"
Oct 30 06:56:56.455324 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:56.455337    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cf410657-29d1-408b-a956-e7e5ef548a33-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-xh6wz\" (UID: \"cf410657-29d1-408b-a956-e7e5ef548a33\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xh6wz"
Oct 30 06:56:56.455525 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:56.455449    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cf410657-29d1-408b-a956-e7e5ef548a33-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-xh6wz\" (UID: \"cf410657-29d1-408b-a956-e7e5ef548a33\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xh6wz"
Oct 30 06:56:56.455525 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:56.455470    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf410657-29d1-408b-a956-e7e5ef548a33-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-xh6wz\" (UID: \"cf410657-29d1-408b-a956-e7e5ef548a33\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xh6wz"
Oct 30 06:56:56.456383 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:56.456357    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf410657-29d1-408b-a956-e7e5ef548a33-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-xh6wz\" (UID: \"cf410657-29d1-408b-a956-e7e5ef548a33\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xh6wz"
Oct 30 06:56:56.459421 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:56.459397    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cf410657-29d1-408b-a956-e7e5ef548a33-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-xh6wz\" (UID: \"cf410657-29d1-408b-a956-e7e5ef548a33\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xh6wz"
Oct 30 06:56:56.459421 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:56.459406    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cf410657-29d1-408b-a956-e7e5ef548a33-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-xh6wz\" (UID: \"cf410657-29d1-408b-a956-e7e5ef548a33\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xh6wz"
Oct 30 06:56:56.477229 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:56.477204    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr6w5\" (UniqueName: \"kubernetes.io/projected/cf410657-29d1-408b-a956-e7e5ef548a33-kube-api-access-gr6w5\") pod \"prometheus-operator-9cd89b74c-xh6wz\" (UID: \"cf410657-29d1-408b-a956-e7e5ef548a33\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xh6wz"
Oct 30 06:56:56.601731 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:56.601660    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-xh6wz"
Oct 30 06:56:56.713229 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:56.713195    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dncbb" event={"ID":"dda66b1a-94e1-40dd-a77d-bfe37dd22b55","Type":"ContainerStarted","Data":"324d34b1f361a9aab73661d2f6751c66368b72f3d14db3ff3ca7344f0bc3555d"}
Oct 30 06:56:56.713355 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:56.713257    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dncbb" event={"ID":"dda66b1a-94e1-40dd-a77d-bfe37dd22b55","Type":"ContainerStarted","Data":"e2156bd89c6c371e6095f04207dfd929e04d2b153a35745a7e0873293158000d"}
Oct 30 06:56:56.713355 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:56.713288    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-dncbb"
Oct 30 06:56:56.715524 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:56.715490    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ltdxc" event={"ID":"9c5ccc21-fce2-4221-9c19-4a35fef32eff","Type":"ContainerStarted","Data":"484240124ef942bfe63f409a07052b91b93873763b20263f80b6aa48274b7e51"}
Oct 30 06:56:56.716897 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:56.716873    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-df2z5" event={"ID":"3062f537-9b22-4e15-97aa-9a68703702f8","Type":"ContainerStarted","Data":"c9663caebabb981a4e1726fa00d04fa7ce06597cceba7b05b5456b56865e28f9"}
Oct 30 06:56:56.752027 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:56.751989    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-dncbb" podStartSLOduration=1.959717962 podStartE2EDuration="3.751977924s" totalImagesPullingTime="1.792259962s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:56:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:56.751544417 +0000 UTC m=+46.938858186" watchObservedRunningTime="2026-10-30 06:56:56.751977924 +0000 UTC m=+46.939291692"
Oct 30 06:56:56.783281 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:56.783262    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-l9mfn_5d87089b-8b1b-43ea-be1f-72bfb56bf311/node-ca/0.log"
Oct 30 06:56:56.789110 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:56.789076    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-df2z5" podStartSLOduration=1.198303252 podStartE2EDuration="3.789057094s" totalImagesPullingTime="2.590753842s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:56:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:56.788465378 +0000 UTC m=+46.975779146" watchObservedRunningTime="2026-10-30 06:56:56.789057094 +0000 UTC m=+46.976370857"
Oct 30 06:56:56.828303 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:56.828265    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-ltdxc" podStartSLOduration=1.797040574 podStartE2EDuration="3.828254212s" totalImagesPullingTime="2.031213638s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 06:56:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:56.827544584 +0000 UTC m=+47.014858355" watchObservedRunningTime="2026-10-30 06:56:56.828254212 +0000 UTC m=+47.015567980"
Oct 30 06:56:56.900484 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:56.900465    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-xh6wz"]
Oct 30 06:56:56.902513 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:56:56.902486    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf410657_29d1_408b_a956_e7e5ef548a33.slice/crio-919f82c3786444358be28752d8866ba97d19b73cf72883656d55e4b3c2b71592 WatchSource:0}: Error finding container 919f82c3786444358be28752d8866ba97d19b73cf72883656d55e4b3c2b71592: Status 404 returned error can't find the container with id 919f82c3786444358be28752d8866ba97d19b73cf72883656d55e4b3c2b71592
Oct 30 06:56:57.583975 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:57.583948    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-df2z5_3062f537-9b22-4e15-97aa-9a68703702f8/serve-healthcheck-canary/0.log"
Oct 30 06:56:57.721367 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:57.721326    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-xh6wz" event={"ID":"cf410657-29d1-408b-a956-e7e5ef548a33","Type":"ContainerStarted","Data":"919f82c3786444358be28752d8866ba97d19b73cf72883656d55e4b3c2b71592"}
Oct 30 06:56:58.728656 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:58.728616    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-xh6wz" event={"ID":"cf410657-29d1-408b-a956-e7e5ef548a33","Type":"ContainerStarted","Data":"fb5b5edc5d3c745db3b39f5b1e02835d9925d217f75f32e037fd79965d72ea1a"}
Oct 30 06:56:58.729119 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:58.728675    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-xh6wz" event={"ID":"cf410657-29d1-408b-a956-e7e5ef548a33","Type":"ContainerStarted","Data":"4b075632a854e37d9de8c9b96e61f55acfd56a70ae3a43ffba30da735bf970f4"}
Oct 30 06:56:58.767628 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:56:58.767584    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-xh6wz" podStartSLOduration=1.698259013 podStartE2EDuration="2.767569152s" totalImagesPullingTime="1.069310139s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:56:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:58.767080892 +0000 UTC m=+48.954394661" watchObservedRunningTime="2026-10-30 06:56:58.767569152 +0000 UTC m=+48.954882921"
Oct 30 06:57:01.014015 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.013980    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5c7875b857-wv9lf"]
Oct 30 06:57:01.016024 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.016010    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c7875b857-wv9lf"
Oct 30 06:57:01.031315 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.031278    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 30 06:57:01.031404 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.031317    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-vjmtx\""
Oct 30 06:57:01.031675 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.031660    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 30 06:57:01.031783 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.031755    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 06:57:01.031862 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.031832    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 30 06:57:01.032686 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.032652    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 06:57:01.032786 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.032767    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 30 06:57:01.032786 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.032775    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 30 06:57:01.044982 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.044962    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c7875b857-wv9lf"]
Oct 30 06:57:01.088056 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.088026    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9d24d508-ec6f-4165-8394-f17b09134b34-console-oauth-config\") pod \"console-5c7875b857-wv9lf\" (UID: \"9d24d508-ec6f-4165-8394-f17b09134b34\") " pod="openshift-console/console-5c7875b857-wv9lf"
Oct 30 06:57:01.088056 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.088058    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9d24d508-ec6f-4165-8394-f17b09134b34-service-ca\") pod \"console-5c7875b857-wv9lf\" (UID: \"9d24d508-ec6f-4165-8394-f17b09134b34\") " pod="openshift-console/console-5c7875b857-wv9lf"
Oct 30 06:57:01.088214 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.088090    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x67rl\" (UniqueName: \"kubernetes.io/projected/9d24d508-ec6f-4165-8394-f17b09134b34-kube-api-access-x67rl\") pod \"console-5c7875b857-wv9lf\" (UID: \"9d24d508-ec6f-4165-8394-f17b09134b34\") " pod="openshift-console/console-5c7875b857-wv9lf"
Oct 30 06:57:01.088214 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.088181    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9d24d508-ec6f-4165-8394-f17b09134b34-console-serving-cert\") pod \"console-5c7875b857-wv9lf\" (UID: \"9d24d508-ec6f-4165-8394-f17b09134b34\") " pod="openshift-console/console-5c7875b857-wv9lf"
Oct 30 06:57:01.088214 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.088205    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9d24d508-ec6f-4165-8394-f17b09134b34-console-config\") pod \"console-5c7875b857-wv9lf\" (UID: \"9d24d508-ec6f-4165-8394-f17b09134b34\") " pod="openshift-console/console-5c7875b857-wv9lf"
Oct 30 06:57:01.088301 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.088223    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9d24d508-ec6f-4165-8394-f17b09134b34-oauth-serving-cert\") pod \"console-5c7875b857-wv9lf\" (UID: \"9d24d508-ec6f-4165-8394-f17b09134b34\") " pod="openshift-console/console-5c7875b857-wv9lf"
Oct 30 06:57:01.088357 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.088338    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d24d508-ec6f-4165-8394-f17b09134b34-tmp\") pod \"console-5c7875b857-wv9lf\" (UID: \"9d24d508-ec6f-4165-8394-f17b09134b34\") " pod="openshift-console/console-5c7875b857-wv9lf"
Oct 30 06:57:01.134299 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.134263    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-n7zk9"]
Oct 30 06:57:01.136353 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.136334    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-n7zk9"
Oct 30 06:57:01.188757 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.188728    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f7666c14-ff3f-44a5-9cce-21ff2dfda0e2-node-exporter-tls\") pod \"node-exporter-n7zk9\" (UID: \"f7666c14-ff3f-44a5-9cce-21ff2dfda0e2\") " pod="openshift-monitoring/node-exporter-n7zk9"
Oct 30 06:57:01.188909 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.188783    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9d24d508-ec6f-4165-8394-f17b09134b34-console-oauth-config\") pod \"console-5c7875b857-wv9lf\" (UID: \"9d24d508-ec6f-4165-8394-f17b09134b34\") " pod="openshift-console/console-5c7875b857-wv9lf"
Oct 30 06:57:01.188909 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.188803    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9d24d508-ec6f-4165-8394-f17b09134b34-service-ca\") pod \"console-5c7875b857-wv9lf\" (UID: \"9d24d508-ec6f-4165-8394-f17b09134b34\") " pod="openshift-console/console-5c7875b857-wv9lf"
Oct 30 06:57:01.188909 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.188877    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f7666c14-ff3f-44a5-9cce-21ff2dfda0e2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n7zk9\" (UID: \"f7666c14-ff3f-44a5-9cce-21ff2dfda0e2\") " pod="openshift-monitoring/node-exporter-n7zk9"
Oct 30 06:57:01.189031 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.188918    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9d24d508-ec6f-4165-8394-f17b09134b34-console-serving-cert\") pod \"console-5c7875b857-wv9lf\" (UID: \"9d24d508-ec6f-4165-8394-f17b09134b34\") " pod="openshift-console/console-5c7875b857-wv9lf"
Oct 30 06:57:01.189076 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.189032    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9d24d508-ec6f-4165-8394-f17b09134b34-console-config\") pod \"console-5c7875b857-wv9lf\" (UID: \"9d24d508-ec6f-4165-8394-f17b09134b34\") " pod="openshift-console/console-5c7875b857-wv9lf"
Oct 30 06:57:01.189127 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.189097    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f7666c14-ff3f-44a5-9cce-21ff2dfda0e2-sys\") pod \"node-exporter-n7zk9\" (UID: \"f7666c14-ff3f-44a5-9cce-21ff2dfda0e2\") " pod="openshift-monitoring/node-exporter-n7zk9"
Oct 30 06:57:01.189180 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.189142    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srs22\" (UniqueName: \"kubernetes.io/projected/f7666c14-ff3f-44a5-9cce-21ff2dfda0e2-kube-api-access-srs22\") pod \"node-exporter-n7zk9\" (UID: \"f7666c14-ff3f-44a5-9cce-21ff2dfda0e2\") " pod="openshift-monitoring/node-exporter-n7zk9"
Oct 30 06:57:01.189180 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.189169    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7666c14-ff3f-44a5-9cce-21ff2dfda0e2-metrics-client-ca\") pod \"node-exporter-n7zk9\" (UID: \"f7666c14-ff3f-44a5-9cce-21ff2dfda0e2\") " pod="openshift-monitoring/node-exporter-n7zk9"
Oct 30 06:57:01.189294 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.189270    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x67rl\" (UniqueName: \"kubernetes.io/projected/9d24d508-ec6f-4165-8394-f17b09134b34-kube-api-access-x67rl\") pod \"console-5c7875b857-wv9lf\" (UID: \"9d24d508-ec6f-4165-8394-f17b09134b34\") " pod="openshift-console/console-5c7875b857-wv9lf"
Oct 30 06:57:01.189350 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.189310    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f7666c14-ff3f-44a5-9cce-21ff2dfda0e2-node-exporter-textfile\") pod \"node-exporter-n7zk9\" (UID: \"f7666c14-ff3f-44a5-9cce-21ff2dfda0e2\") " pod="openshift-monitoring/node-exporter-n7zk9"
Oct 30 06:57:01.189399 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.189371    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9d24d508-ec6f-4165-8394-f17b09134b34-oauth-serving-cert\") pod \"console-5c7875b857-wv9lf\" (UID: \"9d24d508-ec6f-4165-8394-f17b09134b34\") " pod="openshift-console/console-5c7875b857-wv9lf"
Oct 30 06:57:01.189452 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.189410    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f7666c14-ff3f-44a5-9cce-21ff2dfda0e2-root\") pod \"node-exporter-n7zk9\" (UID: \"f7666c14-ff3f-44a5-9cce-21ff2dfda0e2\") " pod="openshift-monitoring/node-exporter-n7zk9"
Oct 30 06:57:01.189452 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.189443    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f7666c14-ff3f-44a5-9cce-21ff2dfda0e2-node-exporter-wtmp\") pod \"node-exporter-n7zk9\" (UID: \"f7666c14-ff3f-44a5-9cce-21ff2dfda0e2\") " pod="openshift-monitoring/node-exporter-n7zk9"
Oct 30 06:57:01.189550 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.189492    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d24d508-ec6f-4165-8394-f17b09134b34-tmp\") pod \"console-5c7875b857-wv9lf\" (UID: \"9d24d508-ec6f-4165-8394-f17b09134b34\") " pod="openshift-console/console-5c7875b857-wv9lf"
Oct 30 06:57:01.189550 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.189521    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f7666c14-ff3f-44a5-9cce-21ff2dfda0e2-node-exporter-accelerators-collector-config\") pod \"node-exporter-n7zk9\" (UID: \"f7666c14-ff3f-44a5-9cce-21ff2dfda0e2\") " pod="openshift-monitoring/node-exporter-n7zk9"
Oct 30 06:57:01.189795 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.189778    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9d24d508-ec6f-4165-8394-f17b09134b34-service-ca\") pod \"console-5c7875b857-wv9lf\" (UID: \"9d24d508-ec6f-4165-8394-f17b09134b34\") " pod="openshift-console/console-5c7875b857-wv9lf"
Oct 30 06:57:01.189925 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.189905    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d24d508-ec6f-4165-8394-f17b09134b34-tmp\") pod \"console-5c7875b857-wv9lf\" (UID: \"9d24d508-ec6f-4165-8394-f17b09134b34\") " pod="openshift-console/console-5c7875b857-wv9lf"
Oct 30 06:57:01.189970 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.189948    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9d24d508-ec6f-4165-8394-f17b09134b34-console-config\") pod \"console-5c7875b857-wv9lf\" (UID: \"9d24d508-ec6f-4165-8394-f17b09134b34\") " pod="openshift-console/console-5c7875b857-wv9lf"
Oct 30 06:57:01.190037 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.190022    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9d24d508-ec6f-4165-8394-f17b09134b34-oauth-serving-cert\") pod \"console-5c7875b857-wv9lf\" (UID: \"9d24d508-ec6f-4165-8394-f17b09134b34\") " pod="openshift-console/console-5c7875b857-wv9lf"
Oct 30 06:57:01.191177 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.191158    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9d24d508-ec6f-4165-8394-f17b09134b34-console-oauth-config\") pod \"console-5c7875b857-wv9lf\" (UID: \"9d24d508-ec6f-4165-8394-f17b09134b34\") " pod="openshift-console/console-5c7875b857-wv9lf"
Oct 30 06:57:01.191332 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.191318    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9d24d508-ec6f-4165-8394-f17b09134b34-console-serving-cert\") pod \"console-5c7875b857-wv9lf\" (UID: \"9d24d508-ec6f-4165-8394-f17b09134b34\") " pod="openshift-console/console-5c7875b857-wv9lf"
Oct 30 06:57:01.210681 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.210654    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x67rl\" (UniqueName: \"kubernetes.io/projected/9d24d508-ec6f-4165-8394-f17b09134b34-kube-api-access-x67rl\") pod \"console-5c7875b857-wv9lf\" (UID: \"9d24d508-ec6f-4165-8394-f17b09134b34\") " pod="openshift-console/console-5c7875b857-wv9lf"
Oct 30 06:57:01.223957 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.223937    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 06:57:01.230703 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.230686    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 06:57:01.230903 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.230888    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-djnbg\""
Oct 30 06:57:01.231160 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.231147    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 06:57:01.290498 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.290476    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srs22\" (UniqueName: \"kubernetes.io/projected/f7666c14-ff3f-44a5-9cce-21ff2dfda0e2-kube-api-access-srs22\") pod \"node-exporter-n7zk9\" (UID: \"f7666c14-ff3f-44a5-9cce-21ff2dfda0e2\") " pod="openshift-monitoring/node-exporter-n7zk9"
Oct 30 06:57:01.290590 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.290502    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7666c14-ff3f-44a5-9cce-21ff2dfda0e2-metrics-client-ca\") pod \"node-exporter-n7zk9\" (UID: \"f7666c14-ff3f-44a5-9cce-21ff2dfda0e2\") " pod="openshift-monitoring/node-exporter-n7zk9"
Oct 30 06:57:01.290590 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.290529    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f7666c14-ff3f-44a5-9cce-21ff2dfda0e2-node-exporter-textfile\") pod \"node-exporter-n7zk9\" (UID: \"f7666c14-ff3f-44a5-9cce-21ff2dfda0e2\") " pod="openshift-monitoring/node-exporter-n7zk9"
Oct 30 06:57:01.290590 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.290548    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f7666c14-ff3f-44a5-9cce-21ff2dfda0e2-root\") pod \"node-exporter-n7zk9\" (UID: \"f7666c14-ff3f-44a5-9cce-21ff2dfda0e2\") " pod="openshift-monitoring/node-exporter-n7zk9"
Oct 30 06:57:01.290590 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.290565    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f7666c14-ff3f-44a5-9cce-21ff2dfda0e2-node-exporter-wtmp\") pod \"node-exporter-n7zk9\" (UID: \"f7666c14-ff3f-44a5-9cce-21ff2dfda0e2\") " pod="openshift-monitoring/node-exporter-n7zk9"
Oct 30 06:57:01.290721 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.290628    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f7666c14-ff3f-44a5-9cce-21ff2dfda0e2-root\") pod \"node-exporter-n7zk9\" (UID: \"f7666c14-ff3f-44a5-9cce-21ff2dfda0e2\") " pod="openshift-monitoring/node-exporter-n7zk9"
Oct 30 06:57:01.290721 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.290666    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f7666c14-ff3f-44a5-9cce-21ff2dfda0e2-node-exporter-accelerators-collector-config\") pod \"node-exporter-n7zk9\" (UID: \"f7666c14-ff3f-44a5-9cce-21ff2dfda0e2\") " pod="openshift-monitoring/node-exporter-n7zk9"
Oct 30 06:57:01.290721 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.290696    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f7666c14-ff3f-44a5-9cce-21ff2dfda0e2-node-exporter-tls\") pod \"node-exporter-n7zk9\" (UID: \"f7666c14-ff3f-44a5-9cce-21ff2dfda0e2\") " pod="openshift-monitoring/node-exporter-n7zk9"
Oct 30 06:57:01.290721 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.290710    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f7666c14-ff3f-44a5-9cce-21ff2dfda0e2-node-exporter-wtmp\") pod \"node-exporter-n7zk9\" (UID: \"f7666c14-ff3f-44a5-9cce-21ff2dfda0e2\") " pod="openshift-monitoring/node-exporter-n7zk9"
Oct 30 06:57:01.290900 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.290735    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f7666c14-ff3f-44a5-9cce-21ff2dfda0e2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n7zk9\" (UID: \"f7666c14-ff3f-44a5-9cce-21ff2dfda0e2\") " pod="openshift-monitoring/node-exporter-n7zk9"
Oct 30 06:57:01.290900 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.290769    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f7666c14-ff3f-44a5-9cce-21ff2dfda0e2-sys\") pod \"node-exporter-n7zk9\" (UID: \"f7666c14-ff3f-44a5-9cce-21ff2dfda0e2\") " pod="openshift-monitoring/node-exporter-n7zk9"
Oct 30 06:57:01.290900 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.290867    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f7666c14-ff3f-44a5-9cce-21ff2dfda0e2-sys\") pod \"node-exporter-n7zk9\" (UID: \"f7666c14-ff3f-44a5-9cce-21ff2dfda0e2\") " pod="openshift-monitoring/node-exporter-n7zk9"
Oct 30 06:57:01.290900 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.290870    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f7666c14-ff3f-44a5-9cce-21ff2dfda0e2-node-exporter-textfile\") pod \"node-exporter-n7zk9\" (UID: \"f7666c14-ff3f-44a5-9cce-21ff2dfda0e2\") " pod="openshift-monitoring/node-exporter-n7zk9"
Oct 30 06:57:01.291149 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.291131    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7666c14-ff3f-44a5-9cce-21ff2dfda0e2-metrics-client-ca\") pod \"node-exporter-n7zk9\" (UID: \"f7666c14-ff3f-44a5-9cce-21ff2dfda0e2\") " pod="openshift-monitoring/node-exporter-n7zk9"
Oct 30 06:57:01.291282 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.291263    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f7666c14-ff3f-44a5-9cce-21ff2dfda0e2-node-exporter-accelerators-collector-config\") pod \"node-exporter-n7zk9\" (UID: \"f7666c14-ff3f-44a5-9cce-21ff2dfda0e2\") " pod="openshift-monitoring/node-exporter-n7zk9"
Oct 30 06:57:01.292817 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.292785    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f7666c14-ff3f-44a5-9cce-21ff2dfda0e2-node-exporter-tls\") pod \"node-exporter-n7zk9\" (UID: \"f7666c14-ff3f-44a5-9cce-21ff2dfda0e2\") " pod="openshift-monitoring/node-exporter-n7zk9"
Oct 30 06:57:01.292886 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.292839    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f7666c14-ff3f-44a5-9cce-21ff2dfda0e2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n7zk9\" (UID: \"f7666c14-ff3f-44a5-9cce-21ff2dfda0e2\") " pod="openshift-monitoring/node-exporter-n7zk9"
Oct 30 06:57:01.313596 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.313575    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srs22\" (UniqueName: \"kubernetes.io/projected/f7666c14-ff3f-44a5-9cce-21ff2dfda0e2-kube-api-access-srs22\") pod \"node-exporter-n7zk9\" (UID: \"f7666c14-ff3f-44a5-9cce-21ff2dfda0e2\") " pod="openshift-monitoring/node-exporter-n7zk9"
Oct 30 06:57:01.325563 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.325541    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c7875b857-wv9lf"
Oct 30 06:57:01.449289 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.449263    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-n7zk9"
Oct 30 06:57:01.457918 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:57:01.457895    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7666c14_ff3f_44a5_9cce_21ff2dfda0e2.slice/crio-86c6ac0388a077eacf9d769b76ab3a06c1c1dea7ae0982f56c78f7da301fd226 WatchSource:0}: Error finding container 86c6ac0388a077eacf9d769b76ab3a06c1c1dea7ae0982f56c78f7da301fd226: Status 404 returned error can't find the container with id 86c6ac0388a077eacf9d769b76ab3a06c1c1dea7ae0982f56c78f7da301fd226
Oct 30 06:57:01.613619 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.613552    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c7875b857-wv9lf"]
Oct 30 06:57:01.616192 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:57:01.616162    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d24d508_ec6f_4165_8394_f17b09134b34.slice/crio-3af572320fb94341a428cb0d1ff52bf19e935a13aa08043036427d1aed82f43b WatchSource:0}: Error finding container 3af572320fb94341a428cb0d1ff52bf19e935a13aa08043036427d1aed82f43b: Status 404 returned error can't find the container with id 3af572320fb94341a428cb0d1ff52bf19e935a13aa08043036427d1aed82f43b
Oct 30 06:57:01.738766 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.738727    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n7zk9" event={"ID":"f7666c14-ff3f-44a5-9cce-21ff2dfda0e2","Type":"ContainerStarted","Data":"86c6ac0388a077eacf9d769b76ab3a06c1c1dea7ae0982f56c78f7da301fd226"}
Oct 30 06:57:01.739843 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:01.739823    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c7875b857-wv9lf" event={"ID":"9d24d508-ec6f-4165-8394-f17b09134b34","Type":"ContainerStarted","Data":"3af572320fb94341a428cb0d1ff52bf19e935a13aa08043036427d1aed82f43b"}
Oct 30 06:57:02.744626 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:02.744589    2580 generic.go:356] "Generic (PLEG): container finished" podID="f7666c14-ff3f-44a5-9cce-21ff2dfda0e2" containerID="e8fddbe735abe1593d14d52087d383e4c21cdfc06e1af64d5b9495ef30b4b862" exitCode=0
Oct 30 06:57:02.745068 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:02.744681    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n7zk9" event={"ID":"f7666c14-ff3f-44a5-9cce-21ff2dfda0e2","Type":"ContainerDied","Data":"e8fddbe735abe1593d14d52087d383e4c21cdfc06e1af64d5b9495ef30b4b862"}
Oct 30 06:57:03.751931 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:03.751896    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n7zk9" event={"ID":"f7666c14-ff3f-44a5-9cce-21ff2dfda0e2","Type":"ContainerStarted","Data":"6a81b1cfdd252b3679863818c93aee12e3571cdf2334f0744b8ff67238b23353"}
Oct 30 06:57:03.752417 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:03.751959    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n7zk9" event={"ID":"f7666c14-ff3f-44a5-9cce-21ff2dfda0e2","Type":"ContainerStarted","Data":"4e32ac210be52c6fee8a2cd678a3354d58e8f2b3700803597a8224f2334192ff"}
Oct 30 06:57:03.813752 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:03.813690    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-n7zk9" podStartSLOduration=1.948744678 podStartE2EDuration="2.813672374s" totalImagesPullingTime="789.899934ms" totalInitContainerRuntime="75.027762ms" isStatefulPod=true podCreationTimestamp="2026-10-30 06:57:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:57:03.803052379 +0000 UTC m=+53.990366147" watchObservedRunningTime="2026-10-30 06:57:03.813672374 +0000 UTC m=+54.000986148"
Oct 30 06:57:04.756794 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:04.756709    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c7875b857-wv9lf" event={"ID":"9d24d508-ec6f-4165-8394-f17b09134b34","Type":"ContainerStarted","Data":"49e9a6769599b7b0985a13e2cb66627a350ba5f98575e7facdc0ab191db03837"}
Oct 30 06:57:04.796462 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:04.796409    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5c7875b857-wv9lf" podStartSLOduration=1.9538780629999999 podStartE2EDuration="4.796398487s" totalImagesPullingTime="2.842520424s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:57:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:57:04.795659721 +0000 UTC m=+54.982973493" watchObservedRunningTime="2026-10-30 06:57:04.796398487 +0000 UTC m=+54.983712254"
Oct 30 06:57:05.799454 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:05.799421    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-769d4f7fd-5sbjs"]
Oct 30 06:57:05.804167 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:05.804142    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-769d4f7fd-5sbjs"
Oct 30 06:57:05.827077 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:05.827058    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 30 06:57:05.827207 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:05.827188    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c00279a4-113f-432d-8531-baa80c405f3a-secret-metrics-server-tls\") pod \"metrics-server-769d4f7fd-5sbjs\" (UID: \"c00279a4-113f-432d-8531-baa80c405f3a\") " pod="openshift-monitoring/metrics-server-769d4f7fd-5sbjs"
Oct 30 06:57:05.827261 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:05.827229    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 30 06:57:05.827261 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:05.827239    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c00279a4-113f-432d-8531-baa80c405f3a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-769d4f7fd-5sbjs\" (UID: \"c00279a4-113f-432d-8531-baa80c405f3a\") " pod="openshift-monitoring/metrics-server-769d4f7fd-5sbjs"
Oct 30 06:57:05.827341 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:05.827325    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mnxb\" (UniqueName: \"kubernetes.io/projected/c00279a4-113f-432d-8531-baa80c405f3a-kube-api-access-7mnxb\") pod \"metrics-server-769d4f7fd-5sbjs\" (UID: \"c00279a4-113f-432d-8531-baa80c405f3a\") " pod="openshift-monitoring/metrics-server-769d4f7fd-5sbjs"
Oct 30 06:57:05.827387 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:05.827377    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c00279a4-113f-432d-8531-baa80c405f3a-audit-log\") pod \"metrics-server-769d4f7fd-5sbjs\" (UID: \"c00279a4-113f-432d-8531-baa80c405f3a\") " pod="openshift-monitoring/metrics-server-769d4f7fd-5sbjs"
Oct 30 06:57:05.827439 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:05.827425    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c00279a4-113f-432d-8531-baa80c405f3a-client-ca-bundle\") pod \"metrics-server-769d4f7fd-5sbjs\" (UID: \"c00279a4-113f-432d-8531-baa80c405f3a\") " pod="openshift-monitoring/metrics-server-769d4f7fd-5sbjs"
Oct 30 06:57:05.827488 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:05.827447    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c00279a4-113f-432d-8531-baa80c405f3a-metrics-server-audit-profiles\") pod \"metrics-server-769d4f7fd-5sbjs\" (UID: \"c00279a4-113f-432d-8531-baa80c405f3a\") " pod="openshift-monitoring/metrics-server-769d4f7fd-5sbjs"
Oct 30 06:57:05.827488 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:05.827464    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c00279a4-113f-432d-8531-baa80c405f3a-secret-metrics-server-client-certs\") pod \"metrics-server-769d4f7fd-5sbjs\" (UID: \"c00279a4-113f-432d-8531-baa80c405f3a\") " pod="openshift-monitoring/metrics-server-769d4f7fd-5sbjs"
Oct 30 06:57:05.833886 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:05.833863    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-4wfx8\""
Oct 30 06:57:05.833886 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:05.833865    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 30 06:57:05.834060 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:05.833889    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 30 06:57:05.838347 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:05.838332    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-ac0ke4d4dbh3m\""
Oct 30 06:57:05.839196 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:05.839180    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-769d4f7fd-5sbjs"]
Oct 30 06:57:05.928936 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:05.928909    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c00279a4-113f-432d-8531-baa80c405f3a-client-ca-bundle\") pod \"metrics-server-769d4f7fd-5sbjs\" (UID: \"c00279a4-113f-432d-8531-baa80c405f3a\") " pod="openshift-monitoring/metrics-server-769d4f7fd-5sbjs"
Oct 30 06:57:05.929062 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:05.928944    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c00279a4-113f-432d-8531-baa80c405f3a-metrics-server-audit-profiles\") pod \"metrics-server-769d4f7fd-5sbjs\" (UID: \"c00279a4-113f-432d-8531-baa80c405f3a\") " pod="openshift-monitoring/metrics-server-769d4f7fd-5sbjs"
Oct 30 06:57:05.929062 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:05.928964    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c00279a4-113f-432d-8531-baa80c405f3a-secret-metrics-server-client-certs\") pod \"metrics-server-769d4f7fd-5sbjs\" (UID: \"c00279a4-113f-432d-8531-baa80c405f3a\") " pod="openshift-monitoring/metrics-server-769d4f7fd-5sbjs"
Oct 30 06:57:05.929062 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:05.928988    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c00279a4-113f-432d-8531-baa80c405f3a-secret-metrics-server-tls\") pod \"metrics-server-769d4f7fd-5sbjs\" (UID: \"c00279a4-113f-432d-8531-baa80c405f3a\") " pod="openshift-monitoring/metrics-server-769d4f7fd-5sbjs"
Oct 30 06:57:05.929184 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:05.929152    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c00279a4-113f-432d-8531-baa80c405f3a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-769d4f7fd-5sbjs\" (UID: \"c00279a4-113f-432d-8531-baa80c405f3a\") " pod="openshift-monitoring/metrics-server-769d4f7fd-5sbjs"
Oct 30 06:57:05.929228 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:05.929204    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mnxb\" (UniqueName: \"kubernetes.io/projected/c00279a4-113f-432d-8531-baa80c405f3a-kube-api-access-7mnxb\") pod \"metrics-server-769d4f7fd-5sbjs\" (UID: \"c00279a4-113f-432d-8531-baa80c405f3a\") " pod="openshift-monitoring/metrics-server-769d4f7fd-5sbjs"
Oct 30 06:57:05.929277 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:05.929229    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c00279a4-113f-432d-8531-baa80c405f3a-audit-log\") pod \"metrics-server-769d4f7fd-5sbjs\" (UID: \"c00279a4-113f-432d-8531-baa80c405f3a\") " pod="openshift-monitoring/metrics-server-769d4f7fd-5sbjs"
Oct 30 06:57:05.929723 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:05.929689    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c00279a4-113f-432d-8531-baa80c405f3a-audit-log\") pod \"metrics-server-769d4f7fd-5sbjs\" (UID: \"c00279a4-113f-432d-8531-baa80c405f3a\") " pod="openshift-monitoring/metrics-server-769d4f7fd-5sbjs"
Oct 30 06:57:05.929902 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:05.929885    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c00279a4-113f-432d-8531-baa80c405f3a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-769d4f7fd-5sbjs\" (UID: \"c00279a4-113f-432d-8531-baa80c405f3a\") " pod="openshift-monitoring/metrics-server-769d4f7fd-5sbjs"
Oct 30 06:57:05.930090 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:05.930065    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c00279a4-113f-432d-8531-baa80c405f3a-metrics-server-audit-profiles\") pod \"metrics-server-769d4f7fd-5sbjs\" (UID: \"c00279a4-113f-432d-8531-baa80c405f3a\") " pod="openshift-monitoring/metrics-server-769d4f7fd-5sbjs"
Oct 30 06:57:05.931458 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:05.931432    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c00279a4-113f-432d-8531-baa80c405f3a-client-ca-bundle\") pod \"metrics-server-769d4f7fd-5sbjs\" (UID: \"c00279a4-113f-432d-8531-baa80c405f3a\") " pod="openshift-monitoring/metrics-server-769d4f7fd-5sbjs"
Oct 30 06:57:05.931538 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:05.931482    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c00279a4-113f-432d-8531-baa80c405f3a-secret-metrics-server-tls\") pod \"metrics-server-769d4f7fd-5sbjs\" (UID: \"c00279a4-113f-432d-8531-baa80c405f3a\") " pod="openshift-monitoring/metrics-server-769d4f7fd-5sbjs"
Oct 30 06:57:05.931538 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:05.931523    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c00279a4-113f-432d-8531-baa80c405f3a-secret-metrics-server-client-certs\") pod \"metrics-server-769d4f7fd-5sbjs\" (UID: \"c00279a4-113f-432d-8531-baa80c405f3a\") " pod="openshift-monitoring/metrics-server-769d4f7fd-5sbjs"
Oct 30 06:57:05.950568 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:05.950546    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mnxb\" (UniqueName: \"kubernetes.io/projected/c00279a4-113f-432d-8531-baa80c405f3a-kube-api-access-7mnxb\") pod \"metrics-server-769d4f7fd-5sbjs\" (UID: \"c00279a4-113f-432d-8531-baa80c405f3a\") " pod="openshift-monitoring/metrics-server-769d4f7fd-5sbjs"
Oct 30 06:57:06.114442 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:06.114374    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-769d4f7fd-5sbjs"
Oct 30 06:57:06.416073 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:06.416052    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-769d4f7fd-5sbjs"]
Oct 30 06:57:06.417694 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:57:06.417669    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc00279a4_113f_432d_8531_baa80c405f3a.slice/crio-562e7614715b08bbae86360ec40dcac18e315ad97b411cf869f6f82cf0d30116 WatchSource:0}: Error finding container 562e7614715b08bbae86360ec40dcac18e315ad97b411cf869f6f82cf0d30116: Status 404 returned error can't find the container with id 562e7614715b08bbae86360ec40dcac18e315ad97b411cf869f6f82cf0d30116
Oct 30 06:57:06.550459 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:06.550425    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-766978d4bc-rqpdj"]
Oct 30 06:57:06.554717 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:06.554697    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-766978d4bc-rqpdj"
Oct 30 06:57:06.568986 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:06.568948    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-5br4m\""
Oct 30 06:57:06.568986 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:06.568956    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Oct 30 06:57:06.568986 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:06.568980    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Oct 30 06:57:06.569321 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:06.569096    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Oct 30 06:57:06.569321 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:06.569169    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Oct 30 06:57:06.569597 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:06.569578    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Oct 30 06:57:06.591225 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:06.591197    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Oct 30 06:57:06.592363 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:06.592346    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-766978d4bc-rqpdj"]
Oct 30 06:57:06.634766 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:06.634743    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f09eaaa4-5ed2-4bee-931f-b425019f9c10-federate-client-tls\") pod \"telemeter-client-766978d4bc-rqpdj\" (UID: \"f09eaaa4-5ed2-4bee-931f-b425019f9c10\") " pod="openshift-monitoring/telemeter-client-766978d4bc-rqpdj"
Oct 30 06:57:06.634873 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:06.634777    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f09eaaa4-5ed2-4bee-931f-b425019f9c10-telemeter-client-tls\") pod \"telemeter-client-766978d4bc-rqpdj\" (UID: \"f09eaaa4-5ed2-4bee-931f-b425019f9c10\") " pod="openshift-monitoring/telemeter-client-766978d4bc-rqpdj"
Oct 30 06:57:06.634873 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:06.634793    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f09eaaa4-5ed2-4bee-931f-b425019f9c10-metrics-client-ca\") pod \"telemeter-client-766978d4bc-rqpdj\" (UID: \"f09eaaa4-5ed2-4bee-931f-b425019f9c10\") " pod="openshift-monitoring/telemeter-client-766978d4bc-rqpdj"
Oct 30 06:57:06.634873 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:06.634835    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f09eaaa4-5ed2-4bee-931f-b425019f9c10-serving-certs-ca-bundle\") pod \"telemeter-client-766978d4bc-rqpdj\" (UID: \"f09eaaa4-5ed2-4bee-931f-b425019f9c10\") " pod="openshift-monitoring/telemeter-client-766978d4bc-rqpdj"
Oct 30 06:57:06.634968 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:06.634921    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f09eaaa4-5ed2-4bee-931f-b425019f9c10-telemeter-trusted-ca-bundle\") pod \"telemeter-client-766978d4bc-rqpdj\" (UID: \"f09eaaa4-5ed2-4bee-931f-b425019f9c10\") " pod="openshift-monitoring/telemeter-client-766978d4bc-rqpdj"
Oct 30 06:57:06.634968 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:06.634951    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f09eaaa4-5ed2-4bee-931f-b425019f9c10-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-766978d4bc-rqpdj\" (UID: \"f09eaaa4-5ed2-4bee-931f-b425019f9c10\") " pod="openshift-monitoring/telemeter-client-766978d4bc-rqpdj"
Oct 30 06:57:06.635029 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:06.634973    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjw82\" (UniqueName: \"kubernetes.io/projected/f09eaaa4-5ed2-4bee-931f-b425019f9c10-kube-api-access-cjw82\") pod \"telemeter-client-766978d4bc-rqpdj\" (UID: \"f09eaaa4-5ed2-4bee-931f-b425019f9c10\") " pod="openshift-monitoring/telemeter-client-766978d4bc-rqpdj"
Oct 30 06:57:06.635075 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:06.635058    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f09eaaa4-5ed2-4bee-931f-b425019f9c10-secret-telemeter-client\") pod \"telemeter-client-766978d4bc-rqpdj\" (UID: \"f09eaaa4-5ed2-4bee-931f-b425019f9c10\") " pod="openshift-monitoring/telemeter-client-766978d4bc-rqpdj"
Oct 30 06:57:06.723406 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:06.723342    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dncbb"
Oct 30 06:57:06.735973 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:06.735954    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f09eaaa4-5ed2-4bee-931f-b425019f9c10-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-766978d4bc-rqpdj\" (UID: \"f09eaaa4-5ed2-4bee-931f-b425019f9c10\") " pod="openshift-monitoring/telemeter-client-766978d4bc-rqpdj"
Oct 30 06:57:06.736044 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:06.735986    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjw82\" (UniqueName: \"kubernetes.io/projected/f09eaaa4-5ed2-4bee-931f-b425019f9c10-kube-api-access-cjw82\") pod \"telemeter-client-766978d4bc-rqpdj\" (UID: \"f09eaaa4-5ed2-4bee-931f-b425019f9c10\") " pod="openshift-monitoring/telemeter-client-766978d4bc-rqpdj"
Oct 30 06:57:06.736044 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:06.736017    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f09eaaa4-5ed2-4bee-931f-b425019f9c10-secret-telemeter-client\") pod \"telemeter-client-766978d4bc-rqpdj\" (UID: \"f09eaaa4-5ed2-4bee-931f-b425019f9c10\") " pod="openshift-monitoring/telemeter-client-766978d4bc-rqpdj"
Oct 30 06:57:06.736116 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:06.736082    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f09eaaa4-5ed2-4bee-931f-b425019f9c10-federate-client-tls\") pod \"telemeter-client-766978d4bc-rqpdj\" (UID: \"f09eaaa4-5ed2-4bee-931f-b425019f9c10\") " pod="openshift-monitoring/telemeter-client-766978d4bc-rqpdj"
Oct 30 06:57:06.736154 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:06.736116    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f09eaaa4-5ed2-4bee-931f-b425019f9c10-telemeter-client-tls\") pod \"telemeter-client-766978d4bc-rqpdj\" (UID: \"f09eaaa4-5ed2-4bee-931f-b425019f9c10\") " pod="openshift-monitoring/telemeter-client-766978d4bc-rqpdj"
Oct 30 06:57:06.736154 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:06.736142    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f09eaaa4-5ed2-4bee-931f-b425019f9c10-metrics-client-ca\") pod \"telemeter-client-766978d4bc-rqpdj\" (UID: \"f09eaaa4-5ed2-4bee-931f-b425019f9c10\") " pod="openshift-monitoring/telemeter-client-766978d4bc-rqpdj"
Oct 30 06:57:06.736313 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:06.736176    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f09eaaa4-5ed2-4bee-931f-b425019f9c10-serving-certs-ca-bundle\") pod \"telemeter-client-766978d4bc-rqpdj\" (UID: \"f09eaaa4-5ed2-4bee-931f-b425019f9c10\") " pod="openshift-monitoring/telemeter-client-766978d4bc-rqpdj"
Oct 30 06:57:06.736313 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:06.736221    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f09eaaa4-5ed2-4bee-931f-b425019f9c10-telemeter-trusted-ca-bundle\") pod \"telemeter-client-766978d4bc-rqpdj\" (UID: \"f09eaaa4-5ed2-4bee-931f-b425019f9c10\") " pod="openshift-monitoring/telemeter-client-766978d4bc-rqpdj"
Oct 30 06:57:06.736981 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:06.736944    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f09eaaa4-5ed2-4bee-931f-b425019f9c10-metrics-client-ca\") pod \"telemeter-client-766978d4bc-rqpdj\" (UID: \"f09eaaa4-5ed2-4bee-931f-b425019f9c10\") " pod="openshift-monitoring/telemeter-client-766978d4bc-rqpdj"
Oct 30 06:57:06.736981 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:06.736950    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f09eaaa4-5ed2-4bee-931f-b425019f9c10-serving-certs-ca-bundle\") pod \"telemeter-client-766978d4bc-rqpdj\" (UID: \"f09eaaa4-5ed2-4bee-931f-b425019f9c10\") " pod="openshift-monitoring/telemeter-client-766978d4bc-rqpdj"
Oct 30 06:57:06.737130 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:06.737107    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f09eaaa4-5ed2-4bee-931f-b425019f9c10-telemeter-trusted-ca-bundle\") pod \"telemeter-client-766978d4bc-rqpdj\" (UID: \"f09eaaa4-5ed2-4bee-931f-b425019f9c10\") " pod="openshift-monitoring/telemeter-client-766978d4bc-rqpdj"
Oct 30 06:57:06.738607 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:06.738590    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f09eaaa4-5ed2-4bee-931f-b425019f9c10-secret-telemeter-client\") pod \"telemeter-client-766978d4bc-rqpdj\" (UID: \"f09eaaa4-5ed2-4bee-931f-b425019f9c10\") " pod="openshift-monitoring/telemeter-client-766978d4bc-rqpdj"
Oct 30 06:57:06.738667 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:06.738649    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f09eaaa4-5ed2-4bee-931f-b425019f9c10-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-766978d4bc-rqpdj\" (UID: \"f09eaaa4-5ed2-4bee-931f-b425019f9c10\") " pod="openshift-monitoring/telemeter-client-766978d4bc-rqpdj"
Oct 30 06:57:06.739017 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:06.739001    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f09eaaa4-5ed2-4bee-931f-b425019f9c10-federate-client-tls\") pod \"telemeter-client-766978d4bc-rqpdj\" (UID: \"f09eaaa4-5ed2-4bee-931f-b425019f9c10\") " pod="openshift-monitoring/telemeter-client-766978d4bc-rqpdj"
Oct 30 06:57:06.739084 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:06.739068    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f09eaaa4-5ed2-4bee-931f-b425019f9c10-telemeter-client-tls\") pod \"telemeter-client-766978d4bc-rqpdj\" (UID: \"f09eaaa4-5ed2-4bee-931f-b425019f9c10\") " pod="openshift-monitoring/telemeter-client-766978d4bc-rqpdj"
Oct 30 06:57:06.760782 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:06.760763    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjw82\" (UniqueName: \"kubernetes.io/projected/f09eaaa4-5ed2-4bee-931f-b425019f9c10-kube-api-access-cjw82\") pod \"telemeter-client-766978d4bc-rqpdj\" (UID: \"f09eaaa4-5ed2-4bee-931f-b425019f9c10\") " pod="openshift-monitoring/telemeter-client-766978d4bc-rqpdj"
Oct 30 06:57:06.764668 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:06.764646    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-769d4f7fd-5sbjs" event={"ID":"c00279a4-113f-432d-8531-baa80c405f3a","Type":"ContainerStarted","Data":"562e7614715b08bbae86360ec40dcac18e315ad97b411cf869f6f82cf0d30116"}
Oct 30 06:57:06.870456 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:06.870423    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-766978d4bc-rqpdj"
Oct 30 06:57:07.194599 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:07.194570    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-766978d4bc-rqpdj"]
Oct 30 06:57:07.197471 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:57:07.197430    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf09eaaa4_5ed2_4bee_931f_b425019f9c10.slice/crio-0d57907e4151320d3a4ce60e0d55b8fec5237232120b7685df5b8319b65099ca WatchSource:0}: Error finding container 0d57907e4151320d3a4ce60e0d55b8fec5237232120b7685df5b8319b65099ca: Status 404 returned error can't find the container with id 0d57907e4151320d3a4ce60e0d55b8fec5237232120b7685df5b8319b65099ca
Oct 30 06:57:07.772322 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:07.772288    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-769d4f7fd-5sbjs" event={"ID":"c00279a4-113f-432d-8531-baa80c405f3a","Type":"ContainerStarted","Data":"d4cdf0ff71732ae860a309f489832510ccbc3177a37a7c8c65ccb6ee08f5265d"}
Oct 30 06:57:07.774131 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:07.774102    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-766978d4bc-rqpdj" event={"ID":"f09eaaa4-5ed2-4bee-931f-b425019f9c10","Type":"ContainerStarted","Data":"0d57907e4151320d3a4ce60e0d55b8fec5237232120b7685df5b8319b65099ca"}
Oct 30 06:57:07.814209 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:07.814147    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-769d4f7fd-5sbjs" podStartSLOduration=1.64725693 podStartE2EDuration="2.814131505s" totalImagesPullingTime="1.166874575s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:57:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:57:07.813193493 +0000 UTC m=+58.000507263" watchObservedRunningTime="2026-10-30 06:57:07.814131505 +0000 UTC m=+58.001445275"
Oct 30 06:57:07.996846 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:07.996750    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 06:57:08.000905 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.000882    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:08.037247 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.037220    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 06:57:08.037549 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.037532    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 06:57:08.037681 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.037597    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 06:57:08.050959 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.050307    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:08.050959 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.050355    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:08.050959 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.050381    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 06:57:08.050959 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.050416    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:08.050959 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.050440    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:08.050959 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.050460    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 06:57:08.050959 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.050467    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:08.050959 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.050477    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 06:57:08.050959 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.050495    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:08.050959 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.050518    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:08.050959 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.050539    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 06:57:08.050959 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.050548    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:08.050959 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.050574    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 06:57:08.050959 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.050574    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-config-out\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:08.050959 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.050608    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 06:57:08.050959 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.050627    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9bkg\" (UniqueName: \"kubernetes.io/projected/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-kube-api-access-n9bkg\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:08.050959 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.050663    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 06:57:08.050959 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.050695    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-6rnqp\""
Oct 30 06:57:08.050959 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.050665    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 06:57:08.050959 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.050792    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 06:57:08.050959 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.050800    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:08.050959 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.050796    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 06:57:08.050959 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.050843    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-e9mv09h28asd\""
Oct 30 06:57:08.052061 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.050883    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:08.052061 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.050898    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 06:57:08.052061 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.050934    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-web-config\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:08.052061 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.051022    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:08.052061 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.051049    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:08.052061 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.051079    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-config\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:08.052061 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.051102    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:08.052061 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.051134    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:08.052061 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.051176    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:08.052061 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.051204    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:08.058538 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.058361    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 06:57:08.151802 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.151767    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:08.151980 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.151828    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:08.151980 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.151859    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:08.151980 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.151891    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:08.151980 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.151919    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:08.151980 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.151950    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:08.151980 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.151978    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-config-out\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:08.152254 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.152003    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9bkg\" (UniqueName: \"kubernetes.io/projected/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-kube-api-access-n9bkg\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:08.152254 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.152033    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:08.152254 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.152058    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:08.152254 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.152109    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-web-config\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:08.152254 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.152135    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:08.152254 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.152158    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:08.152254 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.152187    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-config\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:08.152254 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.152218    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:08.152254 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.152255    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:08.152657 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.152298    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:08.152657 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.152327    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:08.152657 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.152357    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:08.152657 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.152393    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:08.152657 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.152505    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:08.152657 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.152612    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:08.153259 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.153234    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:08.158784 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.155566    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:08.158784 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.156038    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:08.158784 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.156726    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:08.158784 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.157315    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:08.160892 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.160863    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:08.162034 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.161938    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:08.162899 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.162852    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:08.163437 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.163410    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-config-out\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:08.164003 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.163976    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-config\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:08.164161 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.164001    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:08.164497 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.164476    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:08.164589 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.164478    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:08.164894 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.164870    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:08.166305 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.166281    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:08.166870 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.166723    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:08.166870 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.166826    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-web-config\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:08.177396 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.177350    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9bkg\" (UniqueName: \"kubernetes.io/projected/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-kube-api-access-n9bkg\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:08.332170 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.332133    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:08.675057 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.674975    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 06:57:08.687577 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:57:08.687544    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12c298a7_e78a_45d7_abe1_7b7ef6f5df03.slice/crio-61eb4fbe80838432dc327dc06a8580e4a1d35a985986b0c2c113de76ac733a9c WatchSource:0}: Error finding container 61eb4fbe80838432dc327dc06a8580e4a1d35a985986b0c2c113de76ac733a9c: Status 404 returned error can't find the container with id 61eb4fbe80838432dc327dc06a8580e4a1d35a985986b0c2c113de76ac733a9c
Oct 30 06:57:08.779883 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:08.779850    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"12c298a7-e78a-45d7-abe1-7b7ef6f5df03","Type":"ContainerStarted","Data":"61eb4fbe80838432dc327dc06a8580e4a1d35a985986b0c2c113de76ac733a9c"}
Oct 30 06:57:09.784879 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:09.784767    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-766978d4bc-rqpdj" event={"ID":"f09eaaa4-5ed2-4bee-931f-b425019f9c10","Type":"ContainerStarted","Data":"79a628303731fa058815fed2bf12cc8af6f0e63e97e51b2c9624abbc99487b8a"}
Oct 30 06:57:10.795511 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:10.795480    2580 generic.go:356] "Generic (PLEG): container finished" podID="12c298a7-e78a-45d7-abe1-7b7ef6f5df03" containerID="7ea8feb9571f349fc2119c3f7c6e1421fe27e22e55f90a4b5370acc7af3d92bb" exitCode=0
Oct 30 06:57:10.795964 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:10.795573    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"12c298a7-e78a-45d7-abe1-7b7ef6f5df03","Type":"ContainerDied","Data":"7ea8feb9571f349fc2119c3f7c6e1421fe27e22e55f90a4b5370acc7af3d92bb"}
Oct 30 06:57:10.797866 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:10.797844    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-766978d4bc-rqpdj" event={"ID":"f09eaaa4-5ed2-4bee-931f-b425019f9c10","Type":"ContainerStarted","Data":"6304f8ecf641fb1a9551c79a3380e6a76a17e0a48366090d6c280274a3bee697"}
Oct 30 06:57:10.797975 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:10.797886    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-766978d4bc-rqpdj" event={"ID":"f09eaaa4-5ed2-4bee-931f-b425019f9c10","Type":"ContainerStarted","Data":"8cd287cfe505802c369bff9b4ceadb84738b34aad314eea268ccab74294a2dcd"}
Oct 30 06:57:10.921096 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:10.921047    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-766978d4bc-rqpdj" podStartSLOduration=2.278373064 podStartE2EDuration="4.921032792s" totalImagesPullingTime="2.642659728s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:57:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:57:10.920062467 +0000 UTC m=+61.107376235" watchObservedRunningTime="2026-10-30 06:57:10.921032792 +0000 UTC m=+61.108346561"
Oct 30 06:57:11.325644 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:11.325611    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5c7875b857-wv9lf"
Oct 30 06:57:11.325784 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:11.325675    2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5c7875b857-wv9lf"
Oct 30 06:57:11.330087 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:11.330066    2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5c7875b857-wv9lf"
Oct 30 06:57:11.806025 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:11.805998    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5c7875b857-wv9lf"
Oct 30 06:57:14.836238 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:14.836191    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"12c298a7-e78a-45d7-abe1-7b7ef6f5df03","Type":"ContainerStarted","Data":"26f5047a4a9d0850c61a1f7288ba0f74492952019ef4ac4beaec3e9e53c98040"}
Oct 30 06:57:14.836717 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:14.836293    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"12c298a7-e78a-45d7-abe1-7b7ef6f5df03","Type":"ContainerStarted","Data":"36e72f7afaaed1114f7178cf75aadb86bb06d852fa4d40991839fe3c650e7d00"}
Oct 30 06:57:16.144101 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:16.144066    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9092e2e8-ad23-426c-b6f1-14e1a2b9221e-original-pull-secret\") pod \"global-pull-secret-syncer-qktbg\" (UID: \"9092e2e8-ad23-426c-b6f1-14e1a2b9221e\") " pod="kube-system/global-pull-secret-syncer-qktbg"
Oct 30 06:57:16.144490 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:16.144153    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f8b75af-cc86-49c6-a66c-1798309f9806-metrics-certs\") pod \"network-metrics-daemon-r4d72\" (UID: \"6f8b75af-cc86-49c6-a66c-1798309f9806\") " pod="openshift-multus/network-metrics-daemon-r4d72"
Oct 30 06:57:16.167645 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:16.167615    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 06:57:16.167777 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:16.167719    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 06:57:16.177238 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:16.177210    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9092e2e8-ad23-426c-b6f1-14e1a2b9221e-original-pull-secret\") pod \"global-pull-secret-syncer-qktbg\" (UID: \"9092e2e8-ad23-426c-b6f1-14e1a2b9221e\") " pod="kube-system/global-pull-secret-syncer-qktbg"
Oct 30 06:57:16.177319 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:16.177258    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f8b75af-cc86-49c6-a66c-1798309f9806-metrics-certs\") pod \"network-metrics-daemon-r4d72\" (UID: \"6f8b75af-cc86-49c6-a66c-1798309f9806\") " pod="openshift-multus/network-metrics-daemon-r4d72"
Oct 30 06:57:16.212709 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:16.212688    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hkqzl\""
Oct 30 06:57:16.216563 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:16.216541    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-wlntw\""
Oct 30 06:57:16.220639 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:16.220620    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4d72"
Oct 30 06:57:16.225140 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:16.225104    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qktbg"
Oct 30 06:57:16.245285 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:16.244818    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cwrb\" (UniqueName: \"kubernetes.io/projected/6d493f94-3aef-445e-9b60-32a10c184eac-kube-api-access-2cwrb\") pod \"network-check-target-c6trh\" (UID: \"6d493f94-3aef-445e-9b60-32a10c184eac\") " pod="openshift-network-diagnostics/network-check-target-c6trh"
Oct 30 06:57:16.259840 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:16.257517    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 06:57:16.279292 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:16.279164    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 06:57:16.282301 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:16.282239    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cwrb\" (UniqueName: \"kubernetes.io/projected/6d493f94-3aef-445e-9b60-32a10c184eac-kube-api-access-2cwrb\") pod \"network-check-target-c6trh\" (UID: \"6d493f94-3aef-445e-9b60-32a10c184eac\") " pod="openshift-network-diagnostics/network-check-target-c6trh"
Oct 30 06:57:16.519627 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:16.519519    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-r4d72"]
Oct 30 06:57:16.521603 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:57:16.521572    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f8b75af_cc86_49c6_a66c_1798309f9806.slice/crio-fa5bcb3e624bd2241115c27f02a8efaebd5ed1c8d6d0851c075cd1001227e2b9 WatchSource:0}: Error finding container fa5bcb3e624bd2241115c27f02a8efaebd5ed1c8d6d0851c075cd1001227e2b9: Status 404 returned error can't find the container with id fa5bcb3e624bd2241115c27f02a8efaebd5ed1c8d6d0851c075cd1001227e2b9
Oct 30 06:57:16.521731 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:16.521656    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9hhpl\""
Oct 30 06:57:16.529650 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:16.529629    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6trh"
Oct 30 06:57:16.529799 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:16.529780    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qktbg"]
Oct 30 06:57:16.531696 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:57:16.531676    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9092e2e8_ad23_426c_b6f1_14e1a2b9221e.slice/crio-1d1240054d453d5b62713793962afa616b2b7c7a1c817ed63febd27be4ceeef9 WatchSource:0}: Error finding container 1d1240054d453d5b62713793962afa616b2b7c7a1c817ed63febd27be4ceeef9: Status 404 returned error can't find the container with id 1d1240054d453d5b62713793962afa616b2b7c7a1c817ed63febd27be4ceeef9
Oct 30 06:57:16.818960 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:16.818938    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-c6trh"]
Oct 30 06:57:16.821105 ip-10-0-128-125 kubenswrapper[2580]: W1030 06:57:16.821077    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d493f94_3aef_445e_9b60_32a10c184eac.slice/crio-39db0820745d212e03b98cd445e052d84820fb8ad42c98ed8fa2d370026ecd90 WatchSource:0}: Error finding container 39db0820745d212e03b98cd445e052d84820fb8ad42c98ed8fa2d370026ecd90: Status 404 returned error can't find the container with id 39db0820745d212e03b98cd445e052d84820fb8ad42c98ed8fa2d370026ecd90
Oct 30 06:57:16.844455 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:16.844419    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qktbg" event={"ID":"9092e2e8-ad23-426c-b6f1-14e1a2b9221e","Type":"ContainerStarted","Data":"1d1240054d453d5b62713793962afa616b2b7c7a1c817ed63febd27be4ceeef9"}
Oct 30 06:57:16.845500 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:16.845478    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-c6trh" event={"ID":"6d493f94-3aef-445e-9b60-32a10c184eac","Type":"ContainerStarted","Data":"39db0820745d212e03b98cd445e052d84820fb8ad42c98ed8fa2d370026ecd90"}
Oct 30 06:57:16.848822 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:16.848788    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"12c298a7-e78a-45d7-abe1-7b7ef6f5df03","Type":"ContainerStarted","Data":"82d8cccc3e759d6d5db1f4f50f4fdc6bb433649ad57238011dd1b6f318c733b7"}
Oct 30 06:57:16.848905 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:16.848875    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"12c298a7-e78a-45d7-abe1-7b7ef6f5df03","Type":"ContainerStarted","Data":"cecc64cca52be72b9ea23c344fb7d680a37d2d23b00465da4008fd8f08d83587"}
Oct 30 06:57:16.848954 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:16.848907    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"12c298a7-e78a-45d7-abe1-7b7ef6f5df03","Type":"ContainerStarted","Data":"ed4a02af2dab9b59fe33955fd09ad859f10218c846369b3f974b28f274e9e777"}
Oct 30 06:57:16.848954 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:16.848942    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"12c298a7-e78a-45d7-abe1-7b7ef6f5df03","Type":"ContainerStarted","Data":"88c9314ef41c71965d607688400aa8998adb43abe9255652d83e65e5e9d5e549"}
Oct 30 06:57:16.850112 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:16.850089    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r4d72" event={"ID":"6f8b75af-cc86-49c6-a66c-1798309f9806","Type":"ContainerStarted","Data":"fa5bcb3e624bd2241115c27f02a8efaebd5ed1c8d6d0851c075cd1001227e2b9"}
Oct 30 06:57:16.912665 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:16.912614    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.621387743 podStartE2EDuration="9.912597148s" totalImagesPullingTime="6.273084725s" totalInitContainerRuntime="18.12468ms" isStatefulPod=false podCreationTimestamp="2026-10-30 06:57:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:57:16.911685421 +0000 UTC m=+67.098999191" watchObservedRunningTime="2026-10-30 06:57:16.912597148 +0000 UTC m=+67.099910922"
Oct 30 06:57:17.425244 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:17.425211    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-f866fbc59-wnrl6"]
Oct 30 06:57:17.428603 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:17.428580    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f866fbc59-wnrl6"
Oct 30 06:57:17.458893 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:17.454344    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 30 06:57:17.458893 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:17.455712    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f866fbc59-wnrl6"]
Oct 30 06:57:17.556763 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:17.556731    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-service-ca\") pod \"console-f866fbc59-wnrl6\" (UID: \"bdc803a1-bbb2-407b-baa5-1aa3a54b7021\") " pod="openshift-console/console-f866fbc59-wnrl6"
Oct 30 06:57:17.556886 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:17.556773    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-oauth-serving-cert\") pod \"console-f866fbc59-wnrl6\" (UID: \"bdc803a1-bbb2-407b-baa5-1aa3a54b7021\") " pod="openshift-console/console-f866fbc59-wnrl6"
Oct 30 06:57:17.556886 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:17.556800    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-console-oauth-config\") pod \"console-f866fbc59-wnrl6\" (UID: \"bdc803a1-bbb2-407b-baa5-1aa3a54b7021\") " pod="openshift-console/console-f866fbc59-wnrl6"
Oct 30 06:57:17.556886 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:17.556833    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-trusted-ca-bundle\") pod \"console-f866fbc59-wnrl6\" (UID: \"bdc803a1-bbb2-407b-baa5-1aa3a54b7021\") " pod="openshift-console/console-f866fbc59-wnrl6"
Oct 30 06:57:17.556886 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:17.556865    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7cdq\" (UniqueName: \"kubernetes.io/projected/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-kube-api-access-q7cdq\") pod \"console-f866fbc59-wnrl6\" (UID: \"bdc803a1-bbb2-407b-baa5-1aa3a54b7021\") " pod="openshift-console/console-f866fbc59-wnrl6"
Oct 30 06:57:17.557096 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:17.556901    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-tmp\") pod \"console-f866fbc59-wnrl6\" (UID: \"bdc803a1-bbb2-407b-baa5-1aa3a54b7021\") " pod="openshift-console/console-f866fbc59-wnrl6"
Oct 30 06:57:17.557096 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:17.556937    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-console-serving-cert\") pod \"console-f866fbc59-wnrl6\" (UID: \"bdc803a1-bbb2-407b-baa5-1aa3a54b7021\") " pod="openshift-console/console-f866fbc59-wnrl6"
Oct 30 06:57:17.557096 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:17.557013    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-console-config\") pod \"console-f866fbc59-wnrl6\" (UID: \"bdc803a1-bbb2-407b-baa5-1aa3a54b7021\") " pod="openshift-console/console-f866fbc59-wnrl6"
Oct 30 06:57:17.657799 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:17.657764    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7cdq\" (UniqueName: \"kubernetes.io/projected/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-kube-api-access-q7cdq\") pod \"console-f866fbc59-wnrl6\" (UID: \"bdc803a1-bbb2-407b-baa5-1aa3a54b7021\") " pod="openshift-console/console-f866fbc59-wnrl6"
Oct 30 06:57:17.657930 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:17.657887    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-tmp\") pod \"console-f866fbc59-wnrl6\" (UID: \"bdc803a1-bbb2-407b-baa5-1aa3a54b7021\") " pod="openshift-console/console-f866fbc59-wnrl6"
Oct 30 06:57:17.657968 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:17.657927    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-console-serving-cert\") pod \"console-f866fbc59-wnrl6\" (UID: \"bdc803a1-bbb2-407b-baa5-1aa3a54b7021\") " pod="openshift-console/console-f866fbc59-wnrl6"
Oct 30 06:57:17.657998 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:17.657991    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-console-config\") pod \"console-f866fbc59-wnrl6\" (UID: \"bdc803a1-bbb2-407b-baa5-1aa3a54b7021\") " pod="openshift-console/console-f866fbc59-wnrl6"
Oct 30 06:57:17.658030 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:17.658020    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-service-ca\") pod \"console-f866fbc59-wnrl6\" (UID: \"bdc803a1-bbb2-407b-baa5-1aa3a54b7021\") " pod="openshift-console/console-f866fbc59-wnrl6"
Oct 30 06:57:17.658059 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:17.658044    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-oauth-serving-cert\") pod \"console-f866fbc59-wnrl6\" (UID: \"bdc803a1-bbb2-407b-baa5-1aa3a54b7021\") " pod="openshift-console/console-f866fbc59-wnrl6"
Oct 30 06:57:17.658092 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:17.658067    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-console-oauth-config\") pod \"console-f866fbc59-wnrl6\" (UID: \"bdc803a1-bbb2-407b-baa5-1aa3a54b7021\") " pod="openshift-console/console-f866fbc59-wnrl6"
Oct 30 06:57:17.658092 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:17.658081    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-trusted-ca-bundle\") pod \"console-f866fbc59-wnrl6\" (UID: \"bdc803a1-bbb2-407b-baa5-1aa3a54b7021\") " pod="openshift-console/console-f866fbc59-wnrl6"
Oct 30 06:57:17.658935 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:17.658894    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-trusted-ca-bundle\") pod \"console-f866fbc59-wnrl6\" (UID: \"bdc803a1-bbb2-407b-baa5-1aa3a54b7021\") " pod="openshift-console/console-f866fbc59-wnrl6"
Oct 30 06:57:17.659293 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:17.659260    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-tmp\") pod \"console-f866fbc59-wnrl6\" (UID: \"bdc803a1-bbb2-407b-baa5-1aa3a54b7021\") " pod="openshift-console/console-f866fbc59-wnrl6"
Oct 30 06:57:17.659518 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:17.659485    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-service-ca\") pod \"console-f866fbc59-wnrl6\" (UID: \"bdc803a1-bbb2-407b-baa5-1aa3a54b7021\") " pod="openshift-console/console-f866fbc59-wnrl6"
Oct 30 06:57:17.659955 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:17.659929    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-console-config\") pod \"console-f866fbc59-wnrl6\" (UID: \"bdc803a1-bbb2-407b-baa5-1aa3a54b7021\") " pod="openshift-console/console-f866fbc59-wnrl6"
Oct 30 06:57:17.660302 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:17.660281    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-oauth-serving-cert\") pod \"console-f866fbc59-wnrl6\" (UID: \"bdc803a1-bbb2-407b-baa5-1aa3a54b7021\") " pod="openshift-console/console-f866fbc59-wnrl6"
Oct 30 06:57:17.660678 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:17.660656    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-console-serving-cert\") pod \"console-f866fbc59-wnrl6\" (UID: \"bdc803a1-bbb2-407b-baa5-1aa3a54b7021\") " pod="openshift-console/console-f866fbc59-wnrl6"
Oct 30 06:57:17.660883 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:17.660865    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-console-oauth-config\") pod \"console-f866fbc59-wnrl6\" (UID: \"bdc803a1-bbb2-407b-baa5-1aa3a54b7021\") " pod="openshift-console/console-f866fbc59-wnrl6"
Oct 30 06:57:17.684878 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:17.684768    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7cdq\" (UniqueName: \"kubernetes.io/projected/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-kube-api-access-q7cdq\") pod \"console-f866fbc59-wnrl6\" (UID: \"bdc803a1-bbb2-407b-baa5-1aa3a54b7021\") " pod="openshift-console/console-f866fbc59-wnrl6"
Oct 30 06:57:17.742237 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:17.742204    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f866fbc59-wnrl6"
Oct 30 06:57:17.858284 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:17.858236    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r4d72" event={"ID":"6f8b75af-cc86-49c6-a66c-1798309f9806","Type":"ContainerStarted","Data":"cf99f888fe433c63811de12e1120c97f42e9467dd62a9b839732fe6e34d1eb04"}
Oct 30 06:57:17.858463 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:17.858350    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r4d72" event={"ID":"6f8b75af-cc86-49c6-a66c-1798309f9806","Type":"ContainerStarted","Data":"5d9166963f9e20e4031220e236e5b8a8d48d03c799d21c7f526d4a336c7942c6"}
Oct 30 06:57:17.895424 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:17.895366    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-r4d72" podStartSLOduration=65.936717544 podStartE2EDuration="1m6.895347956s" totalImagesPullingTime="958.630412ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:56:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:57:17.895212289 +0000 UTC m=+68.082526059" watchObservedRunningTime="2026-10-30 06:57:17.895347956 +0000 UTC m=+68.082661725"
Oct 30 06:57:18.046236 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:18.046206    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f866fbc59-wnrl6"]
Oct 30 06:57:18.333262 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:18.333192    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:18.864922 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:18.864878    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f866fbc59-wnrl6" event={"ID":"bdc803a1-bbb2-407b-baa5-1aa3a54b7021","Type":"ContainerStarted","Data":"63d5d853ed7c50fbf974b4fdbf7dcbdfff2c42308ef84ec2c05e635f404d02cc"}
Oct 30 06:57:18.865347 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:18.864933    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f866fbc59-wnrl6" event={"ID":"bdc803a1-bbb2-407b-baa5-1aa3a54b7021","Type":"ContainerStarted","Data":"12c2c077b5e8d30e4257b0ea0ae956cbbab35ee764ee7f301a9dc1e01d073c97"}
Oct 30 06:57:18.910050 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:18.909991    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-f866fbc59-wnrl6" podStartSLOduration=1.909969572 podStartE2EDuration="1.909969572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:57:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:57:18.908555387 +0000 UTC m=+69.095869160" watchObservedRunningTime="2026-10-30 06:57:18.909969572 +0000 UTC m=+69.097283341"
Oct 30 06:57:20.874896 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:20.874854    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-c6trh" event={"ID":"6d493f94-3aef-445e-9b60-32a10c184eac","Type":"ContainerStarted","Data":"1f93a00122683b9446370be703f5d80cf0ec851c778af91d3fc093169c398abe"}
Oct 30 06:57:20.875380 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:20.875011    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-c6trh"
Oct 30 06:57:20.914299 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:20.914247    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-c6trh" podStartSLOduration=66.786393854 podStartE2EDuration="1m9.914230531s" totalImagesPullingTime="3.127836677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:56:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:57:20.913208404 +0000 UTC m=+71.100522175" watchObservedRunningTime="2026-10-30 06:57:20.914230531 +0000 UTC m=+71.101544299"
Oct 30 06:57:21.334476 ip-10-0-128-125 kubenswrapper[2580]: I1030 06:57:21.334374    2580 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 06:57:21.334576 ip-10-0-128-125 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 06:57:21.345478 ip-10-0-128-125 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 06:57:21.345690 ip-10-0-128-125 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 06:57:21.345954 ip-10-0-128-125 systemd[1]: kubelet.service: Consumed 3.967s CPU time.
Oct 30 06:57:21.360377 ip-10-0-128-125 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 06:57:21.419897 ip-10-0-128-125 kubenswrapper[6224]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 06:57:21.419897 ip-10-0-128-125 kubenswrapper[6224]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 06:57:21.419897 ip-10-0-128-125 kubenswrapper[6224]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 06:57:21.419897 ip-10-0-128-125 kubenswrapper[6224]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 06:57:21.421867 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.421780    6224 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 06:57:21.421867 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.421867    6224 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 06:57:21.421939 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.421871    6224 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 06:57:21.421939 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.421875    6224 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 06:57:21.421939 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.421878    6224 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 06:57:21.421939 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.421881    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 06:57:21.421939 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.421883    6224 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 06:57:21.421939 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.421886    6224 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 06:57:21.421939 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.421889    6224 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 06:57:21.421939 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.421894    6224 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 06:57:21.421939 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.421898    6224 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 06:57:21.421939 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.421901    6224 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 06:57:21.421939 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.421908    6224 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 06:57:21.421939 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.421911    6224 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 06:57:21.421939 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.421914    6224 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 06:57:21.421939 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.421916    6224 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 06:57:21.421939 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.421919    6224 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 06:57:21.421939 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.421922    6224 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 06:57:21.421939 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.421925    6224 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 06:57:21.421939 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.421927    6224 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 06:57:21.422380 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.421932    6224 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 06:57:21.422380 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.421936    6224 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 06:57:21.422380 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.421940    6224 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 06:57:21.422380 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.421943    6224 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 06:57:21.422380 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.421946    6224 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 06:57:21.422380 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.421949    6224 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 06:57:21.422380 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.421951    6224 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 06:57:21.422380 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.421954    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 06:57:21.422380 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.421957    6224 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 06:57:21.422380 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.421960    6224 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 06:57:21.422380 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.421963    6224 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 06:57:21.422380 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.421966    6224 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 06:57:21.422380 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.421968    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 06:57:21.422380 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.421971    6224 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 06:57:21.422380 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.421974    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 06:57:21.422380 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.421976    6224 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 06:57:21.422380 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.421979    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 06:57:21.422380 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.421982    6224 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 06:57:21.422380 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.421984    6224 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 06:57:21.422380 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.421987    6224 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 06:57:21.422883 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.421989    6224 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 06:57:21.422883 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.421992    6224 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 06:57:21.422883 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.421994    6224 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 06:57:21.422883 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.421997    6224 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 06:57:21.422883 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422000    6224 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 06:57:21.422883 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422003    6224 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 06:57:21.422883 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422006    6224 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 06:57:21.422883 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422009    6224 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 06:57:21.422883 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422011    6224 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 06:57:21.422883 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422014    6224 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 06:57:21.422883 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422017    6224 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 06:57:21.422883 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422020    6224 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 06:57:21.422883 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422023    6224 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 06:57:21.422883 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422025    6224 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 06:57:21.422883 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422028    6224 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 06:57:21.422883 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422031    6224 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 06:57:21.422883 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422033    6224 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 06:57:21.422883 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422036    6224 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 06:57:21.422883 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422038    6224 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 06:57:21.422883 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422041    6224 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 06:57:21.423382 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422044    6224 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 06:57:21.423382 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422047    6224 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 06:57:21.423382 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422049    6224 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 06:57:21.423382 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422052    6224 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 06:57:21.423382 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422055    6224 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 06:57:21.423382 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422058    6224 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 06:57:21.423382 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422060    6224 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 06:57:21.423382 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422063    6224 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 06:57:21.423382 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422065    6224 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 06:57:21.423382 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422068    6224 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 06:57:21.423382 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422070    6224 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 06:57:21.423382 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422073    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 06:57:21.423382 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422076    6224 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 06:57:21.423382 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422079    6224 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 06:57:21.423382 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422081    6224 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 06:57:21.423382 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422084    6224 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 06:57:21.423382 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422087    6224 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 06:57:21.423382 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422090    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 06:57:21.423382 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422092    6224 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 06:57:21.423382 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422095    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 06:57:21.423884 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422098    6224 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 06:57:21.423884 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422101    6224 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 06:57:21.423884 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422104    6224 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 06:57:21.423884 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422107    6224 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 06:57:21.423884 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422110    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 06:57:21.423884 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422112    6224 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 06:57:21.423884 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422115    6224 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 06:57:21.423884 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422117    6224 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 06:57:21.423884 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422120    6224 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 06:57:21.423884 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422123    6224 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 06:57:21.423884 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422125    6224 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 06:57:21.423884 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422128    6224 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 06:57:21.423884 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422130    6224 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 06:57:21.423884 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422133    6224 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 06:57:21.423884 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422136    6224 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 06:57:21.423884 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422138    6224 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 06:57:21.423884 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422141    6224 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 06:57:21.423884 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422143    6224 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 06:57:21.423884 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422146    6224 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 06:57:21.423884 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422149    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 06:57:21.424411 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422151    6224 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 06:57:21.424411 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422154    6224 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 06:57:21.424411 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422157    6224 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 06:57:21.424411 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422160    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 06:57:21.424411 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422163    6224 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 06:57:21.424411 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422166    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 06:57:21.424411 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422924    6224 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 06:57:21.424411 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422933    6224 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 06:57:21.424411 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422936    6224 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 06:57:21.424411 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422940    6224 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 06:57:21.424411 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422943    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 06:57:21.424411 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422946    6224 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 06:57:21.424411 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422950    6224 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 06:57:21.424411 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422952    6224 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 06:57:21.424411 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422956    6224 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 06:57:21.424411 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422959    6224 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 06:57:21.424411 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422962    6224 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 06:57:21.424411 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422965    6224 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 06:57:21.424411 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422967    6224 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 06:57:21.424411 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422970    6224 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 06:57:21.424914 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422973    6224 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 06:57:21.424914 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422976    6224 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 06:57:21.424914 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422979    6224 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 06:57:21.424914 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422981    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 06:57:21.424914 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422984    6224 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 06:57:21.424914 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422987    6224 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 06:57:21.424914 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422989    6224 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 06:57:21.424914 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422992    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 06:57:21.424914 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422995    6224 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 06:57:21.424914 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.422998    6224 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 06:57:21.424914 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423000    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 06:57:21.424914 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423003    6224 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 06:57:21.424914 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423006    6224 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 06:57:21.424914 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423008    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 06:57:21.424914 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423011    6224 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 06:57:21.424914 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423014    6224 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 06:57:21.424914 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423017    6224 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 06:57:21.424914 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423020    6224 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 06:57:21.424914 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423022    6224 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 06:57:21.424914 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423025    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 06:57:21.425411 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423027    6224 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 06:57:21.425411 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423030    6224 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 06:57:21.425411 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423033    6224 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 06:57:21.425411 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423036    6224 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 06:57:21.425411 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423039    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 06:57:21.425411 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423041    6224 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 06:57:21.425411 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423044    6224 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 06:57:21.425411 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423047    6224 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 06:57:21.425411 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423049    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 06:57:21.425411 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423052    6224 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 06:57:21.425411 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423054    6224 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 06:57:21.425411 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423057    6224 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 06:57:21.425411 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423059    6224 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 06:57:21.425411 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423062    6224 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 06:57:21.425411 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423064    6224 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 06:57:21.425411 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423069    6224 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 06:57:21.425411 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423080    6224 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 06:57:21.425411 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423084    6224 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 06:57:21.425411 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423088    6224 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 06:57:21.425411 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423091    6224 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 06:57:21.425944 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423093    6224 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 06:57:21.425944 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423096    6224 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 06:57:21.425944 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423098    6224 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 06:57:21.425944 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423101    6224 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 06:57:21.425944 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423104    6224 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 06:57:21.425944 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423107    6224 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 06:57:21.425944 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423109    6224 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 06:57:21.425944 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423112    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 06:57:21.425944 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423116    6224 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 06:57:21.425944 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423120    6224 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 06:57:21.425944 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423123    6224 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 06:57:21.425944 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423126    6224 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 06:57:21.425944 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423129    6224 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 06:57:21.425944 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423132    6224 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 06:57:21.425944 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423136    6224 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 06:57:21.425944 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423142    6224 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 06:57:21.425944 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423145    6224 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 06:57:21.425944 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423148    6224 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 06:57:21.425944 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423151    6224 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 06:57:21.426394 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423154    6224 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 06:57:21.426394 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423157    6224 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 06:57:21.426394 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423159    6224 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 06:57:21.426394 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423162    6224 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 06:57:21.426394 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423165    6224 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 06:57:21.426394 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423167    6224 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 06:57:21.426394 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423170    6224 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 06:57:21.426394 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423172    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 06:57:21.426394 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423175    6224 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 06:57:21.426394 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423178    6224 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 06:57:21.426394 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423180    6224 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 06:57:21.426394 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423183    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 06:57:21.426394 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423185    6224 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 06:57:21.426394 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423188    6224 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 06:57:21.426394 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423190    6224 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 06:57:21.426394 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423193    6224 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 06:57:21.426394 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423195    6224 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 06:57:21.426394 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423198    6224 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 06:57:21.426394 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423202    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 06:57:21.426873 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423204    6224 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 06:57:21.426873 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423207    6224 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 06:57:21.426873 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423209    6224 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 06:57:21.426873 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423212    6224 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 06:57:21.426873 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423215    6224 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 06:57:21.426873 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423217    6224 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 06:57:21.426873 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423220    6224 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 06:57:21.426873 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423222    6224 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 06:57:21.426873 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423225    6224 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 06:57:21.426873 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423227    6224 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 06:57:21.426873 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423230    6224 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 06:57:21.426873 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423233    6224 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 06:57:21.426873 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423235    6224 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 06:57:21.426873 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423238    6224 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 06:57:21.426873 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423718    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 06:57:21.426873 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423723    6224 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 06:57:21.426873 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423727    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 06:57:21.426873 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423730    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 06:57:21.426873 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423733    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 06:57:21.426873 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423736    6224 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 06:57:21.427454 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423739    6224 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 06:57:21.427454 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423743    6224 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 06:57:21.427454 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423747    6224 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 06:57:21.427454 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423751    6224 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 06:57:21.427454 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423754    6224 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 06:57:21.427454 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423757    6224 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 06:57:21.427454 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423760    6224 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 06:57:21.427454 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423763    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 06:57:21.427454 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423769    6224 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 06:57:21.427454 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423772    6224 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 06:57:21.427454 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423775    6224 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 06:57:21.427454 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423777    6224 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 06:57:21.427454 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423780    6224 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 06:57:21.427454 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423783    6224 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 06:57:21.427454 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423785    6224 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 06:57:21.427454 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423788    6224 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 06:57:21.427454 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423791    6224 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 06:57:21.427454 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423793    6224 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 06:57:21.427454 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423796    6224 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 06:57:21.427454 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423798    6224 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 06:57:21.428362 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423801    6224 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 06:57:21.428362 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423819    6224 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 06:57:21.428362 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423823    6224 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 06:57:21.428362 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423826    6224 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 06:57:21.428362 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423829    6224 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 06:57:21.428362 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423831    6224 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 06:57:21.428362 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423834    6224 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 06:57:21.428362 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423837    6224 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 06:57:21.428362 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423839    6224 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 06:57:21.428362 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423842    6224 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 06:57:21.428362 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423844    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 06:57:21.428362 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423847    6224 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 06:57:21.428362 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423850    6224 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 06:57:21.428362 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423852    6224 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 06:57:21.428362 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423855    6224 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 06:57:21.428362 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423857    6224 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 06:57:21.428362 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423860    6224 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 06:57:21.428362 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423862    6224 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 06:57:21.428362 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423865    6224 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 06:57:21.428362 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423867    6224 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 06:57:21.429309 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423872    6224 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 06:57:21.429309 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423874    6224 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 06:57:21.429309 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423877    6224 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 06:57:21.429309 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423880    6224 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 06:57:21.429309 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423882    6224 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 06:57:21.429309 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423885    6224 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 06:57:21.429309 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423888    6224 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 06:57:21.429309 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423891    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 06:57:21.429309 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423894    6224 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 06:57:21.429309 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423898    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 06:57:21.429309 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423901    6224 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 06:57:21.429309 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423904    6224 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 06:57:21.429309 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423906    6224 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 06:57:21.429309 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423909    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 06:57:21.429309 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423911    6224 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 06:57:21.429309 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423915    6224 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 06:57:21.429309 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423918    6224 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 06:57:21.429309 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423920    6224 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 06:57:21.429309 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423923    6224 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 06:57:21.429309 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423926    6224 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 06:57:21.430196 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423928    6224 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 06:57:21.430196 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423930    6224 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 06:57:21.430196 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423933    6224 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 06:57:21.430196 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423936    6224 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 06:57:21.430196 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423938    6224 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 06:57:21.430196 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423941    6224 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 06:57:21.430196 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423943    6224 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 06:57:21.430196 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423946    6224 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 06:57:21.430196 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423949    6224 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 06:57:21.430196 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423951    6224 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 06:57:21.430196 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423954    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 06:57:21.430196 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423958    6224 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 06:57:21.430196 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423964    6224 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 06:57:21.430196 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423968    6224 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 06:57:21.430196 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423971    6224 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 06:57:21.430196 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423973    6224 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 06:57:21.430196 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423976    6224 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 06:57:21.430196 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423978    6224 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 06:57:21.430196 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423981    6224 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 06:57:21.431025 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423985    6224 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 06:57:21.431025 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423988    6224 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 06:57:21.431025 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423991    6224 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 06:57:21.431025 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423994    6224 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 06:57:21.431025 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423996    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 06:57:21.431025 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.423999    6224 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 06:57:21.431025 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424002    6224 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 06:57:21.431025 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424004    6224 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 06:57:21.431025 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424007    6224 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 06:57:21.431025 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424010    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 06:57:21.431025 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424012    6224 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 06:57:21.431025 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424015    6224 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 06:57:21.431025 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424017    6224 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 06:57:21.431025 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424020    6224 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 06:57:21.431025 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424022    6224 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 06:57:21.431025 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424025    6224 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 06:57:21.431025 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424027    6224 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 06:57:21.431025 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424030    6224 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 06:57:21.431025 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424032    6224 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 06:57:21.431977 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424035    6224 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 06:57:21.431977 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424037    6224 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 06:57:21.431977 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424129    6224 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 06:57:21.431977 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424136    6224 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 06:57:21.431977 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424142    6224 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 06:57:21.431977 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424146    6224 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 06:57:21.431977 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424152    6224 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 06:57:21.431977 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424156    6224 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 06:57:21.431977 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424160    6224 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 06:57:21.431977 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424164    6224 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 06:57:21.431977 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424168    6224 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 06:57:21.431977 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424171    6224 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 06:57:21.431977 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424174    6224 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 06:57:21.431977 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424177    6224 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 06:57:21.431977 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424180    6224 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 06:57:21.431977 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424185    6224 flags.go:64] FLAG: --cgroup-root=""
Oct 30 06:57:21.431977 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424188    6224 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 06:57:21.431977 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424191    6224 flags.go:64] FLAG: --client-ca-file=""
Oct 30 06:57:21.431977 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424194    6224 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 06:57:21.431977 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424197    6224 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 06:57:21.431977 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424200    6224 flags.go:64] FLAG: --cluster-domain=""
Oct 30 06:57:21.431977 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424203    6224 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 06:57:21.431977 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424206    6224 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 06:57:21.431977 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424209    6224 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 06:57:21.433026 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424213    6224 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 06:57:21.433026 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424217    6224 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 06:57:21.433026 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424220    6224 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 06:57:21.433026 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424223    6224 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 06:57:21.433026 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424227    6224 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 06:57:21.433026 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424230    6224 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 06:57:21.433026 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424233    6224 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 06:57:21.433026 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424236    6224 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 06:57:21.433026 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424239    6224 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 06:57:21.433026 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424242    6224 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 06:57:21.433026 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424247    6224 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 06:57:21.433026 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424250    6224 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 06:57:21.433026 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424253    6224 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 06:57:21.433026 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424256    6224 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 06:57:21.433026 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424260    6224 flags.go:64] FLAG: --enable-server="true"
Oct 30 06:57:21.433026 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424263    6224 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 06:57:21.433026 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424267    6224 flags.go:64] FLAG: --event-burst="100"
Oct 30 06:57:21.433026 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424270    6224 flags.go:64] FLAG: --event-qps="50"
Oct 30 06:57:21.433026 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424273    6224 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 06:57:21.433026 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424276    6224 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 06:57:21.433026 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424279    6224 flags.go:64] FLAG: --eviction-hard=""
Oct 30 06:57:21.433026 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424283    6224 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 06:57:21.433026 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424286    6224 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 06:57:21.433026 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424290    6224 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 06:57:21.433026 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424293    6224 flags.go:64] FLAG: --eviction-soft=""
Oct 30 06:57:21.434155 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424296    6224 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 06:57:21.434155 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424299    6224 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 06:57:21.434155 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424302    6224 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 06:57:21.434155 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424305    6224 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 06:57:21.434155 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424308    6224 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 06:57:21.434155 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424311    6224 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 06:57:21.434155 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424314    6224 flags.go:64] FLAG: --feature-gates=""
Oct 30 06:57:21.434155 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424324    6224 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 06:57:21.434155 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424327    6224 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 06:57:21.434155 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424330    6224 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 06:57:21.434155 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424333    6224 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 06:57:21.434155 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424336    6224 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 06:57:21.434155 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424339    6224 flags.go:64] FLAG: --help="false"
Oct 30 06:57:21.434155 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424342    6224 flags.go:64] FLAG: --hostname-override="ip-10-0-128-125.ec2.internal"
Oct 30 06:57:21.434155 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424346    6224 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 06:57:21.434155 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424349    6224 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 06:57:21.434155 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424351    6224 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 06:57:21.434155 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424355    6224 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 06:57:21.434155 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424358    6224 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 06:57:21.434155 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424361    6224 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 06:57:21.434155 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424363    6224 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 06:57:21.434155 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424368    6224 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 06:57:21.434155 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424371    6224 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 06:57:21.435187 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424374    6224 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 06:57:21.435187 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424377    6224 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 06:57:21.435187 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424380    6224 flags.go:64] FLAG: --kube-reserved=""
Oct 30 06:57:21.435187 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424383    6224 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 06:57:21.435187 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424386    6224 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 06:57:21.435187 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424389    6224 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 06:57:21.435187 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424392    6224 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 06:57:21.435187 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424396    6224 flags.go:64] FLAG: --lock-file=""
Oct 30 06:57:21.435187 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424399    6224 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 06:57:21.435187 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424402    6224 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 06:57:21.435187 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424405    6224 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 06:57:21.435187 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424410    6224 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 06:57:21.435187 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424413    6224 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 06:57:21.435187 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424417    6224 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 06:57:21.435187 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424419    6224 flags.go:64] FLAG: --logging-format="text"
Oct 30 06:57:21.435187 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424422    6224 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 06:57:21.435187 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424426    6224 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 06:57:21.435187 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424429    6224 flags.go:64] FLAG: --manifest-url=""
Oct 30 06:57:21.435187 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424432    6224 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 06:57:21.435187 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424437    6224 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 06:57:21.435187 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424440    6224 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 06:57:21.435187 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424443    6224 flags.go:64] FLAG: --max-pods="110"
Oct 30 06:57:21.435187 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424446    6224 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 06:57:21.435187 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424449    6224 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 06:57:21.435187 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424452    6224 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 06:57:21.436251 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424455    6224 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 06:57:21.436251 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424458    6224 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 06:57:21.436251 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424461    6224 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 06:57:21.436251 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424464    6224 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 06:57:21.436251 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424471    6224 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 06:57:21.436251 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424476    6224 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 06:57:21.436251 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424479    6224 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 06:57:21.436251 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424482    6224 flags.go:64] FLAG: --pod-cidr=""
Oct 30 06:57:21.436251 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424485    6224 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 06:57:21.436251 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424488    6224 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 06:57:21.436251 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424491    6224 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 06:57:21.436251 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424494    6224 flags.go:64] FLAG: --port="10250"
Oct 30 06:57:21.436251 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424496    6224 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 06:57:21.436251 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424499    6224 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d46400f776fab97e"
Oct 30 06:57:21.436251 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424504    6224 flags.go:64] FLAG: --qos-reserved=""
Oct 30 06:57:21.436251 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424507    6224 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 06:57:21.436251 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424510    6224 flags.go:64] FLAG: --register-node="true"
Oct 30 06:57:21.436251 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424513    6224 flags.go:64] FLAG: --register-with-taints=""
Oct 30 06:57:21.436251 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424517    6224 flags.go:64] FLAG: --registry-burst="10"
Oct 30 06:57:21.436251 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424520    6224 flags.go:64] FLAG: --registry-qps="5"
Oct 30 06:57:21.436251 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424523    6224 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 06:57:21.436251 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424525    6224 flags.go:64] FLAG: --reserved-memory=""
Oct 30 06:57:21.436251 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424530    6224 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 06:57:21.436251 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424533    6224 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 06:57:21.436251 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424536    6224 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 06:57:21.436251 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424540    6224 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 06:57:21.437262 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424543    6224 flags.go:64] FLAG: --runonce="false"
Oct 30 06:57:21.437262 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424546    6224 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 06:57:21.437262 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424549    6224 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 06:57:21.437262 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424552    6224 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 06:57:21.437262 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424555    6224 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 06:57:21.437262 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424558    6224 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 06:57:21.437262 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424561    6224 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 06:57:21.437262 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424564    6224 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 06:57:21.437262 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424567    6224 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 06:57:21.437262 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424570    6224 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 06:57:21.437262 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424572    6224 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 06:57:21.437262 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424578    6224 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 06:57:21.437262 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424581    6224 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 06:57:21.437262 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424584    6224 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 06:57:21.437262 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424587    6224 flags.go:64] FLAG: --system-cgroups=""
Oct 30 06:57:21.437262 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424590    6224 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 06:57:21.437262 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424595    6224 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 06:57:21.437262 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424598    6224 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 06:57:21.437262 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424600    6224 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 06:57:21.437262 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424604    6224 flags.go:64] FLAG: --tls-min-version=""
Oct 30 06:57:21.437262 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424608    6224 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 06:57:21.437262 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424611    6224 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 06:57:21.437262 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424614    6224 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 06:57:21.437262 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424617    6224 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 06:57:21.437262 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424620    6224 flags.go:64] FLAG: --v="2"
Oct 30 06:57:21.438110 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424624    6224 flags.go:64] FLAG: --version="false"
Oct 30 06:57:21.438110 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424628    6224 flags.go:64] FLAG: --vmodule=""
Oct 30 06:57:21.438110 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424632    6224 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 06:57:21.438110 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.424636    6224 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 06:57:21.438110 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424760    6224 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 06:57:21.438110 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424764    6224 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 06:57:21.438110 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424767    6224 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 06:57:21.438110 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424770    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 06:57:21.438110 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424773    6224 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 06:57:21.438110 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424776    6224 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 06:57:21.438110 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424778    6224 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 06:57:21.438110 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424781    6224 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 06:57:21.438110 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424784    6224 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 06:57:21.438110 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424786    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 06:57:21.438110 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424789    6224 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 06:57:21.438110 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424791    6224 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 06:57:21.438110 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424794    6224 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 06:57:21.438110 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424798    6224 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 06:57:21.438110 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424817    6224 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 06:57:21.438110 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424820    6224 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 06:57:21.438595 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424823    6224 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 06:57:21.438595 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424825    6224 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 06:57:21.438595 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424828    6224 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 06:57:21.438595 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424830    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 06:57:21.438595 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424833    6224 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 06:57:21.438595 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424836    6224 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 06:57:21.438595 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424838    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 06:57:21.438595 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424843    6224 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 06:57:21.438595 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424845    6224 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 06:57:21.438595 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424848    6224 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 06:57:21.438595 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424850    6224 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 06:57:21.438595 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424853    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 06:57:21.438595 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424855    6224 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 06:57:21.438595 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424858    6224 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 06:57:21.438595 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424861    6224 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 06:57:21.438595 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424863    6224 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 06:57:21.438595 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424866    6224 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 06:57:21.438595 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424869    6224 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 06:57:21.438595 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424871    6224 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 06:57:21.439143 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424874    6224 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 06:57:21.439143 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424877    6224 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 06:57:21.439143 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424879    6224 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 06:57:21.439143 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424882    6224 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 06:57:21.439143 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424885    6224 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 06:57:21.439143 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424887    6224 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 06:57:21.439143 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424890    6224 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 06:57:21.439143 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424892    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 06:57:21.439143 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424895    6224 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 06:57:21.439143 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424897    6224 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 06:57:21.439143 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424900    6224 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 06:57:21.439143 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424903    6224 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 06:57:21.439143 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424906    6224 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 06:57:21.439143 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424909    6224 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 06:57:21.439143 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424912    6224 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 06:57:21.439143 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424914    6224 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 06:57:21.439143 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424917    6224 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 06:57:21.439143 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424920    6224 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 06:57:21.439143 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424924    6224 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 06:57:21.439843 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424928    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 06:57:21.439843 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424932    6224 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 06:57:21.439843 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424935    6224 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 06:57:21.439843 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424938    6224 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 06:57:21.439843 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424940    6224 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 06:57:21.439843 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424943    6224 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 06:57:21.439843 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424945    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 06:57:21.439843 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424948    6224 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 06:57:21.439843 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424950    6224 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 06:57:21.439843 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424953    6224 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 06:57:21.439843 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424955    6224 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 06:57:21.439843 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424958    6224 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 06:57:21.439843 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424960    6224 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 06:57:21.439843 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424963    6224 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 06:57:21.439843 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424966    6224 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 06:57:21.439843 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424969    6224 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 06:57:21.439843 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424971    6224 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 06:57:21.439843 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424974    6224 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 06:57:21.439843 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424976    6224 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 06:57:21.439843 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424979    6224 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 06:57:21.440402 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424981    6224 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 06:57:21.440402 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424984    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 06:57:21.440402 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424987    6224 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 06:57:21.440402 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424989    6224 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 06:57:21.440402 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424993    6224 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 06:57:21.440402 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424995    6224 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 06:57:21.440402 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.424998    6224 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 06:57:21.440402 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425000    6224 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 06:57:21.440402 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425003    6224 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 06:57:21.440402 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425005    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 06:57:21.440402 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425008    6224 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 06:57:21.440402 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425010    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 06:57:21.440402 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425013    6224 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 06:57:21.440402 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425016    6224 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 06:57:21.440402 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425019    6224 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 06:57:21.440402 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425021    6224 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 06:57:21.440402 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425024    6224 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 06:57:21.440402 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425027    6224 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 06:57:21.440402 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425029    6224 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 06:57:21.440402 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425032    6224 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 06:57:21.440927 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425034    6224 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 06:57:21.440927 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425037    6224 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 06:57:21.440927 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425039    6224 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 06:57:21.440927 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425042    6224 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 06:57:21.440927 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425044    6224 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 06:57:21.440927 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425047    6224 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 06:57:21.440927 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425050    6224 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 06:57:21.440927 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425052    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 06:57:21.440927 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425055    6224 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 06:57:21.440927 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425058    6224 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 06:57:21.440927 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425060    6224 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 06:57:21.440927 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425063    6224 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 06:57:21.440927 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.425088    6224 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 06:57:21.440927 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425120    6224 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 06:57:21.440927 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425125    6224 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 06:57:21.441334 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425128    6224 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 06:57:21.441334 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425131    6224 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 06:57:21.441334 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425133    6224 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 06:57:21.441334 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425136    6224 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 06:57:21.441334 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425146    6224 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 06:57:21.441334 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425148    6224 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 06:57:21.441334 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425151    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 06:57:21.441334 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425154    6224 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 06:57:21.441334 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425156    6224 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 06:57:21.441334 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425159    6224 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 06:57:21.441334 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425163    6224 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 06:57:21.441334 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425166    6224 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 06:57:21.441334 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425169    6224 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 06:57:21.441334 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425172    6224 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 06:57:21.441334 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425174    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 06:57:21.441334 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425177    6224 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 06:57:21.441334 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425179    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 06:57:21.441334 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425182    6224 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 06:57:21.441334 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425185    6224 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 06:57:21.441334 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425188    6224 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 06:57:21.441937 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425190    6224 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 06:57:21.441937 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425193    6224 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 06:57:21.441937 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425196    6224 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 06:57:21.441937 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425198    6224 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 06:57:21.441937 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425201    6224 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 06:57:21.441937 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425204    6224 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 06:57:21.441937 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425207    6224 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 06:57:21.441937 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425210    6224 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 06:57:21.441937 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425213    6224 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 06:57:21.441937 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425215    6224 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 06:57:21.441937 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425218    6224 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 06:57:21.441937 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425220    6224 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 06:57:21.441937 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425223    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 06:57:21.441937 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425225    6224 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 06:57:21.441937 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425228    6224 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 06:57:21.441937 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425231    6224 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 06:57:21.441937 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425233    6224 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 06:57:21.441937 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425236    6224 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 06:57:21.441937 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425239    6224 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 06:57:21.442613 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425241    6224 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 06:57:21.442613 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425243    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 06:57:21.442613 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425246    6224 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 06:57:21.442613 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425250    6224 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 06:57:21.442613 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425253    6224 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 06:57:21.442613 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425255    6224 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 06:57:21.442613 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425258    6224 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 06:57:21.442613 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425260    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 06:57:21.442613 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425263    6224 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 06:57:21.442613 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425265    6224 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 06:57:21.442613 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425268    6224 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 06:57:21.442613 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425273    6224 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 06:57:21.442613 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425276    6224 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 06:57:21.442613 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425279    6224 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 06:57:21.442613 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425282    6224 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 06:57:21.442613 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425284    6224 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 06:57:21.442613 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425287    6224 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 06:57:21.442613 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425289    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 06:57:21.442613 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425292    6224 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 06:57:21.443110 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425294    6224 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 06:57:21.443110 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425297    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 06:57:21.443110 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425300    6224 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 06:57:21.443110 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425303    6224 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 06:57:21.443110 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425305    6224 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 06:57:21.443110 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425308    6224 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 06:57:21.443110 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425310    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 06:57:21.443110 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425313    6224 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 06:57:21.443110 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425315    6224 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 06:57:21.443110 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425318    6224 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 06:57:21.443110 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425320    6224 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 06:57:21.443110 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425322    6224 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 06:57:21.443110 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425325    6224 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 06:57:21.443110 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425327    6224 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 06:57:21.443110 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425330    6224 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 06:57:21.443110 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425332    6224 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 06:57:21.443110 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425336    6224 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 06:57:21.443110 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425339    6224 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 06:57:21.443110 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425341    6224 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 06:57:21.443575 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425344    6224 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 06:57:21.443575 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425346    6224 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 06:57:21.443575 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425349    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 06:57:21.443575 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425351    6224 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 06:57:21.443575 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425354    6224 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 06:57:21.443575 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425358    6224 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 06:57:21.443575 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425360    6224 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 06:57:21.443575 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425363    6224 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 06:57:21.443575 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425365    6224 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 06:57:21.443575 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425368    6224 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 06:57:21.443575 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425371    6224 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 06:57:21.443575 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425373    6224 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 06:57:21.443575 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425376    6224 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 06:57:21.443575 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425379    6224 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 06:57:21.443575 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425383    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 06:57:21.443575 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425387    6224 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 06:57:21.443575 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425390    6224 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 06:57:21.443575 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425392    6224 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 06:57:21.443575 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425395    6224 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 06:57:21.444047 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425398    6224 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 06:57:21.444047 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425401    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 06:57:21.444047 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425404    6224 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 06:57:21.444047 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425406    6224 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 06:57:21.444047 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425409    6224 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 06:57:21.444047 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425411    6224 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 06:57:21.444047 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425414    6224 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 06:57:21.444047 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.425417    6224 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 06:57:21.444047 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.430485    6224 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 06:57:21.444047 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.430505    6224 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 06:57:21.444047 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.430530    6224 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 06:57:21.444047 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.430537    6224 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 06:57:21.444047 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430627    6224 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 06:57:21.444047 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430635    6224 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 06:57:21.444047 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430641    6224 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 06:57:21.444047 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430646    6224 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 06:57:21.444047 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430650    6224 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 06:57:21.444047 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430655    6224 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 06:57:21.444047 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430660    6224 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 06:57:21.444047 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430664    6224 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 06:57:21.444547 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430668    6224 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 06:57:21.444547 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430673    6224 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 06:57:21.444547 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430677    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 06:57:21.444547 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430682    6224 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 06:57:21.444547 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430686    6224 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 06:57:21.444547 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430691    6224 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 06:57:21.444547 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430695    6224 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 06:57:21.444547 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430700    6224 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 06:57:21.444547 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430704    6224 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 06:57:21.444547 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430709    6224 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 06:57:21.444547 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430713    6224 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 06:57:21.444547 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430718    6224 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 06:57:21.444547 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430722    6224 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 06:57:21.444547 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430726    6224 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 06:57:21.444547 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430731    6224 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 06:57:21.444547 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430735    6224 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 06:57:21.444547 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430740    6224 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 06:57:21.444547 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430744    6224 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 06:57:21.444547 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430748    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 06:57:21.444547 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430753    6224 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 06:57:21.445158 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430760    6224 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 06:57:21.445158 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430776    6224 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 06:57:21.445158 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430782    6224 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 06:57:21.445158 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430786    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 06:57:21.445158 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430790    6224 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 06:57:21.445158 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430795    6224 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 06:57:21.445158 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430799    6224 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 06:57:21.445158 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430819    6224 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 06:57:21.445158 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430824    6224 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 06:57:21.445158 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430828    6224 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 06:57:21.445158 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430833    6224 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 06:57:21.445158 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430837    6224 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 06:57:21.445158 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430842    6224 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 06:57:21.445158 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430846    6224 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 06:57:21.445158 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430850    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 06:57:21.445158 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430854    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 06:57:21.445158 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430859    6224 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 06:57:21.445158 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430864    6224 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 06:57:21.445158 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430868    6224 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 06:57:21.445816 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430872    6224 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 06:57:21.445816 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430876    6224 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 06:57:21.445816 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430881    6224 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 06:57:21.445816 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430885    6224 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 06:57:21.445816 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430889    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 06:57:21.445816 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430893    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 06:57:21.445816 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430897    6224 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 06:57:21.445816 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430902    6224 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 06:57:21.445816 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430905    6224 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 06:57:21.445816 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430910    6224 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 06:57:21.445816 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430914    6224 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 06:57:21.445816 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430919    6224 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 06:57:21.445816 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430923    6224 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 06:57:21.445816 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430927    6224 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 06:57:21.445816 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430932    6224 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 06:57:21.445816 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430947    6224 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 06:57:21.445816 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430952    6224 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 06:57:21.445816 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430956    6224 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 06:57:21.445816 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430960    6224 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 06:57:21.445816 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430965    6224 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 06:57:21.446453 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430969    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 06:57:21.446453 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430973    6224 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 06:57:21.446453 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430978    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 06:57:21.446453 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430985    6224 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 06:57:21.446453 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430990    6224 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 06:57:21.446453 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.430995    6224 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 06:57:21.446453 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431000    6224 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 06:57:21.446453 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431004    6224 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 06:57:21.446453 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431008    6224 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 06:57:21.446453 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431012    6224 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 06:57:21.446453 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431017    6224 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 06:57:21.446453 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431021    6224 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 06:57:21.446453 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431025    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 06:57:21.446453 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431029    6224 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 06:57:21.446453 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431033    6224 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 06:57:21.446453 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431037    6224 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 06:57:21.446453 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431041    6224 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 06:57:21.446453 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431045    6224 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 06:57:21.446921 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431050    6224 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 06:57:21.446921 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431054    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 06:57:21.446921 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431059    6224 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 06:57:21.446921 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431063    6224 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 06:57:21.446921 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431067    6224 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 06:57:21.446921 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431072    6224 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 06:57:21.446921 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431076    6224 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 06:57:21.446921 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431080    6224 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 06:57:21.446921 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431084    6224 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 06:57:21.446921 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431089    6224 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 06:57:21.446921 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431101    6224 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 06:57:21.446921 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431106    6224 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 06:57:21.446921 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431110    6224 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 06:57:21.446921 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431115    6224 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 06:57:21.446921 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431119    6224 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 06:57:21.446921 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431123    6224 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 06:57:21.446921 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431127    6224 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 06:57:21.446921 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431132    6224 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 06:57:21.446921 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431136    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 06:57:21.446921 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431140    6224 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 06:57:21.447422 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431145    6224 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 06:57:21.447422 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.431186    6224 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 06:57:21.447422 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431474    6224 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 06:57:21.447422 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431482    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 06:57:21.447422 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431488    6224 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 06:57:21.447422 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431492    6224 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 06:57:21.447422 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431496    6224 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 06:57:21.447422 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431500    6224 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 06:57:21.447422 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431505    6224 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 06:57:21.447422 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431509    6224 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 06:57:21.447422 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431513    6224 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 06:57:21.447422 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431518    6224 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 06:57:21.447422 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431522    6224 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 06:57:21.447422 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431526    6224 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 06:57:21.447422 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431531    6224 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 06:57:21.447422 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431535    6224 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 06:57:21.447824 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431539    6224 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 06:57:21.447824 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431544    6224 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 06:57:21.447824 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431549    6224 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 06:57:21.447824 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431553    6224 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 06:57:21.447824 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431557    6224 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 06:57:21.447824 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431561    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 06:57:21.447824 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431565    6224 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 06:57:21.447824 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431578    6224 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 06:57:21.447824 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431583    6224 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 06:57:21.447824 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431587    6224 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 06:57:21.447824 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431591    6224 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 06:57:21.447824 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431595    6224 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 06:57:21.447824 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431600    6224 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 06:57:21.447824 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431604    6224 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 06:57:21.447824 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431608    6224 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 06:57:21.447824 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431612    6224 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 06:57:21.447824 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431617    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 06:57:21.447824 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431623    6224 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 06:57:21.447824 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431629    6224 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 06:57:21.448292 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431634    6224 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 06:57:21.448292 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431639    6224 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 06:57:21.448292 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431644    6224 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 06:57:21.448292 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431648    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 06:57:21.448292 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431652    6224 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 06:57:21.448292 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431656    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 06:57:21.448292 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431661    6224 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 06:57:21.448292 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431665    6224 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 06:57:21.448292 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431670    6224 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 06:57:21.448292 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431674    6224 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 06:57:21.448292 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431678    6224 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 06:57:21.448292 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431683    6224 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 06:57:21.448292 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431687    6224 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 06:57:21.448292 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431691    6224 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 06:57:21.448292 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431696    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 06:57:21.448292 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431701    6224 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 06:57:21.448292 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431705    6224 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 06:57:21.448292 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431717    6224 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 06:57:21.448292 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431721    6224 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 06:57:21.448292 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431726    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 06:57:21.448767 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431730    6224 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 06:57:21.448767 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431741    6224 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 06:57:21.448767 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431746    6224 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 06:57:21.448767 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431750    6224 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 06:57:21.448767 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431757    6224 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 06:57:21.448767 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431761    6224 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 06:57:21.448767 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431766    6224 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 06:57:21.448767 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431770    6224 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 06:57:21.448767 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431774    6224 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 06:57:21.448767 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431778    6224 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 06:57:21.448767 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431783    6224 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 06:57:21.448767 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431787    6224 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 06:57:21.448767 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431791    6224 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 06:57:21.448767 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431796    6224 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 06:57:21.448767 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431800    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 06:57:21.448767 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431822    6224 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 06:57:21.448767 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431827    6224 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 06:57:21.448767 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431830    6224 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 06:57:21.448767 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431834    6224 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 06:57:21.449244 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431838    6224 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 06:57:21.449244 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431842    6224 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 06:57:21.449244 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431846    6224 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 06:57:21.449244 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431850    6224 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 06:57:21.449244 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431853    6224 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 06:57:21.449244 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431857    6224 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 06:57:21.449244 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431861    6224 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 06:57:21.449244 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431865    6224 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 06:57:21.449244 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431869    6224 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 06:57:21.449244 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431873    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 06:57:21.449244 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431877    6224 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 06:57:21.449244 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431881    6224 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 06:57:21.449244 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431885    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 06:57:21.449244 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431890    6224 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 06:57:21.449244 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431893    6224 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 06:57:21.449244 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431905    6224 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 06:57:21.449244 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431910    6224 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 06:57:21.449244 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431916    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 06:57:21.449244 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431922    6224 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 06:57:21.449697 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431928    6224 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 06:57:21.449697 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431933    6224 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 06:57:21.449697 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431938    6224 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 06:57:21.449697 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431942    6224 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 06:57:21.449697 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431946    6224 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 06:57:21.449697 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431951    6224 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 06:57:21.449697 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431956    6224 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 06:57:21.449697 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431960    6224 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 06:57:21.449697 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431964    6224 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 06:57:21.449697 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431969    6224 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 06:57:21.449697 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431973    6224 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 06:57:21.449697 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431976    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 06:57:21.449697 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431981    6224 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 06:57:21.449697 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431985    6224 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 06:57:21.449697 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.431990    6224 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 06:57:21.449697 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.432032    6224 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 06:57:21.450143 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432102    6224 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 06:57:21.450143 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432109    6224 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 06:57:21.450143 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432114    6224 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 06:57:21.450143 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432120    6224 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 06:57:21.450143 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432124    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 06:57:21.450143 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432131    6224 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 06:57:21.450143 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432138    6224 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 06:57:21.450143 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432142    6224 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 06:57:21.450143 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432147    6224 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 06:57:21.450143 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432151    6224 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 06:57:21.450143 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432155    6224 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 06:57:21.450143 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432160    6224 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 06:57:21.450143 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432165    6224 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 06:57:21.450143 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432177    6224 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 06:57:21.450143 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432182    6224 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 06:57:21.450143 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432188    6224 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 06:57:21.450143 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432193    6224 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 06:57:21.450143 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432197    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 06:57:21.450143 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432202    6224 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 06:57:21.450653 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432206    6224 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 06:57:21.450653 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432210    6224 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 06:57:21.450653 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432214    6224 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 06:57:21.450653 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432219    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 06:57:21.450653 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432223    6224 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 06:57:21.450653 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432227    6224 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 06:57:21.450653 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432232    6224 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 06:57:21.450653 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432236    6224 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 06:57:21.450653 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432240    6224 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 06:57:21.450653 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432244    6224 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 06:57:21.450653 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432249    6224 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 06:57:21.450653 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432253    6224 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 06:57:21.450653 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432257    6224 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 06:57:21.450653 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432261    6224 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 06:57:21.450653 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432266    6224 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 06:57:21.450653 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432271    6224 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 06:57:21.450653 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432275    6224 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 06:57:21.450653 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432279    6224 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 06:57:21.450653 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432283    6224 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 06:57:21.450653 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432288    6224 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 06:57:21.451204 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432292    6224 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 06:57:21.451204 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432296    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 06:57:21.451204 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432301    6224 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 06:57:21.451204 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432305    6224 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 06:57:21.451204 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432309    6224 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 06:57:21.451204 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432313    6224 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 06:57:21.451204 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432318    6224 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 06:57:21.451204 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432329    6224 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 06:57:21.451204 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432335    6224 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 06:57:21.451204 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432339    6224 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 06:57:21.451204 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432344    6224 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 06:57:21.451204 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432348    6224 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 06:57:21.451204 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432353    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 06:57:21.451204 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432360    6224 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 06:57:21.451204 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432366    6224 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 06:57:21.451204 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432371    6224 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 06:57:21.451204 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432375    6224 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 06:57:21.451204 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432380    6224 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 06:57:21.451204 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432384    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 06:57:21.451687 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432388    6224 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 06:57:21.451687 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432392    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 06:57:21.451687 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432397    6224 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 06:57:21.451687 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432401    6224 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 06:57:21.451687 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432405    6224 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 06:57:21.451687 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432409    6224 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 06:57:21.451687 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432413    6224 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 06:57:21.451687 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432418    6224 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 06:57:21.451687 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432422    6224 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 06:57:21.451687 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432426    6224 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 06:57:21.451687 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432431    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 06:57:21.451687 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432435    6224 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 06:57:21.451687 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432439    6224 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 06:57:21.451687 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432444    6224 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 06:57:21.451687 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432448    6224 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 06:57:21.451687 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432452    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 06:57:21.451687 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432456    6224 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 06:57:21.451687 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432461    6224 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 06:57:21.451687 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432465    6224 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 06:57:21.451687 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432469    6224 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 06:57:21.452288 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432473    6224 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 06:57:21.452288 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432487    6224 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 06:57:21.452288 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432492    6224 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 06:57:21.452288 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432497    6224 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 06:57:21.452288 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432501    6224 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 06:57:21.452288 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432505    6224 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 06:57:21.452288 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432510    6224 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 06:57:21.452288 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432514    6224 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 06:57:21.452288 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432518    6224 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 06:57:21.452288 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432522    6224 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 06:57:21.452288 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432527    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 06:57:21.452288 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432532    6224 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 06:57:21.452288 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432536    6224 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 06:57:21.452288 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432541    6224 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 06:57:21.452288 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432545    6224 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 06:57:21.452288 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432549    6224 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 06:57:21.452288 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432605    6224 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 06:57:21.452288 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432611    6224 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 06:57:21.452288 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432615    6224 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 06:57:21.452766 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432620    6224 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 06:57:21.452766 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432631    6224 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 06:57:21.452766 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432635    6224 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 06:57:21.452766 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432639    6224 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 06:57:21.452766 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432644    6224 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 06:57:21.452766 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432649    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 06:57:21.452766 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432653    6224 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 06:57:21.452766 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432658    6224 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 06:57:21.452766 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:21.432663    6224 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 06:57:21.452766 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.432749    6224 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 06:57:21.452766 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.434335    6224 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 06:57:21.452766 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.434402    6224 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 06:57:21.452766 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.434709    6224 server.go:1014] "Starting client certificate rotation"
Oct 30 06:57:21.452766 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.434838    6224 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 06:57:21.452766 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.434885    6224 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 06:51:11 +0000 UTC" deadline="2028-06-17 10:27:34.149487654 +0000 UTC"
Oct 30 06:57:21.452766 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.434910    6224 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14307h30m12.714579501s"
Oct 30 06:57:21.453518 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.435523    6224 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 06:57:21.453518 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.438408    6224 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 06:57:21.453518 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.439543    6224 log.go:25] "Validated CRI v1 runtime API"
Oct 30 06:57:21.453518 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.441401    6224 log.go:25] "Validated CRI v1 image API"
Oct 30 06:57:21.453518 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.442179    6224 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 06:57:21.453518 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.444207    6224 fs.go:135] Filesystem UUIDs: map[404a11c7-7d36-440d-a1a4-22b3ef3b102f:/dev/nvme0n1p3 46e964aa-6867-435b-949d-a90b55fe4c93:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Oct 30 06:57:21.453892 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.444228    6224 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/06224bc406875c768834b53973ff4ff559b88f05bd498fbe25fa9a80007bec57/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/06224bc406875c768834b53973ff4ff559b88f05bd498fbe25fa9a80007bec57/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0d57907e4151320d3a4ce60e0d55b8fec5237232120b7685df5b8319b65099ca/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0d57907e4151320d3a4ce60e0d55b8fec5237232120b7685df5b8319b65099ca/userdata/shm major:0 minor:311 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/12c2c077b5e8d30e4257b0ea0ae956cbbab35ee764ee7f301a9dc1e01d073c97/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/12c2c077b5e8d30e4257b0ea0ae956cbbab35ee764ee7f301a9dc1e01d073c97/userdata/shm major:0 minor:409 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1d1240054d453d5b62713793962afa616b2b7c7a1c817ed63febd27be4ceeef9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1d1240054d453d5b62713793962afa616b2b7c7a1c817ed63febd27be4ceeef9/userdata/shm major:0 minor:373 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3420c9754d38d1b6f1bf2af5f6dd65d8bc93f12c95d4a4f1e9bdfdfd8a76bd7f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3420c9754d38d1b6f1bf2af5f6dd65d8bc93f12c95d4a4f1e9bdfdfd8a76bd7f/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/39db0820745d212e03b98cd445e052d84820fb8ad42c98ed8fa2d370026ecd90/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/39db0820745d212e03b98cd445e052d84820fb8ad42c98ed8fa2d370026ecd90/userdata/shm major:0 minor:395 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3a3410f94fd4c007d13abc13d7e3d4f70224c98d53ea499adc6ae46982b9eebf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3a3410f94fd4c007d13abc13d7e3d4f70224c98d53ea499adc6ae46982b9eebf/userdata/shm major:0 minor:87 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3af572320fb94341a428cb0d1ff52bf19e935a13aa08043036427d1aed82f43b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3af572320fb94341a428cb0d1ff52bf19e935a13aa08043036427d1aed82f43b/userdata/shm major:0 minor:274 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4cf0039f2b797f9938c56521778329a145c93299deea89534bff76814a1fb9cb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4cf0039f2b797f9938c56521778329a145c93299deea89534bff76814a1fb9cb/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/562e7614715b08bbae86360ec40dcac18e315ad97b411cf869f6f82cf0d30116/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/562e7614715b08bbae86360ec40dcac18e315ad97b411cf869f6f82cf0d30116/userdata/shm major:0 minor:302 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/61eb4fbe80838432dc327dc06a8580e4a1d35a985986b0c2c113de76ac733a9c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/61eb4fbe80838432dc327dc06a8580e4a1d35a985986b0c2c113de76ac733a9c/userdata/shm major:0 minor:333 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/80a771c73af370f46a40ec02354dc73456551de82c38c22b9f57bbfd09a7c86a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/80a771c73af370f46a40ec02354dc73456551de82c38c22b9f57bbfd09a7c86a/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/86c6ac0388a077eacf9d769b76ab3a06c1c1dea7ae0982f56c78f7da301fd226/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/86c6ac0388a077eacf9d769b76ab3a06c1c1dea7ae0982f56c78f7da301fd226/userdata/shm major:0 minor:276 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/88c5fae3e58af8696d8b0aa5eba674d92d35383ce719079cc013d2034376d11a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/88c5fae3e58af8696d8b0aa5eba674d92d35383ce719079cc013d2034376d11a/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/919f82c3786444358be28752d8866ba97d19b73cf72883656d55e4b3c2b71592/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/919f82c3786444358be28752d8866ba97d19b73cf72883656d55e4b3c2b71592/userdata/shm major:0 minor:249 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/93adb19d38e4ac992b2167c0faf831a897937178bfbddca1361abf56ba4cb941/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/93adb19d38e4ac992b2167c0faf831a897937178bfbddca1361abf56ba4cb941/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/943ee05bb25ae7fe4215fdaec4395add68669b2a4643aa0d476543979b4c28a6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/943ee05bb25ae7fe4215fdaec4395add68669b2a4643aa0d476543979b4c28a6/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b2f0d0628cf42a429a4f71ec6f40317fa67e7594d24d33bbfc74fd27438d50a9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b2f0d0628cf42a429a4f71ec6f40317fa67e7594d24d33bbfc74fd27438d50a9/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ca2afa0103577be424a378670e76be6f79a1d7d5eccc4adf294a66963c7af7ab/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ca2afa0103577be424a378670e76be6f79a1d7d5eccc4adf294a66963c7af7ab/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cab1d1b7478b01d62e7af6636d06b4f3300b2fe9d55994cd23840f04d88290c3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cab1d1b7478b01d62e7af6636d06b4f3300b2fe9d55994cd23840f04d88290c3/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e04753844ffdbcd2c1f467e3e188f856361230266cc9e241bb967283f893901c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e04753844ffdbcd2c1f467e3e188f856361230266cc9e241bb967283f893901c/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fa5bcb3e624bd2241115c27f02a8efaebd5ed1c8d6d0851c075cd1001227e2b9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fa5bcb3e624bd2241115c27f02a8efaebd5ed1c8d6d0851c075cd1001227e2b9/userdata/shm major:0 minor:368 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fa9ac36aa63c36f432ecb122126c429701322d5758b168a1026b9c3b6353db26/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fa9ac36aa63c36f432ecb122126c429701322d5758b168a1026b9c3b6353db26/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12c298a7-e78a-45d7-abe1-7b7ef6f5df03/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/12c298a7-e78a-45d7-abe1-7b7ef6f5df03/volume-subpaths/web-config/config-reloader/2 major:0 minor:331 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12c298a7-e78a-45d7-abe1-7b7ef6f5df03/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/12c298a7-e78a-45d7-abe1-7b7ef6f5df03/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:331 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12c298a7-e78a-45d7-abe1-7b7ef6f5df03/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/12c298a7-e78a-45d7-abe1-7b7ef6f5df03/volume-subpaths/web-config/prometheus/15 major:0 minor:331 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12c298a7-e78a-45d7-abe1-7b7ef6f5df03/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/12c298a7-e78a-45d7-abe1-7b7ef6f5df03/volumes/kubernetes.io~empty-dir/config-out major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12c298a7-e78a-45d7-abe1-7b7ef6f5df03/volumes/kubernetes.io~projected/kube-api-access-n9bkg:{mountpoint:/var/lib/kubelet/pods/12c298a7-e78a-45d7-abe1-7b7ef6f5df03/volumes/kubernetes.io~projected/kube-api-access-n9bkg major:0 minor:332 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12c298a7-e78a-45d7-abe1-7b7ef6f5df03/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/12c298a7-e78a-45d7-abe1-7b7ef6f5df03/volumes/kubernetes.io~projected/tls-assets major:0 minor:325 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12c298a7-e78a-45d7-abe1-7b7ef6f5df03/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/12c298a7-e78a-45d7-abe1-7b7ef6f5df03/volumes/kubernetes.io~secret/config major:0 minor:327 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12c298a7-e78a-45d7-abe1-7b7ef6f5df03/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/12c298a7-e78a-45d7-abe1-7b7ef6f5df03/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:322 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12c298a7-e78a-45d7-abe1-7b7ef6f5df03/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/12c298a7-e78a-45d7-abe1-7b7ef6f5df03/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12c298a7-e78a-45d7-abe1-7b7ef6f5df03/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/12c298a7-e78a-45d7-abe1-7b7ef6f5df03/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:324 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12c298a7-e78a-45d7-abe1-7b7ef6f5df03/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/12c298a7-e78a-45d7-abe1-7b7ef6f5df03/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:326 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12c298a7-e78a-45d7-abe1-7b7ef6f5df03/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/12c298a7-e78a-45d7-abe1-7b7ef6f5df03/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:323 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12c298a7-e78a-45d7-abe1-7b7ef6f5df03/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/12c298a7-e78a-45d7-abe1-7b7ef6f5df03/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:317 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12c298a7-e78a-45d7-abe1-7b7ef6f5df03/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/12c298a7-e78a-45d7-abe1-7b7ef6f5df03/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12c298a7-e78a-45d7-abe1-7b7ef6f5df03/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/12c298a7-e78a-45d7-abe1-7b7ef6f5df03/volumes/kubernetes.io~secret/web-config major:0 minor:331 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2368101c-6719-4650-8393-ef048500a624/volumes/kubernetes.io~projected/kube-api-access-dklkl:{mountpoint:/var/lib/kubelet/pods/2368101c-6719-4650-8393-ef048500a624/volumes/kubernetes.io~projected/kube-api-access-dklkl major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3062f537-9b22-4e15-97aa-9a68703702f8/volumes/kubernetes.io~projected/kube-api-access-wnrqt:{mountpoint:/var/lib/kubelet/pods/3062f537-9b22-4e15-97aa-9a68703702f8/volumes/kubernetes.io~projected/kube-api-access-wnrqt major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3062f537-9b22-4e15-97aa-9a68703702f8/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/3062f537-9b22-4e15-97aa-9a68703702f8/volumes/kubernetes.io~secret/cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31519c22-050e-435c-aec3-7be0b68ff313/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/31519c22-050e-435c-aec3-7be0b68ff313/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5d87089b-8b1b-43ea-be1f-72bfb56bf311/volumes/kubernetes.io~projected/kube-api-access-tlfr5:{mountpoint:/var/lib/kubelet/pods/5d87089b-8b1b-43ea-be1f-72bfb56bf311/volumes/kubernetes.io~projected/kube-api-access-tlfr5 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/675dc1eb-ff05-40c7-bf0b-fb3161d3cc8c/volumes/kubernetes.io~projected/kube-api-access-b74sr:{mountpoint:/var/lib/kubelet/pods/675dc1eb-ff05-40c7-bf0b-fb3161d3cc8c/volumes/kubernetes.io~projected/kube-api-access-b74sr major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6d493f94-3aef-445e-9b60-32a10c184eac/volumes/kubernetes.io~projected/kube-api-access-2cwrb:{mountpoint:/var/lib/kubelet/pods/6d493f94-3aef-445e-9b60-32a10c184eac/volumes/kubernetes.io~projected/kube-api-access-2cwrb major:0 minor:381 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6f8b75af-cc86-49c6-a66c-1798309f9806/volumes/kubernetes.io~projected/kube-api-access-lmzmm:{mountpoint:/var/lib/kubelet/pods/6f8b75af-cc86-49c6-a66c-1798309f9806/volumes/kubernetes.io~projected/kube-api-access-lmzmm major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6f8b75af-cc86-49c6-a66c-1798309f9806/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/6f8b75af-cc86-49c6-a66c-1798309f9806/volumes/kubernetes.io~secret/metrics-certs major:0 minor:365 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/851fb6ff-4a58-431b-834f-db3f87b67aa9/volumes/kubernetes.io~projected/kube-api-access-hvdd8:{mountpoint:/var/lib/kubelet/pods/851fb6ff-4a58-431b-834f-db3f87b67aa9/volumes/kubernetes.io~projected/kube-api-access-hvdd8 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9092e2e8-ad23-426c-b6f1-14e1a2b9221e/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/9092e2e8-ad23-426c-b6f1-14e1a2b9221e/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:361 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9c5ccc21-fce2-4221-9c19-4a35fef32eff/volumes/kubernetes.io~projected/kube-api-access-lbpk2:{mountpoint:/var/lib/kubelet/pods/9c5ccc21-fce2-4221-9c19-4a35fef32eff/volumes/kubernetes.io~projected/kube-api-access-lbpk2 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9c5ccc21-fce2-4221-9c19-4a35fef32eff/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/9c5ccc21-fce2-4221-9c19-4a35fef32eff/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d24d508-ec6f-4165-8394-f17b09134b34/volumes/kubernetes.io~projected/kube-api-access-x67rl:{mountpoint:/var/lib/kubelet/pods/9d24d508-ec6f-4165-8394-f17b09134b34/volumes/kubernetes.io~projected/kube-api-access-x67rl major:0 minor:270 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d24d508-ec6f-4165-8394-f17b09134b34/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/9d24d508-ec6f-4165-8394-f17b09134b34/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:265 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d24d508-ec6f-4165-8394-f17b09134b34/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/9d24d508-ec6f-4165-8394-f17b09134b34/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:269 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6968152-3c44-4205-a078-f857c413ad91/volumes/kubernetes.io~projected/kube-api-access-rkb8j:{mountpoint:/var/lib/kubelet/pods/a6968152-3c44-4205-a078-f857c413ad91/volumes/kubernetes.io~projected/kube-api-access-rkb8j major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b321b279-a102-48b9-bd9b-8ed938eed0f7/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/b321b279-a102-48b9-bd9b-8ed938eed0f7/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b321b279-a102-48b9-bd9b-8ed938eed0f7/volumes/kubernetes.io~projected/kube-api-access-8kds7:{mountpoint:/var/lib/kubelet/pods/b321b279-a102-48b9-bd9b-8ed938eed0f7/volumes/kubernetes.io~projected/kube-api-access-8kds7 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b321b279-a102-48b9-bd9b-8ed938eed0f7/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/b321b279-a102-48b9-bd9b-8ed938eed0f7/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bdc803a1-bbb2-407b-baa5-1aa3a54b7021/volumes/kubernetes.io~projected/kube-api-access-q7cdq:{mountpoint:/var/lib/kubelet/pods/bdc803a1-bbb2-407b-baa5-1aa3a54b7021/volumes/kubernetes.io~projected/kube-api-access-q7cdq major:0 minor:410 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bdc803a1-bbb2-407b-baa5-1aa3a54b7021/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/bdc803a1-bbb2-407b-baa5-1aa3a54b7021/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:408 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bdc803a1-bbb2-407b-baa5-1aa3a54b7021/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/bdc803a1-bbb2-407b-baa5-1aa3a54b7021/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:407 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c00279a4-113f-432d-8531-baa80c405f3a/volumes/kubernetes.io~projected/kube-api-access-7mnxb:{mountpoint:/var/lib/kubelet/pods/c00279a4-113f-432d-8531-baa80c405f3a/volumes/kubernetes.io~projected/kube-api-access-7mnxb major:0 minor:301 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c00279a4-113f-432d-8531-baa80c405f3a/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/c00279a4-113f-432d-8531-baa80c405f3a/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:294 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c00279a4-113f-432d-8531-baa80c405f3a/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/c00279a4-113f-432d-8531-baa80c405f3a/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c00279a4-113f-432d-8531-baa80c405f3a/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/c00279a4-113f-432d-8531-baa80c405f3a/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:300 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ca422300-6212-4a01-b1e2-5ba04af79ce6/volumes/kubernetes.io~projected/kube-api-access-lmzfj:{mountpoint:/var/lib/kubelet/pods/ca422300-6212-4a01-b1e2-5ba04af79ce6/volumes/kubernetes.io~projected/kube-api-access-lmzfj major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cf410657-29d1-408b-a956-e7e5ef548a33/volumes/kubernetes.io~projected/kube-api-access-gr6w5:{mountpoint:/var/lib/kubelet/pods/cf410657-29d1-408b-a956-e7e5ef548a33/volumes/kubernetes.io~projected/kube-api-access-gr6w5 major:0 minor:246 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cf410657-29d1-408b-a956-e7e5ef548a33/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/cf410657-29d1-408b-a956-e7e5ef548a33/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:245 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cf410657-29d1-408b-a956-e7e5ef548a33/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/cf410657-29d1-408b-a956-e7e5ef548a33/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dda66b1a-94e1-40dd-a77d-bfe37dd22b55/volumes/kubernetes.io~projected/kube-api-access-px9vv:{mountpoint:/var/lib/kubelet/pods/dda66b1a-94e1-40dd-a77d-bfe37dd22b55/volumes/kubernetes.io~projected/kube-api-access-px9vv major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dda66b1a-94e1-40dd-a77d-bfe37dd22b55/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/dda66b1a-94e1-40dd-a77d-bfe37dd22b55/volumes/kubernetes.io~secret/metrics-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f09eaaa4-5ed2-4bee-931f-b425019f9c10/volumes/kubernetes.io~projected/kube-api-access-cjw82:{mountpoint:/var/lib/kubelet/pods/f09eaaa4-5ed2-4bee-931f-b425019f9c10/volumes/kubernetes.io~projected/kube-api-access-cjw82 major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f09eaaa4-5ed2-4bee-931f-b425019f9c10/volumes/kubernetes.io~secret/federate-client-tls:{mountpoint:/var/lib/kubelet/pods/f09eaaa4-5ed2-4bee-931f-b425019f9c10/volumes/kubernetes.io~secret/federate-client-tls major:0 minor:308 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f09eaaa4-5ed2-4bee-931f-b425019f9c10/volumes/kubernetes.io~secret/secret-telemeter-client:{mountpoint:/var/lib/kubelet/pods/f09eaaa4-5ed2-4bee-931f-b425019f9c10/volumes/kubernetes.io~secret/secret-telemeter-client major:0 minor:306 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f09eaaa4-5ed2-4bee-931f-b425019f9c10/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/f09eaaa4-5ed2-4bee-931f-b425019f9c10/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config major:0 minor:307 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f09eaaa4-5ed2-4bee-931f-b425019f9c10/volumes/kubernetes.io~secret/telemeter-client-tls:{mountpoint:/var/lib/kubelet/pods/f09eaaa4-5ed2-4bee-931f-b425019f9c10/volumes/kubernetes.io~secret/telemeter-client-tls major:0 minor:309 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7666c14-ff3f-44a5-9cce-21ff2dfda0e2/volumes/kubernetes.io~projected/kube-api-access-srs22:{mountpoint:/var/lib/kubelet/pods/f7666c14-ff3f-44a5-9cce-21ff2dfda0e2/volumes/kubernetes.io~projected/kube-api-access-srs22 major:0 minor:273 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7666c14-ff3f-44a5-9cce-21ff2dfda0e2/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/f7666c14-ff3f-44a5-9cce-21ff2dfda0e2/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:272 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7666c14-ff3f-44a5-9cce-21ff2dfda0e2/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/f7666c14-ff3f-44a5-9cce-21ff2dfda0e2/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:271 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fe292d5d-6d46-4c34-ab24-7991571cd680/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/fe292d5d-6d46-4c34-ab24-7991571cd680/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fe292d5d-6d46-4c34-ab24-7991571cd680/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/fe292d5d-6d46-4c34-ab24-7991571cd680/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fe292d5d-6d46-4c34-ab24-7991571cd680/volumes/kubernetes.io~projected/kube-api-access-bmkz2:{mountpoint:/var/lib/kubelet/pods/fe292d5d-6d46-4c34-ab24-7991571cd680/volumes/kubernetes.io~projected/kube-api-access-bmkz2 major:0 minor:67 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/3d20ba75343a97d96b5a5a32248b441806b8426c90849276d1657d033db0b29a/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/f2470eec5343a5e099dcb10ef07f677c62295f5671e530b6889fee0dd6939f2a/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/de7ec74eb1fd21d4161bfffe22cbe63e4cafec32f3333581d6578aa2970b62a5/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/987c745712ac3ca5cf7c2122c8591fa7b484ef2e89c30516443139e00937b19f/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/b404c22d20614b74e3998e38acabec403222979e8d69797134a07564acebb64c/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/e1cf3f1958e33420e4d7443bd2fc67f9c21b6ea1e5b502cfa98f90d128d6c616/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/f0f22280f881df14509efeb5701c15714fc836035d5212539bdc5601502c0dbe/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/7c8ef7c68dd8ac527a2145fe8ede57aa894410ec2eb7f6b66aef446f9682fd32/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/4355fedb9f14383915551866e8296c7806c113d4386a98e755dad46726a110ca/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-131:{mountpoint:/var/lib/containers/storage/overlay/93ca9302ae6554bb17683326fadc39f1a4880a014757ee70a9147e83a39be38c/merged major:0 minor:131 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/b3906bd4b6801e4d05e3e519bb3543b5916328cb72eb111d4e75cc64008f7a4e/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/b5d49c2b498068ecd558911d4a97b3b005036e43b489d28bcc06ed8113af829a/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/2f5dcd20d8f9fdbdc5e196533ef790fe6cde46c2a0ef37c0c8c6c3f1a196c1b1/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/6f0417415c278489b644c5f4a9b22fdfc6b741c1e81bbbef3330f80fe74639cd/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/cab2dbcc7d7ebde17fb9e75d281d1122730f7c907608f48a53b5812ada86fbb7/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/93ba545fdb0c3e1d031e946df0c16ec4aa6ebe8569ccf189f5321948aa679a35/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/0ce959699b88fb5cc869673b7d289f130a204d8d87425029a0052d7c863f7aeb/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/32058fec476a025ae201ff7518c5d29174d0e3dd28dae553a62165c422046fde/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-181:{mountpoint:/var/lib/containers/storage/overlay/52e97847e4194ab1cd6a3c19ec753bd029d9bdeeedd4e0e3ff7b8972f3e79b31/merged major:0 minor:181 fsType:overlay blockSize:0} overlay_0-188:{mountpoint:/var/lib/containers/storage/overlay/e0480b7603ff1c9282293952a9b0e619dc1e6649d05d98e03383a5667fa08285/merged major:0 minor:188 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/231bc69065019cf4077ff886c4d309cd94ed9e392f9c9b5bc0dfe368020934c8/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/66ca9508d5823a78347beeae9098a34d6baf551adb1e8a811270715861f198ce/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/0ce455b2f71e23b7b7c25a5dcf49a6c2a999c559386324e31ec687792ea91bf7/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/6d3bf12dbe8011ed0b09aa2eb59e85d156724353598db17f5230d487d2e30649/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/d0517ae8a2a74a5eff3272c6caed741a955fd867dc1a2f6dad77198b866b4ff3/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/1d0270710cd8640a2863185d99e6978936d72da68c0d4ca626b3db662b8c5b2c/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/12d22b1750992ec156d86c933867ffab8447161cec01bb519b2f74e394ffe133/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/cdd8c1f3fa9cf5a8f66a760f03cfde43bf79287e2fb2b7ff6a1661816f39f90a/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/120b8d59e9ebcafa47cbc7f2d2d06877eafa00f9280fffa6ae066556ac7f6156/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/58aa6eac0c0de66c045624b05891476ee2c32525b0de4644913e0adbba13c2c7/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-255:{mountpoint:/var/lib/containers/storage/overlay/882bb1f946245be0552147233211c76501dc66fa213523d66cb9fdbb1a508f89/merged major:0 minor:255 fsType:overlay blockSize:0} overlay_0-257:{mountpoint:/var/lib/containers/storage/overlay/9645db54323729fa0e2e1018768adf8f5123becec2d8f7d56046df06d7933234/merged major:0 minor:257 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/6b9a139ed72a3da73a72b264c482110a7ded8ff1c725d197bff443d83def5a5c/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/71282f49c5db639496cb1f1efa9d6b4b42a02816c07fff5ae06f6bf07473eac1/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/5e928370d153ee752b1df94d4feacecf9e8ad8f7f301b9727101d0e7d9b97d0e/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/537df0e0437758b4c45b8b2c2b6ad022b211c9fe5e85763707d57b75a9255a0c/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/faff8ca8a8c1978763edb0ac4e72178007d52d9da3c598a7e06bcdb1aa54843b/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-289:{mountpoint:/var/lib/containers/storage/overlay/430ecc493acc5ae55748ed5266a0f639b8a06a412d8727655f7867ddcbdf1c7e/merged major:0 minor:289 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/3f44dba40d767f87010e94c1ce22de8a507bf1889fb5b2714027e066860ad790/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-313:{mountpoint:/var/lib/containers/storage/overlay/6e463ebccfc5364784ee7325132d9dafa169d23420401224b715e2e282af0d12/merged major:0 minor:313 fsType:overlay blockSize:0} overlay_0-315:{mountpoint:/var/lib/containers/storage/overlay/0f2723999129550bdb39985d99e39c17541b7c089d4281ef54d567b6be9d6f0a/merged major:0 minor:315 fsType:overlay blockSize:0} overlay_0-335:{mountpoint:/var/lib/containers/storage/overlay/be573e7587ba3de9b7f432863254fa3399c56d769bb881979c419b7bedb757e2/merged major:0 minor:335 fsType:overlay blockSize:0} overlay_0-337:{mountpoint:/var/lib/containers/storage/overlay/07c65d5346cbf6e557f595d7a088d60482e3d89c3ddbcf78d71a4884027f6c79/merged major:0 minor:337 fsType:overlay blockSize:0} overlay_0-339:{mountpoint:/var/lib/containers/storage/overlay/8f2fdb7c28b5d475fa1afab324c62bb3f999f7b2137c256331693a271a2703c6/merged major:0 minor:339 fsType:overlay blockSize:0} overlay_0-345:{mountpoint:/var/lib/containers/storage/overlay/ab94243da5e9d583175c9a17b573c5225cddbea8a4e8f3b64ec9d4ebbfb1f942/merged major:0 minor:345 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/bed629dc5c33acb0ffd337240243dec32bacba361294a2017e03197ce003ba64/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-355:{mountpoint:/var/lib/containers/storage/overlay/dcc058397b922a0097357ff9a3768c2815f916b1682533f4501f7830308ccfb1/merged major:0 minor:355 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/a9ca83feaa48649598202e0c8c9f0b0185e9ffdde9e23866aa422b911d79f11b/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-375:{mountpoint:/var/lib/containers/storage/overlay/e5ecc3abe7e5cbe395091a2ba861a8837af0165065242c1e93beb78c867cbbc0/merged major:0 minor:375 fsType:overlay blockSize:0} overlay_0-377:{mountpoint:/var/lib/containers/storage/overlay/808d9f608ca8329f0ccf12faafd4df06ba343ce55e30959bcc7c2bbf52707793/merged major:0 minor:377 fsType:overlay blockSize:0} overlay_0-383:{mountpoint:/var/lib/containers/storage/overlay/5ba661f953089a91201c463aece3cbeb135780c9c2815f068c6fd3f5cc64bc99/merged major:0 minor:383 fsType:overlay blockSize:0} overlay_0-388:{mountpoint:/var/lib/containers/storage/overlay/010399d4a016474e8fe7fee4454fbaa45a10626f247169c9a0fd23c59287d6ab/merged major:0 minor:388 fsType:overlay blockSize:0} overlay_0-393:{mountpoint:/var/lib/containers/storage/overlay/65cab670e307b153daafbe02f25982d18b104e9ae3f0d504a2d9be915034d499/merged major:0 minor:393 fsType:overlay blockSize:0} overlay_0-397:{mountpoint:/var/lib/containers/storage/overlay/86d9294f45a90efc4b4fc75ddb0efc45cfd56b831f6fd4175b460a818823e475/merged major:0 minor:397 fsType:overlay blockSize:0} overlay_0-399:{mountpoint:/var/lib/containers/storage/overlay/a13a04c3260bd6472f288c92e24de62bbb156239ac8f9ddd48053e732f5876a5/merged major:0 minor:399 fsType:overlay blockSize:0} overlay_0-401:{mountpoint:/var/lib/containers/storage/overlay/2c6266ae552c692094951f1c8e120f8ff93917d7c02f39d5b6d564856bd4dbf7/merged major:0 minor:401 fsType:overlay blockSize:0} overlay_0-415:{mountpoint:/var/lib/containers/storage/overlay/a9a10e0dbba28a3158adef71f9e19650c6b8dcbeb1fd4235fc2dfcc6c916a0c7/merged major:0 minor:415 fsType:overlay blockSize:0} overlay_0-417:{mountpoint:/var/lib/containers/storage/overlay/3a9f1189e2a75a7afe31939a24787520cf2c167b8c10d44b65eda1c898fbb2c5/merged major:0 minor:417 fsType:overlay blockSize:0} overlay_0-419:{mountpoint:/var/lib/containers/storage/overlay/0528dc1eebb045cb6a95c6799a2553c66ce172946ec06936220c02aeede406ae/merged major:0 minor:419 fsType:overlay blockSize:0} overlay_0-425:{mountpoint:/var/lib/containers/storage/overlay/301a2daff01927a9749246e8b982bb495a0d431352c281f42bc1416bef5a1ba5/merged major:0 minor:425 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/fe5bcfbd73389003b286ce290aca536b24756631ecf43c240c6ae4301e7579bb/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/af21045aa773b196307b98c0925b8cf01ec43b6d5de8f0d2af26e83e3cd778fa/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/073677c618cbb91135e8ff9c41fd65f5cdd7ab418ddeff61765befa57b309c5e/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/3e492c2d817d03483ddf87c77e0f9f924e17178371a52e2b7daf2caf55c9879a/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/b590f0df5392eec38660071f1a01fe9d18985cf597a0c9a3c49b73c6f97924ff/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/e63da567b02f5c8fccda951fb1f2cae63dbf6379da602f09fc425bf224d8dcac/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/12e212871fdbf96b398e263a0ef2fb206137d2e71364bbd6ad4061185fab2196/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/8d7cddf8a020e8ed90e5db06bdcded7bae372bc49c9f1755ff8bed74fd3ed917/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/2fbd370414b701131991094a9c21f45c5f59a43cdbc4e9300d466c6b61e1fc21/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/d5317023c20aad6550d08d432b782d0baaec0e67fa3be4910169132a74a4840e/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 06:57:21.454210 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.451548    6224 manager.go:217] Machine: {Timestamp:2026-10-30 06:57:21.451088248 +0000 UTC m=+0.060495208 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099954 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2661e942f7794e68faf61846d0e2d2 SystemUUID:ec2661e9-42f7-794e-68fa-f61846d0e2d2 BootID:2b64aa7f-6a0d-404d-87d7-a2a59206956f Filesystems:[{Device:/var/lib/kubelet/pods/f09eaaa4-5ed2-4bee-931f-b425019f9c10/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:307 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/12c298a7-e78a-45d7-abe1-7b7ef6f5df03/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:325 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-257 DeviceMajor:0 DeviceMinor:257 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-397 DeviceMajor:0 DeviceMinor:397 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/31519c22-050e-435c-aec3-7be0b68ff313/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/6f8b75af-cc86-49c6-a66c-1798309f9806/volumes/kubernetes.io~projected/kube-api-access-lmzmm DeviceMajor:0 DeviceMinor:66 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f09eaaa4-5ed2-4bee-931f-b425019f9c10/volumes/kubernetes.io~secret/secret-telemeter-client DeviceMajor:0 DeviceMinor:306 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/61eb4fbe80838432dc327dc06a8580e4a1d35a985986b0c2c113de76ac733a9c/userdata/shm DeviceMajor:0 DeviceMinor:333 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-375 DeviceMajor:0 DeviceMinor:375 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bdc803a1-bbb2-407b-baa5-1aa3a54b7021/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:407 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/bdc803a1-bbb2-407b-baa5-1aa3a54b7021/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:408 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a6968152-3c44-4205-a078-f857c413ad91/volumes/kubernetes.io~projected/kube-api-access-rkb8j DeviceMajor:0 DeviceMinor:64 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3062f537-9b22-4e15-97aa-9a68703702f8/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/12c298a7-e78a-45d7-abe1-7b7ef6f5df03/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:331 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c00279a4-113f-432d-8531-baa80c405f3a/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:294 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c00279a4-113f-432d-8531-baa80c405f3a/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:300 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/12c298a7-e78a-45d7-abe1-7b7ef6f5df03/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:323 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-425 DeviceMajor:0 DeviceMinor:425 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cf410657-29d1-408b-a956-e7e5ef548a33/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:241 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3af572320fb94341a428cb0d1ff52bf19e935a13aa08043036427d1aed82f43b/userdata/shm DeviceMajor:0 DeviceMinor:274 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bdc803a1-bbb2-407b-baa5-1aa3a54b7021/volumes/kubernetes.io~projected/kube-api-access-q7cdq DeviceMajor:0 DeviceMinor:410 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ca422300-6212-4a01-b1e2-5ba04af79ce6/volumes/kubernetes.io~projected/kube-api-access-lmzfj DeviceMajor:0 DeviceMinor:65 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b2f0d0628cf42a429a4f71ec6f40317fa67e7594d24d33bbfc74fd27438d50a9/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ca2afa0103577be424a378670e76be6f79a1d7d5eccc4adf294a66963c7af7ab/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-313 DeviceMajor:0 DeviceMinor:313 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/12c298a7-e78a-45d7-abe1-7b7ef6f5df03/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:328 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-339 DeviceMajor:0 DeviceMinor:339 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9092e2e8-ad23-426c-b6f1-14e1a2b9221e/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:361 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3420c9754d38d1b6f1bf2af5f6dd65d8bc93f12c95d4a4f1e9bdfdfd8a76bd7f/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c00279a4-113f-432d-8531-baa80c405f3a/volumes/kubernetes.io~projected/kube-api-access-7mnxb DeviceMajor:0 DeviceMinor:301 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/12c298a7-e78a-45d7-abe1-7b7ef6f5df03/volumes/kubernetes.io~projected/kube-api-access-n9bkg DeviceMajor:0 DeviceMinor:332 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9c5ccc21-fce2-4221-9c19-4a35fef32eff/volumes/kubernetes.io~projected/kube-api-access-lbpk2 DeviceMajor:0 DeviceMinor:204 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9d24d508-ec6f-4165-8394-f17b09134b34/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:269 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/12c298a7-e78a-45d7-abe1-7b7ef6f5df03/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:317 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/12c298a7-e78a-45d7-abe1-7b7ef6f5df03/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:327 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-355 DeviceMajor:0 DeviceMinor:355 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b321b279-a102-48b9-bd9b-8ed938eed0f7/volumes/kubernetes.io~projected/kube-api-access-8kds7 DeviceMajor:0 DeviceMinor:61 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/06224bc406875c768834b53973ff4ff559b88f05bd498fbe25fa9a80007bec57/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/919f82c3786444358be28752d8866ba97d19b73cf72883656d55e4b3c2b71592/userdata/shm DeviceMajor:0 DeviceMinor:249 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9d24d508-ec6f-4165-8394-f17b09134b34/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:265 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/562e7614715b08bbae86360ec40dcac18e315ad97b411cf869f6f82cf0d30116/userdata/shm DeviceMajor:0 DeviceMinor:302 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-345 DeviceMajor:0 DeviceMinor:345 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fa5bcb3e624bd2241115c27f02a8efaebd5ed1c8d6d0851c075cd1001227e2b9/userdata/shm DeviceMajor:0 DeviceMinor:368 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-377 DeviceMajor:0 DeviceMinor:377 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-399 DeviceMajor:0 DeviceMinor:399 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-417 DeviceMajor:0 DeviceMinor:417 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-419 DeviceMajor:0 DeviceMinor:419 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b321b279-a102-48b9-bd9b-8ed938eed0f7/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0d57907e4151320d3a4ce60e0d55b8fec5237232120b7685df5b8319b65099ca/userdata/shm DeviceMajor:0 DeviceMinor:311 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-383 DeviceMajor:0 DeviceMinor:383 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-131 DeviceMajor:0 DeviceMinor:131 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fe292d5d-6d46-4c34-ab24-7991571cd680/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/851fb6ff-4a58-431b-834f-db3f87b67aa9/volumes/kubernetes.io~projected/kube-api-access-hvdd8 DeviceMajor:0 DeviceMinor:62 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/80a771c73af370f46a40ec02354dc73456551de82c38c22b9f57bbfd09a7c86a/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4cf0039f2b797f9938c56521778329a145c93299deea89534bff76814a1fb9cb/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-388 DeviceMajor:0 DeviceMinor:388 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/dda66b1a-94e1-40dd-a77d-bfe37dd22b55/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9c5ccc21-fce2-4221-9c19-4a35fef32eff/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/12c298a7-e78a-45d7-abe1-7b7ef6f5df03/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:330 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fe292d5d-6d46-4c34-ab24-7991571cd680/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/943ee05bb25ae7fe4215fdaec4395add68669b2a4643aa0d476543979b4c28a6/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-188 DeviceMajor:0 DeviceMinor:188 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1d1240054d453d5b62713793962afa616b2b7c7a1c817ed63febd27be4ceeef9/userdata/shm DeviceMajor:0 DeviceMinor:373 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/cf410657-29d1-408b-a956-e7e5ef548a33/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:245 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/86c6ac0388a077eacf9d769b76ab3a06c1c1dea7ae0982f56c78f7da301fd226/userdata/shm DeviceMajor:0 DeviceMinor:276 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-393 DeviceMajor:0 DeviceMinor:393 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/12c2c077b5e8d30e4257b0ea0ae956cbbab35ee764ee7f301a9dc1e01d073c97/userdata/shm DeviceMajor:0 DeviceMinor:409 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cab1d1b7478b01d62e7af6636d06b4f3300b2fe9d55994cd23840f04d88290c3/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-181 DeviceMajor:0 DeviceMinor:181 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6d493f94-3aef-445e-9b60-32a10c184eac/volumes/kubernetes.io~projected/kube-api-access-2cwrb DeviceMajor:0 DeviceMinor:381 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f09eaaa4-5ed2-4bee-931f-b425019f9c10/volumes/kubernetes.io~secret/federate-client-tls DeviceMajor:0 DeviceMinor:308 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f09eaaa4-5ed2-4bee-931f-b425019f9c10/volumes/kubernetes.io~secret/telemeter-client-tls DeviceMajor:0 DeviceMinor:309 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/5d87089b-8b1b-43ea-be1f-72bfb56bf311/volumes/kubernetes.io~projected/kube-api-access-tlfr5 DeviceMajor:0 DeviceMinor:60 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/12c298a7-e78a-45d7-abe1-7b7ef6f5df03/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:322 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-337 DeviceMajor:0 DeviceMinor:337 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/12c298a7-e78a-45d7-abe1-7b7ef6f5df03/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:331 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/6f8b75af-cc86-49c6-a66c-1798309f9806/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:365 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/2368101c-6719-4650-8393-ef048500a624/volumes/kubernetes.io~projected/kube-api-access-dklkl DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3a3410f94fd4c007d13abc13d7e3d4f70224c98d53ea499adc6ae46982b9eebf/userdata/shm DeviceMajor:0 DeviceMinor:87 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/88c5fae3e58af8696d8b0aa5eba674d92d35383ce719079cc013d2034376d11a/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9d24d508-ec6f-4165-8394-f17b09134b34/volumes/kubernetes.io~projected/kube-api-access-x67rl DeviceMajor:0 DeviceMinor:270 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/12c298a7-e78a-45d7-abe1-7b7ef6f5df03/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:324 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fe292d5d-6d46-4c34-ab24-7991571cd680/volumes/kubernetes.io~projected/kube-api-access-bmkz2 DeviceMajor:0 DeviceMinor:67 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3062f537-9b22-4e15-97aa-9a68703702f8/volumes/kubernetes.io~projected/kube-api-access-wnrqt DeviceMajor:0 DeviceMinor:203 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/93adb19d38e4ac992b2167c0faf831a897937178bfbddca1361abf56ba4cb941/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/12c298a7-e78a-45d7-abe1-7b7ef6f5df03/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:329 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f09eaaa4-5ed2-4bee-931f-b425019f9c10/volumes/kubernetes.io~projected/kube-api-access-cjw82 DeviceMajor:0 DeviceMinor:310 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-335 DeviceMajor:0 DeviceMinor:335 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-401 DeviceMajor:0 DeviceMinor:401 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-415 DeviceMajor:0 DeviceMinor:415 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e04753844ffdbcd2c1f467e3e188f856361230266cc9e241bb967283f893901c/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/dda66b1a-94e1-40dd-a77d-bfe37dd22b55/volumes/kubernetes.io~projected/kube-api-access-px9vv DeviceMajor:0 DeviceMinor:202 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f7666c14-ff3f-44a5-9cce-21ff2dfda0e2/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:271 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-315 DeviceMajor:0 DeviceMinor:315 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/675dc1eb-ff05-40c7-bf0b-fb3161d3cc8c/volumes/kubernetes.io~projected/kube-api-access-b74sr DeviceMajor:0 DeviceMinor:63 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fa9ac36aa63c36f432ecb122126c429701322d5758b168a1026b9c3b6353db26/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/b321b279-a102-48b9-bd9b-8ed938eed0f7/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-255 DeviceMajor:0 DeviceMinor:255 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f7666c14-ff3f-44a5-9cce-21ff2dfda0e2/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:272 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/12c298a7-e78a-45d7-abe1-7b7ef6f5df03/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:326 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/12c298a7-e78a-45d7-abe1-7b7ef6f5df03/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:331 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/12c298a7-e78a-45d7-abe1-7b7ef6f5df03/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:331 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/39db0820745d212e03b98cd445e052d84820fb8ad42c98ed8fa2d370026ecd90/userdata/shm DeviceMajor:0 DeviceMinor:395 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cf410657-29d1-408b-a956-e7e5ef548a33/volumes/kubernetes.io~projected/kube-api-access-gr6w5 DeviceMajor:0 DeviceMinor:246 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f7666c14-ff3f-44a5-9cce-21ff2dfda0e2/volumes/kubernetes.io~projected/kube-api-access-srs22 DeviceMajor:0 DeviceMinor:273 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-289 DeviceMajor:0 DeviceMinor:289 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c00279a4-113f-432d-8531-baa80c405f3a/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:299 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0d57907e4151320 MacAddress:42:2a:f8:36:80:05 Speed:10000 Mtu:8901} {Name:12c2c077b5e8d30 MacAddress:52:19:ec:cd:57:53 Speed:10000 Mtu:8901} {Name:1d1240054d453d5 MacAddress:8e:4b:e6:a5:13:f1 Speed:10000 Mtu:8901} {Name:39db0820745d212 MacAddress:1e:44:f5:ff:8f:d2 Speed:10000 Mtu:8901} {Name:3af572320fb9434 MacAddress:76:74:63:99:cd:84 Speed:10000 Mtu:8901} {Name:4cf0039f2b797f9 MacAddress:6e:c9:a7:f7:dc:67 Speed:10000 Mtu:8901} {Name:562e7614715b08b MacAddress:0e:55:2e:73:9e:06 Speed:10000 Mtu:8901} {Name:61eb4fbe8083843 MacAddress:e6:79:3d:74:10:7b Speed:10000 Mtu:8901} {Name:919f82c37864443 MacAddress:de:c1:f3:fc:c1:2e Speed:10000 Mtu:8901} {Name:93adb19d38e4ac9 MacAddress:5a:63:a0:5a:59:f8 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:ec:79:d4:b7 Speed:0 Mtu:9001} {Name:br-int MacAddress:ca:05:b7:ed:af:6a Speed:0 Mtu:8901} {Name:ca2afa0103577be MacAddress:d6:e0:62:69:c0:dc Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:ec:79:d4:b7 Speed:0 Mtu:9001} {Name:fa5bcb3e624bd22 MacAddress:86:5d:2c:39:22:28 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:b2:f2:cb:64:22:d0 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:96:28:50:21:8e:78 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 06:57:21.454210 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.451925    6224 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 06:57:21.454210 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.451964    6224 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 06:57:21.454210 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.452211    6224 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 06:57:21.454210 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.452230    6224 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-125.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 06:57:21.454210 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.452350    6224 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 06:57:21.454210 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.452357    6224 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 06:57:21.454210 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.452367    6224 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 06:57:21.454210 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.452386    6224 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 06:57:21.454210 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.452446    6224 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 06:57:21.454210 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.452594    6224 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 06:57:21.454210 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.452701    6224 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 06:57:21.454210 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.452731    6224 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 06:57:21.454210 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.452740    6224 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 06:57:21.454210 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.452750    6224 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 06:57:21.454210 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.452759    6224 kubelet.go:405] "Adding apiserver pod source"
Oct 30 06:57:21.454210 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.452767    6224 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 06:57:21.454210 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.453393    6224 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 06:57:21.454210 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.453403    6224 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 06:57:21.454210 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.453896    6224 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 06:57:21.457923 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.457906    6224 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 06:57:21.457923 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.457920    6224 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 06:57:21.458031 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.458017    6224 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 06:57:21.458212 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.458200    6224 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 06:57:21.458243 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.458231    6224 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 06:57:21.458338 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.458329    6224 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 06:57:21.458367 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.458344    6224 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 06:57:21.458367 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.458350    6224 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 06:57:21.458367 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.458355    6224 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 06:57:21.458367 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.458363    6224 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 06:57:21.458467 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.458373    6224 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 06:57:21.458467 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.458382    6224 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 06:57:21.458467 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.458390    6224 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 06:57:21.458467 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.458398    6224 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 06:57:21.458467 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.458403    6224 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 06:57:21.458467 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.458414    6224 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 06:57:21.458467 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.458423    6224 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 06:57:21.458467 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.458449    6224 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 06:57:21.458467 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.458457    6224 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 06:57:21.461739 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.461724    6224 server.go:1263] "Started kubelet"
Oct 30 06:57:21.461827 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.461768    6224 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 06:57:21.462418 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.462229    6224 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 06:57:21.462510 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.462441    6224 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 06:57:21.462459 ip-10-0-128-125 systemd[1]: Started Kubernetes Kubelet.
Oct 30 06:57:21.462882 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.462863    6224 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 06:57:21.463179 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.463123    6224 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 06:57:21.464966 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.464946    6224 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 06:57:21.464966 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.464970    6224 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 06:57:21.465506 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.465065    6224 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 06:57:21.465506 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.465075    6224 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 06:57:21.465506 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.465104    6224 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 06:51:12 +0000 UTC" deadline="2028-08-13 20:55:31.475351434 +0000 UTC"
Oct 30 06:57:21.465506 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.465136    6224 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15685h58m10.010219251s"
Oct 30 06:57:21.465506 ip-10-0-128-125 kubenswrapper[6224]: E1030 06:57:21.465155    6224 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-125.ec2.internal\" not found"
Oct 30 06:57:21.465506 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.465228    6224 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 06:57:21.465959 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.465897    6224 factory.go:55] Registering systemd factory
Oct 30 06:57:21.465959 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.465914    6224 factory.go:223] Registration of the systemd container factory successfully
Oct 30 06:57:21.466152 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.466139    6224 factory.go:153] Registering CRI-O factory
Oct 30 06:57:21.466234 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.466155    6224 factory.go:223] Registration of the crio container factory successfully
Oct 30 06:57:21.466234 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.466206    6224 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 06:57:21.466234 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.466228    6224 factory.go:103] Registering Raw factory
Oct 30 06:57:21.466393 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.466243    6224 manager.go:1199] Started watching for new ooms in manager
Oct 30 06:57:21.466932 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.466912    6224 manager.go:319] Starting recovery of all containers
Oct 30 06:57:21.470917 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.470875    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12c298a7-e78a-45d7-abe1-7b7ef6f5df03" volumeName="kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-prometheus-k8s-rulefiles-1" seLinuxMountContext=""
Oct 30 06:57:21.471009 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.470942    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d87089b-8b1b-43ea-be1f-72bfb56bf311" volumeName="kubernetes.io/configmap/5d87089b-8b1b-43ea-be1f-72bfb56bf311-serviceca" seLinuxMountContext=""
Oct 30 06:57:21.471009 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.470970    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bdc803a1-bbb2-407b-baa5-1aa3a54b7021" volumeName="kubernetes.io/secret/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-console-oauth-config" seLinuxMountContext=""
Oct 30 06:57:21.471009 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.470990    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca422300-6212-4a01-b1e2-5ba04af79ce6" volumeName="kubernetes.io/projected/ca422300-6212-4a01-b1e2-5ba04af79ce6-kube-api-access-lmzfj" seLinuxMountContext=""
Oct 30 06:57:21.471168 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.471007    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12c298a7-e78a-45d7-abe1-7b7ef6f5df03" volumeName="kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 30 06:57:21.471168 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.471027    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bdc803a1-bbb2-407b-baa5-1aa3a54b7021" volumeName="kubernetes.io/configmap/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-service-ca" seLinuxMountContext=""
Oct 30 06:57:21.471168 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.471041    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bdc803a1-bbb2-407b-baa5-1aa3a54b7021" volumeName="kubernetes.io/secret/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-console-serving-cert" seLinuxMountContext=""
Oct 30 06:57:21.471168 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.471060    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7666c14-ff3f-44a5-9cce-21ff2dfda0e2" volumeName="kubernetes.io/configmap/f7666c14-ff3f-44a5-9cce-21ff2dfda0e2-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Oct 30 06:57:21.471168 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.471080    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12c298a7-e78a-45d7-abe1-7b7ef6f5df03" volumeName="kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-secret-grpc-tls" seLinuxMountContext=""
Oct 30 06:57:21.471168 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.471110    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12c298a7-e78a-45d7-abe1-7b7ef6f5df03" volumeName="kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-secret-kube-rbac-proxy" seLinuxMountContext=""
Oct 30 06:57:21.471168 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.471126    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12c298a7-e78a-45d7-abe1-7b7ef6f5df03" volumeName="kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-web-config" seLinuxMountContext=""
Oct 30 06:57:21.471168 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.471139    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c5ccc21-fce2-4221-9c19-4a35fef32eff" volumeName="kubernetes.io/configmap/9c5ccc21-fce2-4221-9c19-4a35fef32eff-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 30 06:57:21.471488 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.471158    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c5ccc21-fce2-4221-9c19-4a35fef32eff" volumeName="kubernetes.io/projected/9c5ccc21-fce2-4221-9c19-4a35fef32eff-kube-api-access-lbpk2" seLinuxMountContext=""
Oct 30 06:57:21.471488 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.471186    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d24d508-ec6f-4165-8394-f17b09134b34" volumeName="kubernetes.io/secret/9d24d508-ec6f-4165-8394-f17b09134b34-console-serving-cert" seLinuxMountContext=""
Oct 30 06:57:21.471488 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.471210    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b321b279-a102-48b9-bd9b-8ed938eed0f7" volumeName="kubernetes.io/configmap/b321b279-a102-48b9-bd9b-8ed938eed0f7-env-overrides" seLinuxMountContext=""
Oct 30 06:57:21.471488 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.471223    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b321b279-a102-48b9-bd9b-8ed938eed0f7" volumeName="kubernetes.io/configmap/b321b279-a102-48b9-bd9b-8ed938eed0f7-ovnkube-config" seLinuxMountContext=""
Oct 30 06:57:21.471488 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.471242    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12c298a7-e78a-45d7-abe1-7b7ef6f5df03" volumeName="kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-config" seLinuxMountContext=""
Oct 30 06:57:21.471488 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.471254    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2368101c-6719-4650-8393-ef048500a624" volumeName="kubernetes.io/projected/2368101c-6719-4650-8393-ef048500a624-kube-api-access-dklkl" seLinuxMountContext=""
Oct 30 06:57:21.471488 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.471267    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c5ccc21-fce2-4221-9c19-4a35fef32eff" volumeName="kubernetes.io/empty-dir/9c5ccc21-fce2-4221-9c19-4a35fef32eff-data-volume" seLinuxMountContext=""
Oct 30 06:57:21.471488 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.471282    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d24d508-ec6f-4165-8394-f17b09134b34" volumeName="kubernetes.io/secret/9d24d508-ec6f-4165-8394-f17b09134b34-console-oauth-config" seLinuxMountContext=""
Oct 30 06:57:21.471488 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.471294    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bdc803a1-bbb2-407b-baa5-1aa3a54b7021" volumeName="kubernetes.io/configmap/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-oauth-serving-cert" seLinuxMountContext=""
Oct 30 06:57:21.471488 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.471310    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c00279a4-113f-432d-8531-baa80c405f3a" volumeName="kubernetes.io/configmap/c00279a4-113f-432d-8531-baa80c405f3a-metrics-server-audit-profiles" seLinuxMountContext=""
Oct 30 06:57:21.471488 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.471322    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c00279a4-113f-432d-8531-baa80c405f3a" volumeName="kubernetes.io/empty-dir/c00279a4-113f-432d-8531-baa80c405f3a-audit-log" seLinuxMountContext=""
Oct 30 06:57:21.471488 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.471334    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c00279a4-113f-432d-8531-baa80c405f3a" volumeName="kubernetes.io/secret/c00279a4-113f-432d-8531-baa80c405f3a-client-ca-bundle" seLinuxMountContext=""
Oct 30 06:57:21.471488 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.471355    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12c298a7-e78a-45d7-abe1-7b7ef6f5df03" volumeName="kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-configmap-serving-certs-ca-bundle" seLinuxMountContext=""
Oct 30 06:57:21.471488 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.471368    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b321b279-a102-48b9-bd9b-8ed938eed0f7" volumeName="kubernetes.io/projected/b321b279-a102-48b9-bd9b-8ed938eed0f7-kube-api-access-8kds7" seLinuxMountContext=""
Oct 30 06:57:21.471488 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.471385    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf410657-29d1-408b-a956-e7e5ef548a33" volumeName="kubernetes.io/configmap/cf410657-29d1-408b-a956-e7e5ef548a33-metrics-client-ca" seLinuxMountContext=""
Oct 30 06:57:21.471488 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.471398    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f09eaaa4-5ed2-4bee-931f-b425019f9c10" volumeName="kubernetes.io/configmap/f09eaaa4-5ed2-4bee-931f-b425019f9c10-metrics-client-ca" seLinuxMountContext=""
Oct 30 06:57:21.471488 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.471422    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f09eaaa4-5ed2-4bee-931f-b425019f9c10" volumeName="kubernetes.io/projected/f09eaaa4-5ed2-4bee-931f-b425019f9c10-kube-api-access-cjw82" seLinuxMountContext=""
Oct 30 06:57:21.471488 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.471435    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7666c14-ff3f-44a5-9cce-21ff2dfda0e2" volumeName="kubernetes.io/secret/f7666c14-ff3f-44a5-9cce-21ff2dfda0e2-node-exporter-tls" seLinuxMountContext=""
Oct 30 06:57:21.471488 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.471452    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12c298a7-e78a-45d7-abe1-7b7ef6f5df03" volumeName="kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-thanos-prometheus-http-client-file" seLinuxMountContext=""
Oct 30 06:57:21.471488 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.471464    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31519c22-050e-435c-aec3-7be0b68ff313" volumeName="kubernetes.io/secret/31519c22-050e-435c-aec3-7be0b68ff313-agent-certs" seLinuxMountContext=""
Oct 30 06:57:21.471488 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.471480    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="675dc1eb-ff05-40c7-bf0b-fb3161d3cc8c" volumeName="kubernetes.io/empty-dir/675dc1eb-ff05-40c7-bf0b-fb3161d3cc8c-tmp-dir" seLinuxMountContext=""
Oct 30 06:57:21.471488 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.471495    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c00279a4-113f-432d-8531-baa80c405f3a" volumeName="kubernetes.io/projected/c00279a4-113f-432d-8531-baa80c405f3a-kube-api-access-7mnxb" seLinuxMountContext=""
Oct 30 06:57:21.472443 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.471507    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dda66b1a-94e1-40dd-a77d-bfe37dd22b55" volumeName="kubernetes.io/secret/dda66b1a-94e1-40dd-a77d-bfe37dd22b55-metrics-tls" seLinuxMountContext=""
Oct 30 06:57:21.472443 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.471524    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f09eaaa4-5ed2-4bee-931f-b425019f9c10" volumeName="kubernetes.io/secret/f09eaaa4-5ed2-4bee-931f-b425019f9c10-secret-telemeter-client" seLinuxMountContext=""
Oct 30 06:57:21.472443 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.471537    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12c298a7-e78a-45d7-abe1-7b7ef6f5df03" volumeName="kubernetes.io/empty-dir/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-prometheus-k8s-db" seLinuxMountContext=""
Oct 30 06:57:21.472443 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.471580    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12c298a7-e78a-45d7-abe1-7b7ef6f5df03" volumeName="kubernetes.io/empty-dir/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-config-out" seLinuxMountContext=""
Oct 30 06:57:21.472443 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.471592    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f09eaaa4-5ed2-4bee-931f-b425019f9c10" volumeName="kubernetes.io/configmap/f09eaaa4-5ed2-4bee-931f-b425019f9c10-telemeter-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 06:57:21.472443 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.471605    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f09eaaa4-5ed2-4bee-931f-b425019f9c10" volumeName="kubernetes.io/secret/f09eaaa4-5ed2-4bee-931f-b425019f9c10-telemeter-client-tls" seLinuxMountContext=""
Oct 30 06:57:21.472443 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.471621    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7666c14-ff3f-44a5-9cce-21ff2dfda0e2" volumeName="kubernetes.io/configmap/f7666c14-ff3f-44a5-9cce-21ff2dfda0e2-metrics-client-ca" seLinuxMountContext=""
Oct 30 06:57:21.472443 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.471673    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7666c14-ff3f-44a5-9cce-21ff2dfda0e2" volumeName="kubernetes.io/empty-dir/f7666c14-ff3f-44a5-9cce-21ff2dfda0e2-node-exporter-textfile" seLinuxMountContext=""
Oct 30 06:57:21.472443 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.471687    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7666c14-ff3f-44a5-9cce-21ff2dfda0e2" volumeName="kubernetes.io/projected/f7666c14-ff3f-44a5-9cce-21ff2dfda0e2-kube-api-access-srs22" seLinuxMountContext=""
Oct 30 06:57:21.472443 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.472320    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12c298a7-e78a-45d7-abe1-7b7ef6f5df03" volumeName="kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-prometheus-k8s-rulefiles-2" seLinuxMountContext=""
Oct 30 06:57:21.472443 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.472341    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12c298a7-e78a-45d7-abe1-7b7ef6f5df03" volumeName="kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-prometheus-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 06:57:21.472443 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.472356    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3062f537-9b22-4e15-97aa-9a68703702f8" volumeName="kubernetes.io/secret/3062f537-9b22-4e15-97aa-9a68703702f8-cert" seLinuxMountContext=""
Oct 30 06:57:21.472443 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.472370    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="851fb6ff-4a58-431b-834f-db3f87b67aa9" volumeName="kubernetes.io/configmap/851fb6ff-4a58-431b-834f-db3f87b67aa9-iptables-alerter-script" seLinuxMountContext=""
Oct 30 06:57:21.472443 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.472394    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b321b279-a102-48b9-bd9b-8ed938eed0f7" volumeName="kubernetes.io/configmap/b321b279-a102-48b9-bd9b-8ed938eed0f7-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 06:57:21.472443 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.472406    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bdc803a1-bbb2-407b-baa5-1aa3a54b7021" volumeName="kubernetes.io/configmap/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 06:57:21.472443 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.472417    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf410657-29d1-408b-a956-e7e5ef548a33" volumeName="kubernetes.io/projected/cf410657-29d1-408b-a956-e7e5ef548a33-kube-api-access-gr6w5" seLinuxMountContext=""
Oct 30 06:57:21.472443 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.472429    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe292d5d-6d46-4c34-ab24-7991571cd680" volumeName="kubernetes.io/empty-dir/fe292d5d-6d46-4c34-ab24-7991571cd680-etc-tuned" seLinuxMountContext=""
Oct 30 06:57:21.472443 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.472441    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12c298a7-e78a-45d7-abe1-7b7ef6f5df03" volumeName="kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Oct 30 06:57:21.472443 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.472452    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12c298a7-e78a-45d7-abe1-7b7ef6f5df03" volumeName="kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext=""
Oct 30 06:57:21.473378 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.472467    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d87089b-8b1b-43ea-be1f-72bfb56bf311" volumeName="kubernetes.io/projected/5d87089b-8b1b-43ea-be1f-72bfb56bf311-kube-api-access-tlfr5" seLinuxMountContext=""
Oct 30 06:57:21.473378 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.472479    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="851fb6ff-4a58-431b-834f-db3f87b67aa9" volumeName="kubernetes.io/projected/851fb6ff-4a58-431b-834f-db3f87b67aa9-kube-api-access-hvdd8" seLinuxMountContext=""
Oct 30 06:57:21.473378 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.472492    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d24d508-ec6f-4165-8394-f17b09134b34" volumeName="kubernetes.io/configmap/9d24d508-ec6f-4165-8394-f17b09134b34-oauth-serving-cert" seLinuxMountContext=""
Oct 30 06:57:21.473378 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.472513    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bdc803a1-bbb2-407b-baa5-1aa3a54b7021" volumeName="kubernetes.io/configmap/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-console-config" seLinuxMountContext=""
Oct 30 06:57:21.473378 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.472524    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f09eaaa4-5ed2-4bee-931f-b425019f9c10" volumeName="kubernetes.io/secret/f09eaaa4-5ed2-4bee-931f-b425019f9c10-secret-telemeter-client-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 06:57:21.473378 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.472546    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe292d5d-6d46-4c34-ab24-7991571cd680" volumeName="kubernetes.io/projected/fe292d5d-6d46-4c34-ab24-7991571cd680-kube-api-access-bmkz2" seLinuxMountContext=""
Oct 30 06:57:21.473378 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.472559    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12c298a7-e78a-45d7-abe1-7b7ef6f5df03" volumeName="kubernetes.io/projected/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-kube-api-access-n9bkg" seLinuxMountContext=""
Oct 30 06:57:21.473378 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.472571    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2368101c-6719-4650-8393-ef048500a624" volumeName="kubernetes.io/configmap/2368101c-6719-4650-8393-ef048500a624-cni-binary-copy" seLinuxMountContext=""
Oct 30 06:57:21.473378 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.472582    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d24d508-ec6f-4165-8394-f17b09134b34" volumeName="kubernetes.io/configmap/9d24d508-ec6f-4165-8394-f17b09134b34-service-ca" seLinuxMountContext=""
Oct 30 06:57:21.473378 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.472594    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d24d508-ec6f-4165-8394-f17b09134b34" volumeName="kubernetes.io/configmap/9d24d508-ec6f-4165-8394-f17b09134b34-console-config" seLinuxMountContext=""
Oct 30 06:57:21.473378 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.472606    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d24d508-ec6f-4165-8394-f17b09134b34" volumeName="kubernetes.io/empty-dir/9d24d508-ec6f-4165-8394-f17b09134b34-tmp" seLinuxMountContext=""
Oct 30 06:57:21.473378 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.472619    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf410657-29d1-408b-a956-e7e5ef548a33" volumeName="kubernetes.io/secret/cf410657-29d1-408b-a956-e7e5ef548a33-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 06:57:21.473378 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.472633    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dda66b1a-94e1-40dd-a77d-bfe37dd22b55" volumeName="kubernetes.io/empty-dir/dda66b1a-94e1-40dd-a77d-bfe37dd22b55-tmp-dir" seLinuxMountContext=""
Oct 30 06:57:21.473378 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.472646    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dda66b1a-94e1-40dd-a77d-bfe37dd22b55" volumeName="kubernetes.io/projected/dda66b1a-94e1-40dd-a77d-bfe37dd22b55-kube-api-access-px9vv" seLinuxMountContext=""
Oct 30 06:57:21.473378 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.472659    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12c298a7-e78a-45d7-abe1-7b7ef6f5df03" volumeName="kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-secret-prometheus-k8s-tls" seLinuxMountContext=""
Oct 30 06:57:21.473378 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.472671    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2368101c-6719-4650-8393-ef048500a624" volumeName="kubernetes.io/configmap/2368101c-6719-4650-8393-ef048500a624-multus-daemon-config" seLinuxMountContext=""
Oct 30 06:57:21.473378 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.472684    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f09eaaa4-5ed2-4bee-931f-b425019f9c10" volumeName="kubernetes.io/configmap/f09eaaa4-5ed2-4bee-931f-b425019f9c10-serving-certs-ca-bundle" seLinuxMountContext=""
Oct 30 06:57:21.473378 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.472697    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f09eaaa4-5ed2-4bee-931f-b425019f9c10" volumeName="kubernetes.io/secret/f09eaaa4-5ed2-4bee-931f-b425019f9c10-federate-client-tls" seLinuxMountContext=""
Oct 30 06:57:21.473378 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.472711    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12c298a7-e78a-45d7-abe1-7b7ef6f5df03" volumeName="kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-configmap-metrics-client-ca" seLinuxMountContext=""
Oct 30 06:57:21.473378 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.472724    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12c298a7-e78a-45d7-abe1-7b7ef6f5df03" volumeName="kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-prometheus-k8s-rulefiles-0" seLinuxMountContext=""
Oct 30 06:57:21.473378 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.472737    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12c298a7-e78a-45d7-abe1-7b7ef6f5df03" volumeName="kubernetes.io/projected/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-tls-assets" seLinuxMountContext=""
Oct 30 06:57:21.473378 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.472749    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3062f537-9b22-4e15-97aa-9a68703702f8" volumeName="kubernetes.io/projected/3062f537-9b22-4e15-97aa-9a68703702f8-kube-api-access-wnrqt" seLinuxMountContext=""
Oct 30 06:57:21.473378 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.472763    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6d493f94-3aef-445e-9b60-32a10c184eac" volumeName="kubernetes.io/projected/6d493f94-3aef-445e-9b60-32a10c184eac-kube-api-access-2cwrb" seLinuxMountContext=""
Oct 30 06:57:21.473378 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.472776    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f8b75af-cc86-49c6-a66c-1798309f9806" volumeName="kubernetes.io/projected/6f8b75af-cc86-49c6-a66c-1798309f9806-kube-api-access-lmzmm" seLinuxMountContext=""
Oct 30 06:57:21.473378 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.472791    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f8b75af-cc86-49c6-a66c-1798309f9806" volumeName="kubernetes.io/secret/6f8b75af-cc86-49c6-a66c-1798309f9806-metrics-certs" seLinuxMountContext=""
Oct 30 06:57:21.473378 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.472899    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d24d508-ec6f-4165-8394-f17b09134b34" volumeName="kubernetes.io/projected/9d24d508-ec6f-4165-8394-f17b09134b34-kube-api-access-x67rl" seLinuxMountContext=""
Oct 30 06:57:21.473378 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.472913    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="675dc1eb-ff05-40c7-bf0b-fb3161d3cc8c" volumeName="kubernetes.io/projected/675dc1eb-ff05-40c7-bf0b-fb3161d3cc8c-kube-api-access-b74sr" seLinuxMountContext=""
Oct 30 06:57:21.473378 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.472925    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9092e2e8-ad23-426c-b6f1-14e1a2b9221e" volumeName="kubernetes.io/secret/9092e2e8-ad23-426c-b6f1-14e1a2b9221e-original-pull-secret" seLinuxMountContext=""
Oct 30 06:57:21.473378 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.472939    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6968152-3c44-4205-a078-f857c413ad91" volumeName="kubernetes.io/projected/a6968152-3c44-4205-a078-f857c413ad91-kube-api-access-rkb8j" seLinuxMountContext=""
Oct 30 06:57:21.473378 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.472952    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b321b279-a102-48b9-bd9b-8ed938eed0f7" volumeName="kubernetes.io/secret/b321b279-a102-48b9-bd9b-8ed938eed0f7-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 06:57:21.473378 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.472967    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bdc803a1-bbb2-407b-baa5-1aa3a54b7021" volumeName="kubernetes.io/projected/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-kube-api-access-q7cdq" seLinuxMountContext=""
Oct 30 06:57:21.473378 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.472980    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c00279a4-113f-432d-8531-baa80c405f3a" volumeName="kubernetes.io/configmap/c00279a4-113f-432d-8531-baa80c405f3a-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Oct 30 06:57:21.473378 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.472992    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c00279a4-113f-432d-8531-baa80c405f3a" volumeName="kubernetes.io/secret/c00279a4-113f-432d-8531-baa80c405f3a-secret-metrics-server-client-certs" seLinuxMountContext=""
Oct 30 06:57:21.473378 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.473006    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca422300-6212-4a01-b1e2-5ba04af79ce6" volumeName="kubernetes.io/configmap/ca422300-6212-4a01-b1e2-5ba04af79ce6-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 06:57:21.473378 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.473062    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31519c22-050e-435c-aec3-7be0b68ff313" volumeName="kubernetes.io/configmap/31519c22-050e-435c-aec3-7be0b68ff313-konnectivity-ca" seLinuxMountContext=""
Oct 30 06:57:21.473378 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.473077    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c5ccc21-fce2-4221-9c19-4a35fef32eff" volumeName="kubernetes.io/secret/9c5ccc21-fce2-4221-9c19-4a35fef32eff-insights-runtime-extractor-tls" seLinuxMountContext=""
Oct 30 06:57:21.473378 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.473095    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bdc803a1-bbb2-407b-baa5-1aa3a54b7021" volumeName="kubernetes.io/empty-dir/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-tmp" seLinuxMountContext=""
Oct 30 06:57:21.473378 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.473111    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c00279a4-113f-432d-8531-baa80c405f3a" volumeName="kubernetes.io/secret/c00279a4-113f-432d-8531-baa80c405f3a-secret-metrics-server-tls" seLinuxMountContext=""
Oct 30 06:57:21.473378 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.473124    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca422300-6212-4a01-b1e2-5ba04af79ce6" volumeName="kubernetes.io/configmap/ca422300-6212-4a01-b1e2-5ba04af79ce6-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 06:57:21.473378 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.473146    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dda66b1a-94e1-40dd-a77d-bfe37dd22b55" volumeName="kubernetes.io/configmap/dda66b1a-94e1-40dd-a77d-bfe37dd22b55-config-volume" seLinuxMountContext=""
Oct 30 06:57:21.473378 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.473159    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe292d5d-6d46-4c34-ab24-7991571cd680" volumeName="kubernetes.io/empty-dir/fe292d5d-6d46-4c34-ab24-7991571cd680-tmp" seLinuxMountContext=""
Oct 30 06:57:21.473378 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.473174    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12c298a7-e78a-45d7-abe1-7b7ef6f5df03" volumeName="kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-secret-metrics-client-certs" seLinuxMountContext=""
Oct 30 06:57:21.473378 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.473191    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca422300-6212-4a01-b1e2-5ba04af79ce6" volumeName="kubernetes.io/configmap/ca422300-6212-4a01-b1e2-5ba04af79ce6-cni-binary-copy" seLinuxMountContext=""
Oct 30 06:57:21.473378 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.473205    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf410657-29d1-408b-a956-e7e5ef548a33" volumeName="kubernetes.io/secret/cf410657-29d1-408b-a956-e7e5ef548a33-prometheus-operator-tls" seLinuxMountContext=""
Oct 30 06:57:21.473378 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.473218    6224 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7666c14-ff3f-44a5-9cce-21ff2dfda0e2" volumeName="kubernetes.io/secret/f7666c14-ff3f-44a5-9cce-21ff2dfda0e2-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 06:57:21.473378 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.473230    6224 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 06:57:21.473378 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.473237    6224 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 06:57:21.493742 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.493704    6224 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 06:57:21.495237 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.495201    6224 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 06:57:21.495237 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.495220    6224 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 06:57:21.495237 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.495239    6224 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 06:57:21.495456 ip-10-0-128-125 kubenswrapper[6224]: E1030 06:57:21.495281    6224 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 06:57:21.506798 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.506763    6224 generic.go:356] "Generic (PLEG): container finished" podID="0aefa49a1bd7275a0d110a8cd8b72657" containerID="18f085935a5cc80d1a21521a5383aa41052013a0eb2dbaeabb981d4f3977f462" exitCode=0
Oct 30 06:57:21.511146 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.511117    6224 generic.go:356] "Generic (PLEG): container finished" podID="12c298a7-e78a-45d7-abe1-7b7ef6f5df03" containerID="7ea8feb9571f349fc2119c3f7c6e1421fe27e22e55f90a4b5370acc7af3d92bb" exitCode=0
Oct 30 06:57:21.516224 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.516201    6224 generic.go:356] "Generic (PLEG): container finished" podID="f7666c14-ff3f-44a5-9cce-21ff2dfda0e2" containerID="e8fddbe735abe1593d14d52087d383e4c21cdfc06e1af64d5b9495ef30b4b862" exitCode=0
Oct 30 06:57:21.527907 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.527890    6224 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 06:57:21.528021 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.527917    6224 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 06:57:21.528021 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.527974    6224 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 06:57:21.528317 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.528303    6224 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 06:57:21.544999 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.544970    6224 generic.go:356] "Generic (PLEG): container finished" podID="ca422300-6212-4a01-b1e2-5ba04af79ce6" containerID="faa60c8808a69c43b547e9ae5181be751acaee31878affe4d0b60f356a31a544" exitCode=0
Oct 30 06:57:21.544999 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.545001    6224 generic.go:356] "Generic (PLEG): container finished" podID="ca422300-6212-4a01-b1e2-5ba04af79ce6" containerID="f419063bee1726a21df48a53721f93506cc649b2e339443cecf2ef494b95ffc5" exitCode=0
Oct 30 06:57:21.545156 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.545013    6224 generic.go:356] "Generic (PLEG): container finished" podID="ca422300-6212-4a01-b1e2-5ba04af79ce6" containerID="96d297ea28178d5d113805bce6db022c0f37866299edf48ccbcf20e18076d280" exitCode=0
Oct 30 06:57:21.545156 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.545022    6224 generic.go:356] "Generic (PLEG): container finished" podID="ca422300-6212-4a01-b1e2-5ba04af79ce6" containerID="1a939e120e007404225424b099320dded90571a77f93e2ba4aeffa40d4869883" exitCode=0
Oct 30 06:57:21.545156 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.545034    6224 generic.go:356] "Generic (PLEG): container finished" podID="ca422300-6212-4a01-b1e2-5ba04af79ce6" containerID="b0f4b3989ce0cc9774d660e0f4ec409d7bc78150a1cd72d6b842fdcd65c2303e" exitCode=0
Oct 30 06:57:21.545156 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.545043    6224 generic.go:356] "Generic (PLEG): container finished" podID="ca422300-6212-4a01-b1e2-5ba04af79ce6" containerID="baa0a62281350432aff794f00c6f91cfd26c0bf9f27c2a7d51e8b9c2db32aa91" exitCode=0
Oct 30 06:57:21.551472 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.551455    6224 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 06:57:21.574838 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.574819    6224 manager.go:324] Recovery completed
Oct 30 06:57:21.596008 ip-10-0-128-125 kubenswrapper[6224]: E1030 06:57:21.595988    6224 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 30 06:57:21.597157 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.597131    6224 cpu_manager.go:226] "Starting" policy="none"
Oct 30 06:57:21.597157 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.597155    6224 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 06:57:21.597298 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.597175    6224 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 06:57:21.597356 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.597318    6224 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 06:57:21.597356 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.597333    6224 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 06:57:21.597450 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.597356    6224 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 06:57:21.597450 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.597364    6224 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 06:57:21.597450 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.597372    6224 policy_none.go:50] "Start"
Oct 30 06:57:21.597450 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.597384    6224 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 06:57:21.597450 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.597397    6224 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 06:57:21.597685 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.597569    6224 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 06:57:21.597685 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.597579    6224 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 06:57:21.597685 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.597588    6224 policy_none.go:44] "Start"
Oct 30 06:57:21.601932 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.601905    6224 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 06:57:21.604477 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.604465    6224 manager.go:341] "Starting Device Plugin manager"
Oct 30 06:57:21.605001 ip-10-0-128-125 kubenswrapper[6224]: E1030 06:57:21.604495    6224 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 06:57:21.605074 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.605003    6224 server.go:87] "Starting device plugin registration server"
Oct 30 06:57:21.605259 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.605249    6224 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 06:57:21.605296 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.605260    6224 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 06:57:21.605341 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.605327    6224 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 06:57:21.605439 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.605408    6224 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 06:57:21.605491 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.605455    6224 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 06:57:21.605491 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.605462    6224 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 06:57:21.605600 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.605524    6224 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T06:57:21.605439136Z","UUID":"29188c9c-5eb5-41c9-adb6-fe8e7d269ae0","Handler":null,"Name":"","Endpoint":""}
Oct 30 06:57:21.607826 ip-10-0-128-125 kubenswrapper[6224]: E1030 06:57:21.607360    6224 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 06:57:21.607826 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.607573    6224 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 06:57:21.607826 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.607595    6224 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 06:57:21.706028 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.705957    6224 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 06:57:21.707698 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.707679    6224 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-125.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 06:57:21.707786 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.707710    6224 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-125.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 06:57:21.707786 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.707718    6224 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-125.ec2.internal" event="NodeHasSufficientPID"
Oct 30 06:57:21.707786 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.707761    6224 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-125.ec2.internal"
Oct 30 06:57:21.745549 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.745529    6224 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-128-125.ec2.internal"
Oct 30 06:57:21.745618 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.745579    6224 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-125.ec2.internal"
Oct 30 06:57:21.796214 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.796177    6224 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-125.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-125.ec2.internal"]
Oct 30 06:57:21.796738 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.796704    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-125.ec2.internal" event={"ID":"0aefa49a1bd7275a0d110a8cd8b72657","Type":"ContainerStarted","Data":"407f2fdca3e939433b7fdbd96d800d7d4463a0536d04c3711b1bc838e0a189ee"}
Oct 30 06:57:21.796853 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.796741    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-125.ec2.internal" event={"ID":"0aefa49a1bd7275a0d110a8cd8b72657","Type":"ContainerDied","Data":"18f085935a5cc80d1a21521a5383aa41052013a0eb2dbaeabb981d4f3977f462"}
Oct 30 06:57:21.796853 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.796756    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-125.ec2.internal" event={"ID":"0aefa49a1bd7275a0d110a8cd8b72657","Type":"ContainerStarted","Data":"3420c9754d38d1b6f1bf2af5f6dd65d8bc93f12c95d4a4f1e9bdfdfd8a76bd7f"}
Oct 30 06:57:21.796853 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.796797    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-125.ec2.internal" event={"ID":"4ba2b683794afdaba9ca4ed884bfac10","Type":"ContainerStarted","Data":"46906adf2a26b7edee73da7546eae9e7e092b885fd2165608c50a391c952c662"}
Oct 30 06:57:21.796853 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.796825    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-125.ec2.internal" event={"ID":"4ba2b683794afdaba9ca4ed884bfac10","Type":"ContainerStarted","Data":"cab1d1b7478b01d62e7af6636d06b4f3300b2fe9d55994cd23840f04d88290c3"}
Oct 30 06:57:21.796985 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.796890    6224 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-125.ec2.internal"
Oct 30 06:57:21.796985 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.796890    6224 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-125.ec2.internal"
Oct 30 06:57:21.821898 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.821881    6224 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 06:57:21.821989 ip-10-0-128-125 kubenswrapper[6224]: E1030 06:57:21.821922    6224 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-125.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-125.ec2.internal"
Oct 30 06:57:21.822677 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.822659    6224 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 06:57:21.822734 ip-10-0-128-125 kubenswrapper[6224]: E1030 06:57:21.822699    6224 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-125.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-125.ec2.internal"
Oct 30 06:57:21.930155 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.930130    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0aefa49a1bd7275a0d110a8cd8b72657-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-125.ec2.internal\" (UID: \"0aefa49a1bd7275a0d110a8cd8b72657\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-125.ec2.internal"
Oct 30 06:57:21.930268 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.930161    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0aefa49a1bd7275a0d110a8cd8b72657-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-125.ec2.internal\" (UID: \"0aefa49a1bd7275a0d110a8cd8b72657\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-125.ec2.internal"
Oct 30 06:57:21.930268 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:21.930186    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4ba2b683794afdaba9ca4ed884bfac10-config\") pod \"kube-apiserver-proxy-ip-10-0-128-125.ec2.internal\" (UID: \"4ba2b683794afdaba9ca4ed884bfac10\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-125.ec2.internal"
Oct 30 06:57:22.030719 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.030684    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0aefa49a1bd7275a0d110a8cd8b72657-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-125.ec2.internal\" (UID: \"0aefa49a1bd7275a0d110a8cd8b72657\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-125.ec2.internal"
Oct 30 06:57:22.030835 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.030725    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0aefa49a1bd7275a0d110a8cd8b72657-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-125.ec2.internal\" (UID: \"0aefa49a1bd7275a0d110a8cd8b72657\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-125.ec2.internal"
Oct 30 06:57:22.030835 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.030742    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4ba2b683794afdaba9ca4ed884bfac10-config\") pod \"kube-apiserver-proxy-ip-10-0-128-125.ec2.internal\" (UID: \"4ba2b683794afdaba9ca4ed884bfac10\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-125.ec2.internal"
Oct 30 06:57:22.030835 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.030798    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0aefa49a1bd7275a0d110a8cd8b72657-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-125.ec2.internal\" (UID: \"0aefa49a1bd7275a0d110a8cd8b72657\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-125.ec2.internal"
Oct 30 06:57:22.030835 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.030822    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0aefa49a1bd7275a0d110a8cd8b72657-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-125.ec2.internal\" (UID: \"0aefa49a1bd7275a0d110a8cd8b72657\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-125.ec2.internal"
Oct 30 06:57:22.030974 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.030860    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4ba2b683794afdaba9ca4ed884bfac10-config\") pod \"kube-apiserver-proxy-ip-10-0-128-125.ec2.internal\" (UID: \"4ba2b683794afdaba9ca4ed884bfac10\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-125.ec2.internal"
Oct 30 06:57:22.453684 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.453615    6224 apiserver.go:52] "Watching apiserver"
Oct 30 06:57:22.513326 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.513294    6224 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 06:57:22.513971 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.513947    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8kcn","openshift-image-registry/node-ca-l9mfn","openshift-monitoring/prometheus-operator-9cd89b74c-xh6wz","openshift-multus/network-metrics-daemon-r4d72","openshift-dns/node-resolver-5p67n","openshift-network-diagnostics/network-check-target-c6trh","openshift-ovn-kubernetes/ovnkube-node-r9jjx","kube-system/kube-apiserver-proxy-ip-10-0-128-125.ec2.internal","openshift-ingress-canary/ingress-canary-df2z5","openshift-multus/multus-9gqrp","openshift-console/console-f866fbc59-wnrl6","openshift-monitoring/telemeter-client-766978d4bc-rqpdj","kube-system/konnectivity-agent-wv8tb","openshift-dns/dns-default-dncbb","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-125.ec2.internal","openshift-monitoring/node-exporter-n7zk9","openshift-monitoring/prometheus-k8s-0","openshift-monitoring/metrics-server-769d4f7fd-5sbjs","openshift-insights/insights-runtime-extractor-ltdxc","openshift-multus/multus-additional-cni-plugins-4js5r","kube-system/global-pull-secret-syncer-qktbg","openshift-network-operator/iptables-alerter-tq7pf","openshift-cluster-node-tuning-operator/tuned-4kfdw","openshift-console/console-5c7875b857-wv9lf"]
Oct 30 06:57:22.532151 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.532125    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 06:57:22.532258 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.532169    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 06:57:22.532258 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.532225    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 06:57:22.532384 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.532358    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 06:57:22.532433 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.532395    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 06:57:22.532474 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.532430    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7szgh\""
Oct 30 06:57:22.532513 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.532473    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 06:57:22.532558 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.532533    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-f2h44\""
Oct 30 06:57:22.532671 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.532654    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2tccl\""
Oct 30 06:57:22.532715 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.532695    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 06:57:22.532756 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.532746    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 06:57:22.532756 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.532750    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 06:57:22.532844 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.532802    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 06:57:22.532947 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.532933    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-q4jsx\""
Oct 30 06:57:22.533019 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.533000    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9hhpl\""
Oct 30 06:57:22.533129 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.533057    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hkqzl\""
Oct 30 06:57:22.533129 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.533079    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 06:57:22.533234 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.533199    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 06:57:22.533285 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.533252    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 06:57:22.533285 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.533267    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 06:57:22.533351 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.533297    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 06:57:22.533351 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.533300    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 06:57:22.533351 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.533316    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 06:57:22.533449 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.533352    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-th56t\""
Oct 30 06:57:22.549964 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.549947    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 06:57:22.550902 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.550887    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 06:57:22.551626 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.551613    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 06:57:22.551697 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.551683    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 06:57:22.552026 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.552012    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 06:57:22.552520 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.552508    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 06:57:22.552571 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.552559    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 06:57:22.552732 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.552718    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 06:57:22.552832 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.552817    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 06:57:22.552973 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.552957    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 06:57:22.553041 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.553009    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 06:57:22.553041 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.553016    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 06:57:22.553041 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.553023    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 06:57:22.553195 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.553092    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 06:57:22.553195 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.553108    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 06:57:22.553195 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.553132    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 06:57:22.553195 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.553135    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8vngf\""
Oct 30 06:57:22.553195 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.553156    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 30 06:57:22.553195 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.553158    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 06:57:22.553452 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.553287    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 06:57:22.553452 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.553291    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 06:57:22.553452 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.553321    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 06:57:22.553452 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.553325    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pptc7\""
Oct 30 06:57:22.553452 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.553355    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 06:57:22.553452 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.553398    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 30 06:57:22.553452 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.553404    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 06:57:22.553452 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.553430    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 30 06:57:22.553452 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.553450    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6h5pm\""
Oct 30 06:57:22.553452 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.553453    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-mxksr\""
Oct 30 06:57:22.553980 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.553487    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-wlntw\""
Oct 30 06:57:22.553980 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.553569    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 30 06:57:22.553980 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.553688    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 06:57:22.553980 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.553716    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 30 06:57:22.553980 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.553716    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-x5k2w\""
Oct 30 06:57:22.553980 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.553720    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 06:57:22.553980 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.553737    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 06:57:22.553980 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.553747    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 06:57:22.553980 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.553720    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-dpx9k\""
Oct 30 06:57:22.553980 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.553775    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 30 06:57:22.553980 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.553798    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 30 06:57:22.553980 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.553885    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-44sjc\""
Oct 30 06:57:22.553980 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.553937    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 06:57:22.553980 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.553691    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 30 06:57:22.553980 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.553937    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 06:57:22.553980 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.553971    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 30 06:57:22.554412 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.554006    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 06:57:22.554412 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.554011    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-vjmtx\""
Oct 30 06:57:22.554412 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.554040    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-s8tns\""
Oct 30 06:57:22.558521 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.558503    6224 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-125.ec2.internal"
Oct 30 06:57:22.558669 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.558658    6224 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-125.ec2.internal"
Oct 30 06:57:22.567057 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.567035    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Oct 30 06:57:22.567182 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.567162    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 06:57:22.567434 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.567418    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 06:57:22.567658 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.567638    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 06:57:22.568118 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.568101    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 06:57:22.568252 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.568235    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 06:57:22.568321 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.568296    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 30 06:57:22.568649 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.568627    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Oct 30 06:57:22.568761 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.568691    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 06:57:22.568761 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.568697    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 06:57:22.568761 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.568691    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Oct 30 06:57:22.568761 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.568738    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Oct 30 06:57:22.568962 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.568856    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 06:57:22.568962 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.568864    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 30 06:57:22.568962 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.568915    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 06:57:22.569097 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.568980    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 06:57:22.569097 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.569012    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 06:57:22.569097 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.569015    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-6rnqp\""
Oct 30 06:57:22.569097 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.569027    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Oct 30 06:57:22.569097 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.569059    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 06:57:22.569646 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.569605    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-5br4m\""
Oct 30 06:57:22.569841 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.569822    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-djnbg\""
Oct 30 06:57:22.570202 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.570177    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-ac0ke4d4dbh3m\""
Oct 30 06:57:22.570317 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.570216    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 06:57:22.570604 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.570588    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 30 06:57:22.571029 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.571003    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-4wfx8\""
Oct 30 06:57:22.572019 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.571995    6224 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 06:57:22.572220 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.572200    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 06:57:22.581559 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.581538    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-e9mv09h28asd\""
Oct 30 06:57:22.581942 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.581921    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Oct 30 06:57:22.582008 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.581988    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 06:57:22.582171 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.582153    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 06:57:22.584665 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.584646    6224 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 06:57:22.584774 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.584655    6224 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 06:57:22.584855 ip-10-0-128-125 kubenswrapper[6224]: E1030 06:57:22.584790    6224 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-125.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-125.ec2.internal"
Oct 30 06:57:22.584855 ip-10-0-128-125 kubenswrapper[6224]: E1030 06:57:22.584818    6224 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-125.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-125.ec2.internal"
Oct 30 06:57:22.632902 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.632867    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:22.632902 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.632907    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/31519c22-050e-435c-aec3-7be0b68ff313-agent-certs\") pod \"konnectivity-agent-wv8tb\" (UID: \"31519c22-050e-435c-aec3-7be0b68ff313\") " pod="kube-system/konnectivity-agent-wv8tb"
Oct 30 06:57:22.633099 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.632926    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-node-log\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:57:22.633099 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.632953    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:57:22.633220 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.633087    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/851fb6ff-4a58-431b-834f-db3f87b67aa9-iptables-alerter-script\") pod \"iptables-alerter-tq7pf\" (UID: \"851fb6ff-4a58-431b-834f-db3f87b67aa9\") " pod="openshift-network-operator/iptables-alerter-tq7pf"
Oct 30 06:57:22.633220 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.633130    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7666c14-ff3f-44a5-9cce-21ff2dfda0e2-metrics-client-ca\") pod \"node-exporter-n7zk9\" (UID: \"f7666c14-ff3f-44a5-9cce-21ff2dfda0e2\") " pod="openshift-monitoring/node-exporter-n7zk9"
Oct 30 06:57:22.633220 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.633164    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dklkl\" (UniqueName: \"kubernetes.io/projected/2368101c-6719-4650-8393-ef048500a624-kube-api-access-dklkl\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:57:22.633220 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.633181    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:22.633220 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.633189    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dda66b1a-94e1-40dd-a77d-bfe37dd22b55-metrics-tls\") pod \"dns-default-dncbb\" (UID: \"dda66b1a-94e1-40dd-a77d-bfe37dd22b55\") " pod="openshift-dns/dns-default-dncbb"
Oct 30 06:57:22.633220 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.633216    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f8b75af-cc86-49c6-a66c-1798309f9806-metrics-certs\") pod \"network-metrics-daemon-r4d72\" (UID: \"6f8b75af-cc86-49c6-a66c-1798309f9806\") " pod="openshift-multus/network-metrics-daemon-r4d72"
Oct 30 06:57:22.633511 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.633231    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/31519c22-050e-435c-aec3-7be0b68ff313-agent-certs\") pod \"konnectivity-agent-wv8tb\" (UID: \"31519c22-050e-435c-aec3-7be0b68ff313\") " pod="kube-system/konnectivity-agent-wv8tb"
Oct 30 06:57:22.633511 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.633242    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f7666c14-ff3f-44a5-9cce-21ff2dfda0e2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n7zk9\" (UID: \"f7666c14-ff3f-44a5-9cce-21ff2dfda0e2\") " pod="openshift-monitoring/node-exporter-n7zk9"
Oct 30 06:57:22.633511 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.633272    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f7666c14-ff3f-44a5-9cce-21ff2dfda0e2-node-exporter-accelerators-collector-config\") pod \"node-exporter-n7zk9\" (UID: \"f7666c14-ff3f-44a5-9cce-21ff2dfda0e2\") " pod="openshift-monitoring/node-exporter-n7zk9"
Oct 30 06:57:22.633511 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.633298    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5d87089b-8b1b-43ea-be1f-72bfb56bf311-serviceca\") pod \"node-ca-l9mfn\" (UID: \"5d87089b-8b1b-43ea-be1f-72bfb56bf311\") " pod="openshift-image-registry/node-ca-l9mfn"
Oct 30 06:57:22.633511 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.633323    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c00279a4-113f-432d-8531-baa80c405f3a-audit-log\") pod \"metrics-server-769d4f7fd-5sbjs\" (UID: \"c00279a4-113f-432d-8531-baa80c405f3a\") " pod="openshift-monitoring/metrics-server-769d4f7fd-5sbjs"
Oct 30 06:57:22.633511 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.633348    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f09eaaa4-5ed2-4bee-931f-b425019f9c10-federate-client-tls\") pod \"telemeter-client-766978d4bc-rqpdj\" (UID: \"f09eaaa4-5ed2-4bee-931f-b425019f9c10\") " pod="openshift-monitoring/telemeter-client-766978d4bc-rqpdj"
Oct 30 06:57:22.633511 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.633364    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/851fb6ff-4a58-431b-834f-db3f87b67aa9-iptables-alerter-script\") pod \"iptables-alerter-tq7pf\" (UID: \"851fb6ff-4a58-431b-834f-db3f87b67aa9\") " pod="openshift-network-operator/iptables-alerter-tq7pf"
Oct 30 06:57:22.633511 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.633467    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f8b75af-cc86-49c6-a66c-1798309f9806-metrics-certs\") pod \"network-metrics-daemon-r4d72\" (UID: \"6f8b75af-cc86-49c6-a66c-1798309f9806\") " pod="openshift-multus/network-metrics-daemon-r4d72"
Oct 30 06:57:22.633511 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.633475    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c00279a4-113f-432d-8531-baa80c405f3a-audit-log\") pod \"metrics-server-769d4f7fd-5sbjs\" (UID: \"c00279a4-113f-432d-8531-baa80c405f3a\") " pod="openshift-monitoring/metrics-server-769d4f7fd-5sbjs"
Oct 30 06:57:22.633511 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.633403    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7666c14-ff3f-44a5-9cce-21ff2dfda0e2-metrics-client-ca\") pod \"node-exporter-n7zk9\" (UID: \"f7666c14-ff3f-44a5-9cce-21ff2dfda0e2\") " pod="openshift-monitoring/node-exporter-n7zk9"
Oct 30 06:57:22.633511 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.633377    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-system-cni-dir\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:57:22.634038 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.633536    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9d24d508-ec6f-4165-8394-f17b09134b34-service-ca\") pod \"console-5c7875b857-wv9lf\" (UID: \"9d24d508-ec6f-4165-8394-f17b09134b34\") " pod="openshift-console/console-5c7875b857-wv9lf"
Oct 30 06:57:22.634038 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.633565    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9d24d508-ec6f-4165-8394-f17b09134b34-oauth-serving-cert\") pod \"console-5c7875b857-wv9lf\" (UID: \"9d24d508-ec6f-4165-8394-f17b09134b34\") " pod="openshift-console/console-5c7875b857-wv9lf"
Oct 30 06:57:22.634038 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.633597    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:22.634038 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.633623    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:22.634038 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.633657    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a6968152-3c44-4205-a078-f857c413ad91-registration-dir\") pod \"aws-ebs-csi-driver-node-b8kcn\" (UID: \"a6968152-3c44-4205-a078-f857c413ad91\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8kcn"
Oct 30 06:57:22.634038 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.633682    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a6968152-3c44-4205-a078-f857c413ad91-device-dir\") pod \"aws-ebs-csi-driver-node-b8kcn\" (UID: \"a6968152-3c44-4205-a078-f857c413ad91\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8kcn"
Oct 30 06:57:22.634038 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.633696    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f7666c14-ff3f-44a5-9cce-21ff2dfda0e2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n7zk9\" (UID: \"f7666c14-ff3f-44a5-9cce-21ff2dfda0e2\") " pod="openshift-monitoring/node-exporter-n7zk9"
Oct 30 06:57:22.634038 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.633705    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a6968152-3c44-4205-a078-f857c413ad91-etc-selinux\") pod \"aws-ebs-csi-driver-node-b8kcn\" (UID: \"a6968152-3c44-4205-a078-f857c413ad91\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8kcn"
Oct 30 06:57:22.634038 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.633733    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9d24d508-ec6f-4165-8394-f17b09134b34-console-config\") pod \"console-5c7875b857-wv9lf\" (UID: \"9d24d508-ec6f-4165-8394-f17b09134b34\") " pod="openshift-console/console-5c7875b857-wv9lf"
Oct 30 06:57:22.634038 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.633758    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-var-lib-openvswitch\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:57:22.634038 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.633784    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbpk2\" (UniqueName: \"kubernetes.io/projected/9c5ccc21-fce2-4221-9c19-4a35fef32eff-kube-api-access-lbpk2\") pod \"insights-runtime-extractor-ltdxc\" (UID: \"9c5ccc21-fce2-4221-9c19-4a35fef32eff\") " pod="openshift-insights/insights-runtime-extractor-ltdxc"
Oct 30 06:57:22.634038 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.633785    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f7666c14-ff3f-44a5-9cce-21ff2dfda0e2-node-exporter-accelerators-collector-config\") pod \"node-exporter-n7zk9\" (UID: \"f7666c14-ff3f-44a5-9cce-21ff2dfda0e2\") " pod="openshift-monitoring/node-exporter-n7zk9"
Oct 30 06:57:22.634038 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.633841    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe292d5d-6d46-4c34-ab24-7991571cd680-etc-kubernetes\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:57:22.634038 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.633866    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fe292d5d-6d46-4c34-ab24-7991571cd680-etc-sysctl-d\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:57:22.634038 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.633891    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fe292d5d-6d46-4c34-ab24-7991571cd680-etc-sysctl-conf\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:57:22.634038 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.633918    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ca422300-6212-4a01-b1e2-5ba04af79ce6-cni-binary-copy\") pod \"multus-additional-cni-plugins-4js5r\" (UID: \"ca422300-6212-4a01-b1e2-5ba04af79ce6\") " pod="openshift-multus/multus-additional-cni-plugins-4js5r"
Oct 30 06:57:22.634038 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.633932    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:22.634038 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.633942    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d87089b-8b1b-43ea-be1f-72bfb56bf311-host\") pod \"node-ca-l9mfn\" (UID: \"5d87089b-8b1b-43ea-be1f-72bfb56bf311\") " pod="openshift-image-registry/node-ca-l9mfn"
Oct 30 06:57:22.634038 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.634043    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2368101c-6719-4650-8393-ef048500a624-cni-binary-copy\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:57:22.634649 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.634049    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:22.634649 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.634127    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-config\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:22.634649 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.634172    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:22.634649 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.634200    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-etc-openvswitch\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:57:22.634649 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.634220    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5d87089b-8b1b-43ea-be1f-72bfb56bf311-serviceca\") pod \"node-ca-l9mfn\" (UID: \"5d87089b-8b1b-43ea-be1f-72bfb56bf311\") " pod="openshift-image-registry/node-ca-l9mfn"
Oct 30 06:57:22.634649 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.634224    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe292d5d-6d46-4c34-ab24-7991571cd680-var-lib-kubelet\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:57:22.634649 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.634282    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ca422300-6212-4a01-b1e2-5ba04af79ce6-cni-binary-copy\") pod \"multus-additional-cni-plugins-4js5r\" (UID: \"ca422300-6212-4a01-b1e2-5ba04af79ce6\") " pod="openshift-multus/multus-additional-cni-plugins-4js5r"
Oct 30 06:57:22.634649 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.634301    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f7666c14-ff3f-44a5-9cce-21ff2dfda0e2-sys\") pod \"node-exporter-n7zk9\" (UID: \"f7666c14-ff3f-44a5-9cce-21ff2dfda0e2\") " pod="openshift-monitoring/node-exporter-n7zk9"
Oct 30 06:57:22.634649 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.634337    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:22.634649 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.634366    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/31519c22-050e-435c-aec3-7be0b68ff313-konnectivity-ca\") pod \"konnectivity-agent-wv8tb\" (UID: \"31519c22-050e-435c-aec3-7be0b68ff313\") " pod="kube-system/konnectivity-agent-wv8tb"
Oct 30 06:57:22.634649 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.634392    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2368101c-6719-4650-8393-ef048500a624-cni-binary-copy\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:57:22.634649 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.634393    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9c5ccc21-fce2-4221-9c19-4a35fef32eff-crio-socket\") pod \"insights-runtime-extractor-ltdxc\" (UID: \"9c5ccc21-fce2-4221-9c19-4a35fef32eff\") " pod="openshift-insights/insights-runtime-extractor-ltdxc"
Oct 30 06:57:22.634649 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.634436    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mnxb\" (UniqueName: \"kubernetes.io/projected/c00279a4-113f-432d-8531-baa80c405f3a-kube-api-access-7mnxb\") pod \"metrics-server-769d4f7fd-5sbjs\" (UID: \"c00279a4-113f-432d-8531-baa80c405f3a\") " pod="openshift-monitoring/metrics-server-769d4f7fd-5sbjs"
Oct 30 06:57:22.634649 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.634447    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9d24d508-ec6f-4165-8394-f17b09134b34-oauth-serving-cert\") pod \"console-5c7875b857-wv9lf\" (UID: \"9d24d508-ec6f-4165-8394-f17b09134b34\") " pod="openshift-console/console-5c7875b857-wv9lf"
Oct 30 06:57:22.635373 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.634791    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9d24d508-ec6f-4165-8394-f17b09134b34-service-ca\") pod \"console-5c7875b857-wv9lf\" (UID: \"9d24d508-ec6f-4165-8394-f17b09134b34\") " pod="openshift-console/console-5c7875b857-wv9lf"
Oct 30 06:57:22.635373 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.634802    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:22.635373 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.634896    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-host-var-lib-kubelet\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:57:22.635373 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.634930    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9d24d508-ec6f-4165-8394-f17b09134b34-console-serving-cert\") pod \"console-5c7875b857-wv9lf\" (UID: \"9d24d508-ec6f-4165-8394-f17b09134b34\") " pod="openshift-console/console-5c7875b857-wv9lf"
Oct 30 06:57:22.635373 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.634948    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:22.635373 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.634948    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/31519c22-050e-435c-aec3-7be0b68ff313-konnectivity-ca\") pod \"konnectivity-agent-wv8tb\" (UID: \"31519c22-050e-435c-aec3-7be0b68ff313\") " pod="kube-system/konnectivity-agent-wv8tb"
Oct 30 06:57:22.635373 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.635104    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:22.635373 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.635127    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:22.635373 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.635145    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b321b279-a102-48b9-bd9b-8ed938eed0f7-env-overrides\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:57:22.635373 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.635180    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnrqt\" (UniqueName: \"kubernetes.io/projected/3062f537-9b22-4e15-97aa-9a68703702f8-kube-api-access-wnrqt\") pod \"ingress-canary-df2z5\" (UID: \"3062f537-9b22-4e15-97aa-9a68703702f8\") " pod="openshift-ingress-canary/ingress-canary-df2z5"
Oct 30 06:57:22.635373 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.635197    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ca422300-6212-4a01-b1e2-5ba04af79ce6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4js5r\" (UID: \"ca422300-6212-4a01-b1e2-5ba04af79ce6\") " pod="openshift-multus/multus-additional-cni-plugins-4js5r"
Oct 30 06:57:22.635373 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.635215    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f09eaaa4-5ed2-4bee-931f-b425019f9c10-metrics-client-ca\") pod \"telemeter-client-766978d4bc-rqpdj\" (UID: \"f09eaaa4-5ed2-4bee-931f-b425019f9c10\") " pod="openshift-monitoring/telemeter-client-766978d4bc-rqpdj"
Oct 30 06:57:22.635373 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.635229    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fe292d5d-6d46-4c34-ab24-7991571cd680-etc-sysconfig\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:57:22.635373 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.635245    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/851fb6ff-4a58-431b-834f-db3f87b67aa9-host-slash\") pod \"iptables-alerter-tq7pf\" (UID: \"851fb6ff-4a58-431b-834f-db3f87b67aa9\") " pod="openshift-network-operator/iptables-alerter-tq7pf"
Oct 30 06:57:22.635373 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.635289    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dda66b1a-94e1-40dd-a77d-bfe37dd22b55-tmp-dir\") pod \"dns-default-dncbb\" (UID: \"dda66b1a-94e1-40dd-a77d-bfe37dd22b55\") " pod="openshift-dns/dns-default-dncbb"
Oct 30 06:57:22.635373 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.635305    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cf410657-29d1-408b-a956-e7e5ef548a33-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-xh6wz\" (UID: \"cf410657-29d1-408b-a956-e7e5ef548a33\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xh6wz"
Oct 30 06:57:22.635373 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.635328    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-host-cni-bin\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:57:22.635373 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.635343    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f7666c14-ff3f-44a5-9cce-21ff2dfda0e2-node-exporter-textfile\") pod \"node-exporter-n7zk9\" (UID: \"f7666c14-ff3f-44a5-9cce-21ff2dfda0e2\") " pod="openshift-monitoring/node-exporter-n7zk9"
Oct 30 06:57:22.635373 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.635359    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlfr5\" (UniqueName: \"kubernetes.io/projected/5d87089b-8b1b-43ea-be1f-72bfb56bf311-kube-api-access-tlfr5\") pod \"node-ca-l9mfn\" (UID: \"5d87089b-8b1b-43ea-be1f-72bfb56bf311\") " pod="openshift-image-registry/node-ca-l9mfn"
Oct 30 06:57:22.635373 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.635376    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:22.636299 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.635390    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a6968152-3c44-4205-a078-f857c413ad91-sys-fs\") pod \"aws-ebs-csi-driver-node-b8kcn\" (UID: \"a6968152-3c44-4205-a078-f857c413ad91\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8kcn"
Oct 30 06:57:22.636299 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.635399    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9d24d508-ec6f-4165-8394-f17b09134b34-console-config\") pod \"console-5c7875b857-wv9lf\" (UID: \"9d24d508-ec6f-4165-8394-f17b09134b34\") " pod="openshift-console/console-5c7875b857-wv9lf"
Oct 30 06:57:22.636299 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.635406    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b74sr\" (UniqueName: \"kubernetes.io/projected/675dc1eb-ff05-40c7-bf0b-fb3161d3cc8c-kube-api-access-b74sr\") pod \"node-resolver-5p67n\" (UID: \"675dc1eb-ff05-40c7-bf0b-fb3161d3cc8c\") " pod="openshift-dns/node-resolver-5p67n"
Oct 30 06:57:22.636299 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.635427    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f09eaaa4-5ed2-4bee-931f-b425019f9c10-telemeter-client-tls\") pod \"telemeter-client-766978d4bc-rqpdj\" (UID: \"f09eaaa4-5ed2-4bee-931f-b425019f9c10\") " pod="openshift-monitoring/telemeter-client-766978d4bc-rqpdj"
Oct 30 06:57:22.636299 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.635446    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmzmm\" (UniqueName: \"kubernetes.io/projected/6f8b75af-cc86-49c6-a66c-1798309f9806-kube-api-access-lmzmm\") pod \"network-metrics-daemon-r4d72\" (UID: \"6f8b75af-cc86-49c6-a66c-1798309f9806\") " pod="openshift-multus/network-metrics-daemon-r4d72"
Oct 30 06:57:22.636299 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.635525    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-host-run-netns\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:57:22.636299 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.635541    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-etc-kubernetes\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:57:22.636299 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.635550    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-config\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:22.636299 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.635559    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:22.636299 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.635575    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-service-ca\") pod \"console-f866fbc59-wnrl6\" (UID: \"bdc803a1-bbb2-407b-baa5-1aa3a54b7021\") " pod="openshift-console/console-f866fbc59-wnrl6"
Oct 30 06:57:22.636299 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.635590    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-tmp\") pod \"console-f866fbc59-wnrl6\" (UID: \"bdc803a1-bbb2-407b-baa5-1aa3a54b7021\") " pod="openshift-console/console-f866fbc59-wnrl6"
Oct 30 06:57:22.636299 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.635605    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fe292d5d-6d46-4c34-ab24-7991571cd680-sys\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:57:22.636299 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.635622    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9092e2e8-ad23-426c-b6f1-14e1a2b9221e-original-pull-secret\") pod \"global-pull-secret-syncer-qktbg\" (UID: \"9092e2e8-ad23-426c-b6f1-14e1a2b9221e\") " pod="kube-system/global-pull-secret-syncer-qktbg"
Oct 30 06:57:22.636299 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.635637    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-hostroot\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:57:22.636299 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.635652    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-console-oauth-config\") pod \"console-f866fbc59-wnrl6\" (UID: \"bdc803a1-bbb2-407b-baa5-1aa3a54b7021\") " pod="openshift-console/console-f866fbc59-wnrl6"
Oct 30 06:57:22.636299 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.635668    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-host-var-lib-cni-bin\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:57:22.636299 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.635683    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x67rl\" (UniqueName: \"kubernetes.io/projected/9d24d508-ec6f-4165-8394-f17b09134b34-kube-api-access-x67rl\") pod \"console-5c7875b857-wv9lf\" (UID: \"9d24d508-ec6f-4165-8394-f17b09134b34\") " pod="openshift-console/console-5c7875b857-wv9lf"
Oct 30 06:57:22.636299 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.635699    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:22.636299 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.635714    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-host-slash\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:57:22.636299 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.635730    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dda66b1a-94e1-40dd-a77d-bfe37dd22b55-config-volume\") pod \"dns-default-dncbb\" (UID: \"dda66b1a-94e1-40dd-a77d-bfe37dd22b55\") " pod="openshift-dns/dns-default-dncbb"
Oct 30 06:57:22.636299 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.635764    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c00279a4-113f-432d-8531-baa80c405f3a-secret-metrics-server-client-certs\") pod \"metrics-server-769d4f7fd-5sbjs\" (UID: \"c00279a4-113f-432d-8531-baa80c405f3a\") " pod="openshift-monitoring/metrics-server-769d4f7fd-5sbjs"
Oct 30 06:57:22.636299 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.635783    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f09eaaa4-5ed2-4bee-931f-b425019f9c10-telemeter-trusted-ca-bundle\") pod \"telemeter-client-766978d4bc-rqpdj\" (UID: \"f09eaaa4-5ed2-4bee-931f-b425019f9c10\") " pod="openshift-monitoring/telemeter-client-766978d4bc-rqpdj"
Oct 30 06:57:22.636299 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.635591    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dda66b1a-94e1-40dd-a77d-bfe37dd22b55-tmp-dir\") pod \"dns-default-dncbb\" (UID: \"dda66b1a-94e1-40dd-a77d-bfe37dd22b55\") " pod="openshift-dns/dns-default-dncbb"
Oct 30 06:57:22.636299 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.635878    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b321b279-a102-48b9-bd9b-8ed938eed0f7-env-overrides\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:57:22.636299 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.636054    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f09eaaa4-5ed2-4bee-931f-b425019f9c10-federate-client-tls\") pod \"telemeter-client-766978d4bc-rqpdj\" (UID: \"f09eaaa4-5ed2-4bee-931f-b425019f9c10\") " pod="openshift-monitoring/telemeter-client-766978d4bc-rqpdj"
Oct 30 06:57:22.636299 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.636067    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dda66b1a-94e1-40dd-a77d-bfe37dd22b55-config-volume\") pod \"dns-default-dncbb\" (UID: \"dda66b1a-94e1-40dd-a77d-bfe37dd22b55\") " pod="openshift-dns/dns-default-dncbb"
Oct 30 06:57:22.636299 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.636090    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:22.636299 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.636111    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c00279a4-113f-432d-8531-baa80c405f3a-secret-metrics-server-client-certs\") pod \"metrics-server-769d4f7fd-5sbjs\" (UID: \"c00279a4-113f-432d-8531-baa80c405f3a\") " pod="openshift-monitoring/metrics-server-769d4f7fd-5sbjs"
Oct 30 06:57:22.636299 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.636103    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b321b279-a102-48b9-bd9b-8ed938eed0f7-ovnkube-config\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:57:22.636299 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.636153    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fe292d5d-6d46-4c34-ab24-7991571cd680-lib-modules\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:57:22.636299 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.636184    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-web-config\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:22.636299 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.636210    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-host-kubelet\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:57:22.636299 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.636240    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-host-run-ovn-kubernetes\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:57:22.636299 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.636270    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d24d508-ec6f-4165-8394-f17b09134b34-tmp\") pod \"console-5c7875b857-wv9lf\" (UID: \"9d24d508-ec6f-4165-8394-f17b09134b34\") " pod="openshift-console/console-5c7875b857-wv9lf"
Oct 30 06:57:22.636299 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.636297    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-cnibin\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:57:22.637908 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.636328    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2368101c-6719-4650-8393-ef048500a624-multus-daemon-config\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:57:22.637908 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.636353    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-host-cni-netd\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:57:22.637908 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.636382    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-console-serving-cert\") pod \"console-f866fbc59-wnrl6\" (UID: \"bdc803a1-bbb2-407b-baa5-1aa3a54b7021\") " pod="openshift-console/console-f866fbc59-wnrl6"
Oct 30 06:57:22.637908 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.636411    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f09eaaa4-5ed2-4bee-931f-b425019f9c10-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-766978d4bc-rqpdj\" (UID: \"f09eaaa4-5ed2-4bee-931f-b425019f9c10\") " pod="openshift-monitoring/telemeter-client-766978d4bc-rqpdj"
Oct 30 06:57:22.637908 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.636424    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f09eaaa4-5ed2-4bee-931f-b425019f9c10-telemeter-trusted-ca-bundle\") pod \"telemeter-client-766978d4bc-rqpdj\" (UID: \"f09eaaa4-5ed2-4bee-931f-b425019f9c10\") " pod="openshift-monitoring/telemeter-client-766978d4bc-rqpdj"
Oct 30 06:57:22.637908 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.636445    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-oauth-serving-cert\") pod \"console-f866fbc59-wnrl6\" (UID: \"bdc803a1-bbb2-407b-baa5-1aa3a54b7021\") " pod="openshift-console/console-f866fbc59-wnrl6"
Oct 30 06:57:22.637908 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.636471    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fe292d5d-6d46-4c34-ab24-7991571cd680-etc-modprobe-d\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:57:22.637908 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.636467    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-console-oauth-config\") pod \"console-f866fbc59-wnrl6\" (UID: \"bdc803a1-bbb2-407b-baa5-1aa3a54b7021\") " pod="openshift-console/console-f866fbc59-wnrl6"
Oct 30 06:57:22.637908 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.636511    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ca422300-6212-4a01-b1e2-5ba04af79ce6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4js5r\" (UID: \"ca422300-6212-4a01-b1e2-5ba04af79ce6\") " pod="openshift-multus/multus-additional-cni-plugins-4js5r"
Oct 30 06:57:22.637908 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.636546    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjw82\" (UniqueName: \"kubernetes.io/projected/f09eaaa4-5ed2-4bee-931f-b425019f9c10-kube-api-access-cjw82\") pod \"telemeter-client-766978d4bc-rqpdj\" (UID: \"f09eaaa4-5ed2-4bee-931f-b425019f9c10\") " pod="openshift-monitoring/telemeter-client-766978d4bc-rqpdj"
Oct 30 06:57:22.637908 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.636576    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr6w5\" (UniqueName: \"kubernetes.io/projected/cf410657-29d1-408b-a956-e7e5ef548a33-kube-api-access-gr6w5\") pod \"prometheus-operator-9cd89b74c-xh6wz\" (UID: \"cf410657-29d1-408b-a956-e7e5ef548a33\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xh6wz"
Oct 30 06:57:22.637908 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.636638    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cf410657-29d1-408b-a956-e7e5ef548a33-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-xh6wz\" (UID: \"cf410657-29d1-408b-a956-e7e5ef548a33\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xh6wz"
Oct 30 06:57:22.637908 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.636297    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9d24d508-ec6f-4165-8394-f17b09134b34-console-serving-cert\") pod \"console-5c7875b857-wv9lf\" (UID: \"9d24d508-ec6f-4165-8394-f17b09134b34\") " pod="openshift-console/console-5c7875b857-wv9lf"
Oct 30 06:57:22.637908 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.636297    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b321b279-a102-48b9-bd9b-8ed938eed0f7-ovnkube-config\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:57:22.637908 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.636718    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f09eaaa4-5ed2-4bee-931f-b425019f9c10-telemeter-client-tls\") pod \"telemeter-client-766978d4bc-rqpdj\" (UID: \"f09eaaa4-5ed2-4bee-931f-b425019f9c10\") " pod="openshift-monitoring/telemeter-client-766978d4bc-rqpdj"
Oct 30 06:57:22.637908 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.636845    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f7666c14-ff3f-44a5-9cce-21ff2dfda0e2-node-exporter-textfile\") pod \"node-exporter-n7zk9\" (UID: \"f7666c14-ff3f-44a5-9cce-21ff2dfda0e2\") " pod="openshift-monitoring/node-exporter-n7zk9"
Oct 30 06:57:22.637908 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.636883    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:22.637908 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.636911    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kds7\" (UniqueName: \"kubernetes.io/projected/b321b279-a102-48b9-bd9b-8ed938eed0f7-kube-api-access-8kds7\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:57:22.637908 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.636941    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fe292d5d-6d46-4c34-ab24-7991571cd680-run\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:57:22.637908 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.636955    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dda66b1a-94e1-40dd-a77d-bfe37dd22b55-metrics-tls\") pod \"dns-default-dncbb\" (UID: \"dda66b1a-94e1-40dd-a77d-bfe37dd22b55\") " pod="openshift-dns/dns-default-dncbb"
Oct 30 06:57:22.637908 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.636970    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fe292d5d-6d46-4c34-ab24-7991571cd680-etc-tuned\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:57:22.637908 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.637007    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9d24d508-ec6f-4165-8394-f17b09134b34-console-oauth-config\") pod \"console-5c7875b857-wv9lf\" (UID: \"9d24d508-ec6f-4165-8394-f17b09134b34\") " pod="openshift-console/console-5c7875b857-wv9lf"
Oct 30 06:57:22.637908 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.637022    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9092e2e8-ad23-426c-b6f1-14e1a2b9221e-original-pull-secret\") pod \"global-pull-secret-syncer-qktbg\" (UID: \"9092e2e8-ad23-426c-b6f1-14e1a2b9221e\") " pod="kube-system/global-pull-secret-syncer-qktbg"
Oct 30 06:57:22.637908 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.637034    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-host-run-k8s-cni-cncf-io\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:57:22.637908 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.637062    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf410657-29d1-408b-a956-e7e5ef548a33-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-xh6wz\" (UID: \"cf410657-29d1-408b-a956-e7e5ef548a33\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xh6wz"
Oct 30 06:57:22.637908 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.637083    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-web-config\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:22.637908 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.637089    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-config-out\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:22.637908 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.637116    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a6968152-3c44-4205-a078-f857c413ad91-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b8kcn\" (UID: \"a6968152-3c44-4205-a078-f857c413ad91\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8kcn"
Oct 30 06:57:22.637908 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.637153    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-run-systemd\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:57:22.637908 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.637183    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-console-config\") pod \"console-f866fbc59-wnrl6\" (UID: \"bdc803a1-bbb2-407b-baa5-1aa3a54b7021\") " pod="openshift-console/console-f866fbc59-wnrl6"
Oct 30 06:57:22.637908 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.637208    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f09eaaa4-5ed2-4bee-931f-b425019f9c10-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-766978d4bc-rqpdj\" (UID: \"f09eaaa4-5ed2-4bee-931f-b425019f9c10\") " pod="openshift-monitoring/telemeter-client-766978d4bc-rqpdj"
Oct 30 06:57:22.637908 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.637211    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe292d5d-6d46-4c34-ab24-7991571cd680-tmp\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:57:22.637908 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.637249    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a6968152-3c44-4205-a078-f857c413ad91-socket-dir\") pod \"aws-ebs-csi-driver-node-b8kcn\" (UID: \"a6968152-3c44-4205-a078-f857c413ad91\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8kcn"
Oct 30 06:57:22.637908 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.637275    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe292d5d-6d46-4c34-ab24-7991571cd680-tmp\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:57:22.637908 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.637282    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-log-socket\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:57:22.637908 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.637311    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b321b279-a102-48b9-bd9b-8ed938eed0f7-ovnkube-script-lib\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:57:22.637908 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.637335    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d24d508-ec6f-4165-8394-f17b09134b34-tmp\") pod \"console-5c7875b857-wv9lf\" (UID: \"9d24d508-ec6f-4165-8394-f17b09134b34\") " pod="openshift-console/console-5c7875b857-wv9lf"
Oct 30 06:57:22.637908 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.637341    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3062f537-9b22-4e15-97aa-9a68703702f8-cert\") pod \"ingress-canary-df2z5\" (UID: \"3062f537-9b22-4e15-97aa-9a68703702f8\") " pod="openshift-ingress-canary/ingress-canary-df2z5"
Oct 30 06:57:22.637908 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.637525    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2368101c-6719-4650-8393-ef048500a624-multus-daemon-config\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:57:22.637908 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.637562    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3062f537-9b22-4e15-97aa-9a68703702f8-cert\") pod \"ingress-canary-df2z5\" (UID: \"3062f537-9b22-4e15-97aa-9a68703702f8\") " pod="openshift-ingress-canary/ingress-canary-df2z5"
Oct 30 06:57:22.639634 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.638279    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c00279a4-113f-432d-8531-baa80c405f3a-secret-metrics-server-tls\") pod \"metrics-server-769d4f7fd-5sbjs\" (UID: \"c00279a4-113f-432d-8531-baa80c405f3a\") " pod="openshift-monitoring/metrics-server-769d4f7fd-5sbjs"
Oct 30 06:57:22.639634 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.638358    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:22.639634 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.638554    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9d24d508-ec6f-4165-8394-f17b09134b34-console-oauth-config\") pod \"console-5c7875b857-wv9lf\" (UID: \"9d24d508-ec6f-4165-8394-f17b09134b34\") " pod="openshift-console/console-5c7875b857-wv9lf"
Oct 30 06:57:22.639634 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.638700    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:22.639634 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.638894    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-service-ca\") pod \"console-f866fbc59-wnrl6\" (UID: \"bdc803a1-bbb2-407b-baa5-1aa3a54b7021\") " pod="openshift-console/console-f866fbc59-wnrl6"
Oct 30 06:57:22.639634 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.639096    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:22.639634 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.639244    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-oauth-serving-cert\") pod \"console-f866fbc59-wnrl6\" (UID: \"bdc803a1-bbb2-407b-baa5-1aa3a54b7021\") " pod="openshift-console/console-f866fbc59-wnrl6"
Oct 30 06:57:22.639634 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.639274    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:22.639634 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.639397    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ca422300-6212-4a01-b1e2-5ba04af79ce6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4js5r\" (UID: \"ca422300-6212-4a01-b1e2-5ba04af79ce6\") " pod="openshift-multus/multus-additional-cni-plugins-4js5r"
Oct 30 06:57:22.639634 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.639532    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fe292d5d-6d46-4c34-ab24-7991571cd680-etc-tuned\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:57:22.639634 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.639572    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-console-serving-cert\") pod \"console-f866fbc59-wnrl6\" (UID: \"bdc803a1-bbb2-407b-baa5-1aa3a54b7021\") " pod="openshift-console/console-f866fbc59-wnrl6"
Oct 30 06:57:22.639634 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.639589    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-config-out\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:22.640158 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.639776    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf410657-29d1-408b-a956-e7e5ef548a33-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-xh6wz\" (UID: \"cf410657-29d1-408b-a956-e7e5ef548a33\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xh6wz"
Oct 30 06:57:22.640158 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.639797    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-tmp\") pod \"console-f866fbc59-wnrl6\" (UID: \"bdc803a1-bbb2-407b-baa5-1aa3a54b7021\") " pod="openshift-console/console-f866fbc59-wnrl6"
Oct 30 06:57:22.640158 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.640000    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b321b279-a102-48b9-bd9b-8ed938eed0f7-ovnkube-script-lib\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:57:22.640158 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.640007    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f09eaaa4-5ed2-4bee-931f-b425019f9c10-metrics-client-ca\") pod \"telemeter-client-766978d4bc-rqpdj\" (UID: \"f09eaaa4-5ed2-4bee-931f-b425019f9c10\") " pod="openshift-monitoring/telemeter-client-766978d4bc-rqpdj"
Oct 30 06:57:22.640340 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.640166    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-console-config\") pod \"console-f866fbc59-wnrl6\" (UID: \"bdc803a1-bbb2-407b-baa5-1aa3a54b7021\") " pod="openshift-console/console-f866fbc59-wnrl6"
Oct 30 06:57:22.641721 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.640581    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:22.641721 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.637561    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c00279a4-113f-432d-8531-baa80c405f3a-secret-metrics-server-tls\") pod \"metrics-server-769d4f7fd-5sbjs\" (UID: \"c00279a4-113f-432d-8531-baa80c405f3a\") " pod="openshift-monitoring/metrics-server-769d4f7fd-5sbjs"
Oct 30 06:57:22.641721 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.640869    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9bkg\" (UniqueName: \"kubernetes.io/projected/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-kube-api-access-n9bkg\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:22.641721 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.640897    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f09eaaa4-5ed2-4bee-931f-b425019f9c10-secret-telemeter-client\") pod \"telemeter-client-766978d4bc-rqpdj\" (UID: \"f09eaaa4-5ed2-4bee-931f-b425019f9c10\") " pod="openshift-monitoring/telemeter-client-766978d4bc-rqpdj"
Oct 30 06:57:22.641721 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.640923    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9092e2e8-ad23-426c-b6f1-14e1a2b9221e-dbus\") pod \"global-pull-secret-syncer-qktbg\" (UID: \"9092e2e8-ad23-426c-b6f1-14e1a2b9221e\") " pod="kube-system/global-pull-secret-syncer-qktbg"
Oct 30 06:57:22.641721 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.640946    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:22.641721 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.640970    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cwrb\" (UniqueName: \"kubernetes.io/projected/6d493f94-3aef-445e-9b60-32a10c184eac-kube-api-access-2cwrb\") pod \"network-check-target-c6trh\" (UID: \"6d493f94-3aef-445e-9b60-32a10c184eac\") " pod="openshift-network-diagnostics/network-check-target-c6trh"
Oct 30 06:57:22.641721 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.640993    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f7666c14-ff3f-44a5-9cce-21ff2dfda0e2-root\") pod \"node-exporter-n7zk9\" (UID: \"f7666c14-ff3f-44a5-9cce-21ff2dfda0e2\") " pod="openshift-monitoring/node-exporter-n7zk9"
Oct 30 06:57:22.641721 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.641238    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:22.641721 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.641452    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f09eaaa4-5ed2-4bee-931f-b425019f9c10-secret-telemeter-client\") pod \"telemeter-client-766978d4bc-rqpdj\" (UID: \"f09eaaa4-5ed2-4bee-931f-b425019f9c10\") " pod="openshift-monitoring/telemeter-client-766978d4bc-rqpdj"
Oct 30 06:57:22.641721 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.641035    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ca422300-6212-4a01-b1e2-5ba04af79ce6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4js5r\" (UID: \"ca422300-6212-4a01-b1e2-5ba04af79ce6\") " pod="openshift-multus/multus-additional-cni-plugins-4js5r"
Oct 30 06:57:22.641721 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.641504    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-systemd-units\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:57:22.641721 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.641523    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f7666c14-ff3f-44a5-9cce-21ff2dfda0e2-node-exporter-tls\") pod \"node-exporter-n7zk9\" (UID: \"f7666c14-ff3f-44a5-9cce-21ff2dfda0e2\") " pod="openshift-monitoring/node-exporter-n7zk9"
Oct 30 06:57:22.641721 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.641540    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-multus-conf-dir\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:57:22.641721 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.641556    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:22.641721 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.641580    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkb8j\" (UniqueName: \"kubernetes.io/projected/a6968152-3c44-4205-a078-f857c413ad91-kube-api-access-rkb8j\") pod \"aws-ebs-csi-driver-node-b8kcn\" (UID: \"a6968152-3c44-4205-a078-f857c413ad91\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8kcn"
Oct 30 06:57:22.641721 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.641605    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/675dc1eb-ff05-40c7-bf0b-fb3161d3cc8c-hosts-file\") pod \"node-resolver-5p67n\" (UID: \"675dc1eb-ff05-40c7-bf0b-fb3161d3cc8c\") " pod="openshift-dns/node-resolver-5p67n"
Oct 30 06:57:22.641721 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.641628    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fe292d5d-6d46-4c34-ab24-7991571cd680-etc-systemd\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:57:22.641721 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.641653    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvdd8\" (UniqueName: \"kubernetes.io/projected/851fb6ff-4a58-431b-834f-db3f87b67aa9-kube-api-access-hvdd8\") pod \"iptables-alerter-tq7pf\" (UID: \"851fb6ff-4a58-431b-834f-db3f87b67aa9\") " pod="openshift-network-operator/iptables-alerter-tq7pf"
Oct 30 06:57:22.641721 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.641678    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-host-run-netns\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:57:22.641721 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.641703    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9c5ccc21-fce2-4221-9c19-4a35fef32eff-data-volume\") pod \"insights-runtime-extractor-ltdxc\" (UID: \"9c5ccc21-fce2-4221-9c19-4a35fef32eff\") " pod="openshift-insights/insights-runtime-extractor-ltdxc"
Oct 30 06:57:22.641721 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.641727    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-host-var-lib-cni-multus\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:57:22.642543 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.641750    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-run-openvswitch\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:57:22.642543 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.641773    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-run-ovn\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:57:22.642543 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.641802    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c5ccc21-fce2-4221-9c19-4a35fef32eff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ltdxc\" (UID: \"9c5ccc21-fce2-4221-9c19-4a35fef32eff\") " pod="openshift-insights/insights-runtime-extractor-ltdxc"
Oct 30 06:57:22.642543 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.641850    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ca422300-6212-4a01-b1e2-5ba04af79ce6-system-cni-dir\") pod \"multus-additional-cni-plugins-4js5r\" (UID: \"ca422300-6212-4a01-b1e2-5ba04af79ce6\") " pod="openshift-multus/multus-additional-cni-plugins-4js5r"
Oct 30 06:57:22.642543 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.641877    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c00279a4-113f-432d-8531-baa80c405f3a-metrics-server-audit-profiles\") pod \"metrics-server-769d4f7fd-5sbjs\" (UID: \"c00279a4-113f-432d-8531-baa80c405f3a\") " pod="openshift-monitoring/metrics-server-769d4f7fd-5sbjs"
Oct 30 06:57:22.642543 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.641902    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c00279a4-113f-432d-8531-baa80c405f3a-client-ca-bundle\") pod \"metrics-server-769d4f7fd-5sbjs\" (UID: \"c00279a4-113f-432d-8531-baa80c405f3a\") " pod="openshift-monitoring/metrics-server-769d4f7fd-5sbjs"
Oct 30 06:57:22.642543 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.641930    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b321b279-a102-48b9-bd9b-8ed938eed0f7-ovn-node-metrics-cert\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:57:22.642543 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.641955    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9c5ccc21-fce2-4221-9c19-4a35fef32eff-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ltdxc\" (UID: \"9c5ccc21-fce2-4221-9c19-4a35fef32eff\") " pod="openshift-insights/insights-runtime-extractor-ltdxc"
Oct 30 06:57:22.642543 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.641982    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ca422300-6212-4a01-b1e2-5ba04af79ce6-cnibin\") pod \"multus-additional-cni-plugins-4js5r\" (UID: \"ca422300-6212-4a01-b1e2-5ba04af79ce6\") " pod="openshift-multus/multus-additional-cni-plugins-4js5r"
Oct 30 06:57:22.642543 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.642008    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c00279a4-113f-432d-8531-baa80c405f3a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-769d4f7fd-5sbjs\" (UID: \"c00279a4-113f-432d-8531-baa80c405f3a\") " pod="openshift-monitoring/metrics-server-769d4f7fd-5sbjs"
Oct 30 06:57:22.642543 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.642038    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f09eaaa4-5ed2-4bee-931f-b425019f9c10-serving-certs-ca-bundle\") pod \"telemeter-client-766978d4bc-rqpdj\" (UID: \"f09eaaa4-5ed2-4bee-931f-b425019f9c10\") " pod="openshift-monitoring/telemeter-client-766978d4bc-rqpdj"
Oct 30 06:57:22.642543 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.642063    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-host-run-multus-certs\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:57:22.642543 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.642090    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cf410657-29d1-408b-a956-e7e5ef548a33-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-xh6wz\" (UID: \"cf410657-29d1-408b-a956-e7e5ef548a33\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xh6wz"
Oct 30 06:57:22.642543 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.642099    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f7666c14-ff3f-44a5-9cce-21ff2dfda0e2-node-exporter-tls\") pod \"node-exporter-n7zk9\" (UID: \"f7666c14-ff3f-44a5-9cce-21ff2dfda0e2\") " pod="openshift-monitoring/node-exporter-n7zk9"
Oct 30 06:57:22.642543 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.642113    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fe292d5d-6d46-4c34-ab24-7991571cd680-host\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:57:22.642543 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.642305    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px9vv\" (UniqueName: \"kubernetes.io/projected/dda66b1a-94e1-40dd-a77d-bfe37dd22b55-kube-api-access-px9vv\") pod \"dns-default-dncbb\" (UID: \"dda66b1a-94e1-40dd-a77d-bfe37dd22b55\") " pod="openshift-dns/dns-default-dncbb"
Oct 30 06:57:22.642543 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.642314    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:22.644343 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.642608    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c00279a4-113f-432d-8531-baa80c405f3a-metrics-server-audit-profiles\") pod \"metrics-server-769d4f7fd-5sbjs\" (UID: \"c00279a4-113f-432d-8531-baa80c405f3a\") " pod="openshift-monitoring/metrics-server-769d4f7fd-5sbjs"
Oct 30 06:57:22.644343 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.642690    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c00279a4-113f-432d-8531-baa80c405f3a-client-ca-bundle\") pod \"metrics-server-769d4f7fd-5sbjs\" (UID: \"c00279a4-113f-432d-8531-baa80c405f3a\") " pod="openshift-monitoring/metrics-server-769d4f7fd-5sbjs"
Oct 30 06:57:22.644343 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.642771    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9c5ccc21-fce2-4221-9c19-4a35fef32eff-data-volume\") pod \"insights-runtime-extractor-ltdxc\" (UID: \"9c5ccc21-fce2-4221-9c19-4a35fef32eff\") " pod="openshift-insights/insights-runtime-extractor-ltdxc"
Oct 30 06:57:22.644343 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.643012    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b321b279-a102-48b9-bd9b-8ed938eed0f7-ovn-node-metrics-cert\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:57:22.644343 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.643176    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9c5ccc21-fce2-4221-9c19-4a35fef32eff-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ltdxc\" (UID: \"9c5ccc21-fce2-4221-9c19-4a35fef32eff\") " pod="openshift-insights/insights-runtime-extractor-ltdxc"
Oct 30 06:57:22.644343 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.643331    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c00279a4-113f-432d-8531-baa80c405f3a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-769d4f7fd-5sbjs\" (UID: \"c00279a4-113f-432d-8531-baa80c405f3a\") " pod="openshift-monitoring/metrics-server-769d4f7fd-5sbjs"
Oct 30 06:57:22.644343 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.643369    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srs22\" (UniqueName: \"kubernetes.io/projected/f7666c14-ff3f-44a5-9cce-21ff2dfda0e2-kube-api-access-srs22\") pod \"node-exporter-n7zk9\" (UID: \"f7666c14-ff3f-44a5-9cce-21ff2dfda0e2\") " pod="openshift-monitoring/node-exporter-n7zk9"
Oct 30 06:57:22.644343 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.643399    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmzfj\" (UniqueName: \"kubernetes.io/projected/ca422300-6212-4a01-b1e2-5ba04af79ce6-kube-api-access-lmzfj\") pod \"multus-additional-cni-plugins-4js5r\" (UID: \"ca422300-6212-4a01-b1e2-5ba04af79ce6\") " pod="openshift-multus/multus-additional-cni-plugins-4js5r"
Oct 30 06:57:22.644343 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.643426    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9092e2e8-ad23-426c-b6f1-14e1a2b9221e-kubelet-config\") pod \"global-pull-secret-syncer-qktbg\" (UID: \"9092e2e8-ad23-426c-b6f1-14e1a2b9221e\") " pod="kube-system/global-pull-secret-syncer-qktbg"
Oct 30 06:57:22.644343 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.643453    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-multus-socket-dir-parent\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:57:22.644343 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.643480    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:22.644343 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.643507    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:22.644343 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.643777    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/675dc1eb-ff05-40c7-bf0b-fb3161d3cc8c-tmp-dir\") pod \"node-resolver-5p67n\" (UID: \"675dc1eb-ff05-40c7-bf0b-fb3161d3cc8c\") " pod="openshift-dns/node-resolver-5p67n"
Oct 30 06:57:22.644343 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.643823    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-trusted-ca-bundle\") pod \"console-f866fbc59-wnrl6\" (UID: \"bdc803a1-bbb2-407b-baa5-1aa3a54b7021\") " pod="openshift-console/console-f866fbc59-wnrl6"
Oct 30 06:57:22.644343 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.643844    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cf410657-29d1-408b-a956-e7e5ef548a33-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-xh6wz\" (UID: \"cf410657-29d1-408b-a956-e7e5ef548a33\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xh6wz"
Oct 30 06:57:22.644343 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.643855    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7cdq\" (UniqueName: \"kubernetes.io/projected/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-kube-api-access-q7cdq\") pod \"console-f866fbc59-wnrl6\" (UID: \"bdc803a1-bbb2-407b-baa5-1aa3a54b7021\") " pod="openshift-console/console-f866fbc59-wnrl6"
Oct 30 06:57:22.644343 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.643883    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmkz2\" (UniqueName: \"kubernetes.io/projected/fe292d5d-6d46-4c34-ab24-7991571cd680-kube-api-access-bmkz2\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:57:22.644343 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.643911    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f7666c14-ff3f-44a5-9cce-21ff2dfda0e2-node-exporter-wtmp\") pod \"node-exporter-n7zk9\" (UID: \"f7666c14-ff3f-44a5-9cce-21ff2dfda0e2\") " pod="openshift-monitoring/node-exporter-n7zk9"
Oct 30 06:57:22.644343 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.644004    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c5ccc21-fce2-4221-9c19-4a35fef32eff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ltdxc\" (UID: \"9c5ccc21-fce2-4221-9c19-4a35fef32eff\") " pod="openshift-insights/insights-runtime-extractor-ltdxc"
Oct 30 06:57:22.644343 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.644023    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f09eaaa4-5ed2-4bee-931f-b425019f9c10-serving-certs-ca-bundle\") pod \"telemeter-client-766978d4bc-rqpdj\" (UID: \"f09eaaa4-5ed2-4bee-931f-b425019f9c10\") " pod="openshift-monitoring/telemeter-client-766978d4bc-rqpdj"
Oct 30 06:57:22.644343 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.644055    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-multus-cni-dir\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:57:22.644343 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.644072    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:22.644343 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.644079    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:22.644343 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.644242    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:22.644343 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.644323    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:22.645169 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.644507    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/675dc1eb-ff05-40c7-bf0b-fb3161d3cc8c-tmp-dir\") pod \"node-resolver-5p67n\" (UID: \"675dc1eb-ff05-40c7-bf0b-fb3161d3cc8c\") " pod="openshift-dns/node-resolver-5p67n"
Oct 30 06:57:22.645169 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.644540    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-trusted-ca-bundle\") pod \"console-f866fbc59-wnrl6\" (UID: \"bdc803a1-bbb2-407b-baa5-1aa3a54b7021\") " pod="openshift-console/console-f866fbc59-wnrl6"
Oct 30 06:57:22.645169 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.644669    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ca422300-6212-4a01-b1e2-5ba04af79ce6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4js5r\" (UID: \"ca422300-6212-4a01-b1e2-5ba04af79ce6\") " pod="openshift-multus/multus-additional-cni-plugins-4js5r"
Oct 30 06:57:22.645169 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.644817    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c7875b857-wv9lf"]
Oct 30 06:57:22.645169 ip-10-0-128-125 kubenswrapper[6224]: E1030 06:57:22.645071    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-x67rl], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-console/console-5c7875b857-wv9lf" podUID="9d24d508-ec6f-4165-8394-f17b09134b34"
Oct 30 06:57:22.663127 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.663100    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dklkl\" (UniqueName: \"kubernetes.io/projected/2368101c-6719-4650-8393-ef048500a624-kube-api-access-dklkl\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:57:22.667337 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.667285    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmzmm\" (UniqueName: \"kubernetes.io/projected/6f8b75af-cc86-49c6-a66c-1798309f9806-kube-api-access-lmzmm\") pod \"network-metrics-daemon-r4d72\" (UID: \"6f8b75af-cc86-49c6-a66c-1798309f9806\") " pod="openshift-multus/network-metrics-daemon-r4d72"
Oct 30 06:57:22.667703 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.667676    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr6w5\" (UniqueName: \"kubernetes.io/projected/cf410657-29d1-408b-a956-e7e5ef548a33-kube-api-access-gr6w5\") pod \"prometheus-operator-9cd89b74c-xh6wz\" (UID: \"cf410657-29d1-408b-a956-e7e5ef548a33\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xh6wz"
Oct 30 06:57:22.668115 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.668000    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mnxb\" (UniqueName: \"kubernetes.io/projected/c00279a4-113f-432d-8531-baa80c405f3a-kube-api-access-7mnxb\") pod \"metrics-server-769d4f7fd-5sbjs\" (UID: \"c00279a4-113f-432d-8531-baa80c405f3a\") " pod="openshift-monitoring/metrics-server-769d4f7fd-5sbjs"
Oct 30 06:57:22.671198 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.671176    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlfr5\" (UniqueName: \"kubernetes.io/projected/5d87089b-8b1b-43ea-be1f-72bfb56bf311-kube-api-access-tlfr5\") pod \"node-ca-l9mfn\" (UID: \"5d87089b-8b1b-43ea-be1f-72bfb56bf311\") " pod="openshift-image-registry/node-ca-l9mfn"
Oct 30 06:57:22.676719 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.674640    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b74sr\" (UniqueName: \"kubernetes.io/projected/675dc1eb-ff05-40c7-bf0b-fb3161d3cc8c-kube-api-access-b74sr\") pod \"node-resolver-5p67n\" (UID: \"675dc1eb-ff05-40c7-bf0b-fb3161d3cc8c\") " pod="openshift-dns/node-resolver-5p67n"
Oct 30 06:57:22.676719 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.674939    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnrqt\" (UniqueName: \"kubernetes.io/projected/3062f537-9b22-4e15-97aa-9a68703702f8-kube-api-access-wnrqt\") pod \"ingress-canary-df2z5\" (UID: \"3062f537-9b22-4e15-97aa-9a68703702f8\") " pod="openshift-ingress-canary/ingress-canary-df2z5"
Oct 30 06:57:22.676719 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.675066    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kds7\" (UniqueName: \"kubernetes.io/projected/b321b279-a102-48b9-bd9b-8ed938eed0f7-kube-api-access-8kds7\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:57:22.676719 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.675118    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x67rl\" (UniqueName: \"kubernetes.io/projected/9d24d508-ec6f-4165-8394-f17b09134b34-kube-api-access-x67rl\") pod \"console-5c7875b857-wv9lf\" (UID: \"9d24d508-ec6f-4165-8394-f17b09134b34\") " pod="openshift-console/console-5c7875b857-wv9lf"
Oct 30 06:57:22.676719 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.675401    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbpk2\" (UniqueName: \"kubernetes.io/projected/9c5ccc21-fce2-4221-9c19-4a35fef32eff-kube-api-access-lbpk2\") pod \"insights-runtime-extractor-ltdxc\" (UID: \"9c5ccc21-fce2-4221-9c19-4a35fef32eff\") " pod="openshift-insights/insights-runtime-extractor-ltdxc"
Oct 30 06:57:22.676719 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.675511    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjw82\" (UniqueName: \"kubernetes.io/projected/f09eaaa4-5ed2-4bee-931f-b425019f9c10-kube-api-access-cjw82\") pod \"telemeter-client-766978d4bc-rqpdj\" (UID: \"f09eaaa4-5ed2-4bee-931f-b425019f9c10\") " pod="openshift-monitoring/telemeter-client-766978d4bc-rqpdj"
Oct 30 06:57:22.676719 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.675871    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmkz2\" (UniqueName: \"kubernetes.io/projected/fe292d5d-6d46-4c34-ab24-7991571cd680-kube-api-access-bmkz2\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:57:22.676719 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.675984    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkb8j\" (UniqueName: \"kubernetes.io/projected/a6968152-3c44-4205-a078-f857c413ad91-kube-api-access-rkb8j\") pod \"aws-ebs-csi-driver-node-b8kcn\" (UID: \"a6968152-3c44-4205-a078-f857c413ad91\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8kcn"
Oct 30 06:57:22.676719 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.676061    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmzfj\" (UniqueName: \"kubernetes.io/projected/ca422300-6212-4a01-b1e2-5ba04af79ce6-kube-api-access-lmzfj\") pod \"multus-additional-cni-plugins-4js5r\" (UID: \"ca422300-6212-4a01-b1e2-5ba04af79ce6\") " pod="openshift-multus/multus-additional-cni-plugins-4js5r"
Oct 30 06:57:22.676719 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.676248    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-px9vv\" (UniqueName: \"kubernetes.io/projected/dda66b1a-94e1-40dd-a77d-bfe37dd22b55-kube-api-access-px9vv\") pod \"dns-default-dncbb\" (UID: \"dda66b1a-94e1-40dd-a77d-bfe37dd22b55\") " pod="openshift-dns/dns-default-dncbb"
Oct 30 06:57:22.676719 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.676686    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cwrb\" (UniqueName: \"kubernetes.io/projected/6d493f94-3aef-445e-9b60-32a10c184eac-kube-api-access-2cwrb\") pod \"network-check-target-c6trh\" (UID: \"6d493f94-3aef-445e-9b60-32a10c184eac\") " pod="openshift-network-diagnostics/network-check-target-c6trh"
Oct 30 06:57:22.677186 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.676741    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7cdq\" (UniqueName: \"kubernetes.io/projected/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-kube-api-access-q7cdq\") pod \"console-f866fbc59-wnrl6\" (UID: \"bdc803a1-bbb2-407b-baa5-1aa3a54b7021\") " pod="openshift-console/console-f866fbc59-wnrl6"
Oct 30 06:57:22.677186 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.676902    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9bkg\" (UniqueName: \"kubernetes.io/projected/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-kube-api-access-n9bkg\") pod \"prometheus-k8s-0\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:22.677246 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.677191    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srs22\" (UniqueName: \"kubernetes.io/projected/f7666c14-ff3f-44a5-9cce-21ff2dfda0e2-kube-api-access-srs22\") pod \"node-exporter-n7zk9\" (UID: \"f7666c14-ff3f-44a5-9cce-21ff2dfda0e2\") " pod="openshift-monitoring/node-exporter-n7zk9"
Oct 30 06:57:22.682218 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.682195    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvdd8\" (UniqueName: \"kubernetes.io/projected/851fb6ff-4a58-431b-834f-db3f87b67aa9-kube-api-access-hvdd8\") pod \"iptables-alerter-tq7pf\" (UID: \"851fb6ff-4a58-431b-834f-db3f87b67aa9\") " pod="openshift-network-operator/iptables-alerter-tq7pf"
Oct 30 06:57:22.745200 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.745125    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9c5ccc21-fce2-4221-9c19-4a35fef32eff-crio-socket\") pod \"insights-runtime-extractor-ltdxc\" (UID: \"9c5ccc21-fce2-4221-9c19-4a35fef32eff\") " pod="openshift-insights/insights-runtime-extractor-ltdxc"
Oct 30 06:57:22.745200 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.745165    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-host-var-lib-kubelet\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:57:22.745345 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.745203    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ca422300-6212-4a01-b1e2-5ba04af79ce6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4js5r\" (UID: \"ca422300-6212-4a01-b1e2-5ba04af79ce6\") " pod="openshift-multus/multus-additional-cni-plugins-4js5r"
Oct 30 06:57:22.745345 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.745221    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fe292d5d-6d46-4c34-ab24-7991571cd680-etc-sysconfig\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:57:22.745345 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.745240    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/851fb6ff-4a58-431b-834f-db3f87b67aa9-host-slash\") pod \"iptables-alerter-tq7pf\" (UID: \"851fb6ff-4a58-431b-834f-db3f87b67aa9\") " pod="openshift-network-operator/iptables-alerter-tq7pf"
Oct 30 06:57:22.745345 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.745248    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-host-var-lib-kubelet\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:57:22.745345 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.745266    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-host-cni-bin\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:57:22.745345 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.745273    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ca422300-6212-4a01-b1e2-5ba04af79ce6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4js5r\" (UID: \"ca422300-6212-4a01-b1e2-5ba04af79ce6\") " pod="openshift-multus/multus-additional-cni-plugins-4js5r"
Oct 30 06:57:22.745345 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.745251    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9c5ccc21-fce2-4221-9c19-4a35fef32eff-crio-socket\") pod \"insights-runtime-extractor-ltdxc\" (UID: \"9c5ccc21-fce2-4221-9c19-4a35fef32eff\") " pod="openshift-insights/insights-runtime-extractor-ltdxc"
Oct 30 06:57:22.745345 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.745299    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fe292d5d-6d46-4c34-ab24-7991571cd680-etc-sysconfig\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:57:22.745345 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.745308    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a6968152-3c44-4205-a078-f857c413ad91-sys-fs\") pod \"aws-ebs-csi-driver-node-b8kcn\" (UID: \"a6968152-3c44-4205-a078-f857c413ad91\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8kcn"
Oct 30 06:57:22.745345 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.745303    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-host-cni-bin\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:57:22.745345 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.745328    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/851fb6ff-4a58-431b-834f-db3f87b67aa9-host-slash\") pod \"iptables-alerter-tq7pf\" (UID: \"851fb6ff-4a58-431b-834f-db3f87b67aa9\") " pod="openshift-network-operator/iptables-alerter-tq7pf"
Oct 30 06:57:22.745345 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.745337    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-host-run-netns\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:57:22.745910 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.745359    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-etc-kubernetes\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:57:22.745910 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.745393    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-etc-kubernetes\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:57:22.745910 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.745418    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-host-run-netns\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:57:22.745910 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.745426    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a6968152-3c44-4205-a078-f857c413ad91-sys-fs\") pod \"aws-ebs-csi-driver-node-b8kcn\" (UID: \"a6968152-3c44-4205-a078-f857c413ad91\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8kcn"
Oct 30 06:57:22.745910 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.745432    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fe292d5d-6d46-4c34-ab24-7991571cd680-sys\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:57:22.745910 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.745459    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fe292d5d-6d46-4c34-ab24-7991571cd680-sys\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:57:22.745910 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.745481    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-hostroot\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:57:22.745910 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.745509    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-hostroot\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:57:22.745910 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.745515    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-host-var-lib-cni-bin\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:57:22.745910 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.745536    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-host-var-lib-cni-bin\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:57:22.745910 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.745544    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-host-slash\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:57:22.745910 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.745573    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fe292d5d-6d46-4c34-ab24-7991571cd680-lib-modules\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:57:22.745910 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.745590    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-host-slash\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:57:22.745910 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.745596    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-host-kubelet\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:57:22.745910 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.745622    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-host-run-ovn-kubernetes\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:57:22.745910 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.745647    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-cnibin\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:57:22.745910 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.745668    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fe292d5d-6d46-4c34-ab24-7991571cd680-lib-modules\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:57:22.745910 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.745669    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-host-cni-netd\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:57:22.745910 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.745683    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-host-run-ovn-kubernetes\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:57:22.745910 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.745696    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-host-kubelet\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:57:22.745910 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.745696    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fe292d5d-6d46-4c34-ab24-7991571cd680-etc-modprobe-d\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:57:22.745910 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.745707    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-host-cni-netd\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:57:22.745910 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.745710    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-cnibin\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:57:22.745910 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.745720    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fe292d5d-6d46-4c34-ab24-7991571cd680-run\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:57:22.745910 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.745736    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-host-run-k8s-cni-cncf-io\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:57:22.745910 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.745762    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a6968152-3c44-4205-a078-f857c413ad91-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b8kcn\" (UID: \"a6968152-3c44-4205-a078-f857c413ad91\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8kcn"
Oct 30 06:57:22.745910 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.745780    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fe292d5d-6d46-4c34-ab24-7991571cd680-run\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:57:22.745910 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.745785    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-run-systemd\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:57:22.745910 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.745801    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-host-run-k8s-cni-cncf-io\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:57:22.745910 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.745841    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a6968152-3c44-4205-a078-f857c413ad91-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b8kcn\" (UID: \"a6968152-3c44-4205-a078-f857c413ad91\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8kcn"
Oct 30 06:57:22.745910 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.745839    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a6968152-3c44-4205-a078-f857c413ad91-socket-dir\") pod \"aws-ebs-csi-driver-node-b8kcn\" (UID: \"a6968152-3c44-4205-a078-f857c413ad91\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8kcn"
Oct 30 06:57:22.745910 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.745864    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-log-socket\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:57:22.745910 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.745858    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-run-systemd\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:57:22.745910 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.745887    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9092e2e8-ad23-426c-b6f1-14e1a2b9221e-dbus\") pod \"global-pull-secret-syncer-qktbg\" (UID: \"9092e2e8-ad23-426c-b6f1-14e1a2b9221e\") " pod="kube-system/global-pull-secret-syncer-qktbg"
Oct 30 06:57:22.745910 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.745889    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a6968152-3c44-4205-a078-f857c413ad91-socket-dir\") pod \"aws-ebs-csi-driver-node-b8kcn\" (UID: \"a6968152-3c44-4205-a078-f857c413ad91\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8kcn"
Oct 30 06:57:22.745910 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.745920    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f7666c14-ff3f-44a5-9cce-21ff2dfda0e2-root\") pod \"node-exporter-n7zk9\" (UID: \"f7666c14-ff3f-44a5-9cce-21ff2dfda0e2\") " pod="openshift-monitoring/node-exporter-n7zk9"
Oct 30 06:57:22.745910 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.745944    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9092e2e8-ad23-426c-b6f1-14e1a2b9221e-dbus\") pod \"global-pull-secret-syncer-qktbg\" (UID: \"9092e2e8-ad23-426c-b6f1-14e1a2b9221e\") " pod="kube-system/global-pull-secret-syncer-qktbg"
Oct 30 06:57:22.747158 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.745948    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-systemd-units\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:57:22.747158 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.745971    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-multus-conf-dir\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:57:22.747158 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.745987    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/675dc1eb-ff05-40c7-bf0b-fb3161d3cc8c-hosts-file\") pod \"node-resolver-5p67n\" (UID: \"675dc1eb-ff05-40c7-bf0b-fb3161d3cc8c\") " pod="openshift-dns/node-resolver-5p67n"
Oct 30 06:57:22.747158 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.745992    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-systemd-units\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:57:22.747158 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.746001    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fe292d5d-6d46-4c34-ab24-7991571cd680-etc-systemd\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:57:22.747158 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.746006    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-multus-conf-dir\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:57:22.747158 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.745950    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f7666c14-ff3f-44a5-9cce-21ff2dfda0e2-root\") pod \"node-exporter-n7zk9\" (UID: \"f7666c14-ff3f-44a5-9cce-21ff2dfda0e2\") " pod="openshift-monitoring/node-exporter-n7zk9"
Oct 30 06:57:22.747158 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.746023    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-host-run-netns\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:57:22.747158 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.745921    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-log-socket\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:57:22.747158 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.746038    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fe292d5d-6d46-4c34-ab24-7991571cd680-etc-systemd\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:57:22.747158 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.746059    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-host-run-netns\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:57:22.747158 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.746099    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-host-var-lib-cni-multus\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:57:22.747158 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.746123    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-host-var-lib-cni-multus\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:57:22.747158 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.746130    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-run-openvswitch\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:57:22.747158 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.746158    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-run-openvswitch\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:57:22.747158 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.746158    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-run-ovn\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:57:22.747158 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.746180    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ca422300-6212-4a01-b1e2-5ba04af79ce6-system-cni-dir\") pod \"multus-additional-cni-plugins-4js5r\" (UID: \"ca422300-6212-4a01-b1e2-5ba04af79ce6\") " pod="openshift-multus/multus-additional-cni-plugins-4js5r"
Oct 30 06:57:22.747158 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.746189    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-run-ovn\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:57:22.747158 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.746197    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ca422300-6212-4a01-b1e2-5ba04af79ce6-cnibin\") pod \"multus-additional-cni-plugins-4js5r\" (UID: \"ca422300-6212-4a01-b1e2-5ba04af79ce6\") " pod="openshift-multus/multus-additional-cni-plugins-4js5r"
Oct 30 06:57:22.747158 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.746218    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ca422300-6212-4a01-b1e2-5ba04af79ce6-cnibin\") pod \"multus-additional-cni-plugins-4js5r\" (UID: \"ca422300-6212-4a01-b1e2-5ba04af79ce6\") " pod="openshift-multus/multus-additional-cni-plugins-4js5r"
Oct 30 06:57:22.747158 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.745789    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fe292d5d-6d46-4c34-ab24-7991571cd680-etc-modprobe-d\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:57:22.747158 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.746235    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-host-run-multus-certs\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:57:22.747158 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.746249    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ca422300-6212-4a01-b1e2-5ba04af79ce6-system-cni-dir\") pod \"multus-additional-cni-plugins-4js5r\" (UID: \"ca422300-6212-4a01-b1e2-5ba04af79ce6\") " pod="openshift-multus/multus-additional-cni-plugins-4js5r"
Oct 30 06:57:22.747158 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.746265    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fe292d5d-6d46-4c34-ab24-7991571cd680-host\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:57:22.747158 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.746296    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9092e2e8-ad23-426c-b6f1-14e1a2b9221e-kubelet-config\") pod \"global-pull-secret-syncer-qktbg\" (UID: \"9092e2e8-ad23-426c-b6f1-14e1a2b9221e\") " pod="kube-system/global-pull-secret-syncer-qktbg"
Oct 30 06:57:22.747158 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.746311    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-host-run-multus-certs\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:57:22.747158 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.746323    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-multus-socket-dir-parent\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:57:22.747158 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.746348    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9092e2e8-ad23-426c-b6f1-14e1a2b9221e-kubelet-config\") pod \"global-pull-secret-syncer-qktbg\" (UID: \"9092e2e8-ad23-426c-b6f1-14e1a2b9221e\") " pod="kube-system/global-pull-secret-syncer-qktbg"
Oct 30 06:57:22.747158 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.746327    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fe292d5d-6d46-4c34-ab24-7991571cd680-host\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:57:22.747158 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.746359    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f7666c14-ff3f-44a5-9cce-21ff2dfda0e2-node-exporter-wtmp\") pod \"node-exporter-n7zk9\" (UID: \"f7666c14-ff3f-44a5-9cce-21ff2dfda0e2\") " pod="openshift-monitoring/node-exporter-n7zk9"
Oct 30 06:57:22.747158 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.746378    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-multus-socket-dir-parent\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:57:22.747158 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.746385    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-multus-cni-dir\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:57:22.747158 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.746415    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-node-log\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:57:22.747158 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.746442    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:57:22.747158 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.746469    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f7666c14-ff3f-44a5-9cce-21ff2dfda0e2-node-exporter-wtmp\") pod \"node-exporter-n7zk9\" (UID: \"f7666c14-ff3f-44a5-9cce-21ff2dfda0e2\") " pod="openshift-monitoring/node-exporter-n7zk9"
Oct 30 06:57:22.747158 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.746479    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-multus-cni-dir\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:57:22.747158 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.746476    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-node-log\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:57:22.747158 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.746480    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:57:22.747158 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.746506    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/675dc1eb-ff05-40c7-bf0b-fb3161d3cc8c-hosts-file\") pod \"node-resolver-5p67n\" (UID: \"675dc1eb-ff05-40c7-bf0b-fb3161d3cc8c\") " pod="openshift-dns/node-resolver-5p67n"
Oct 30 06:57:22.747158 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.746492    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-system-cni-dir\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:57:22.747158 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.746526    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2368101c-6719-4650-8393-ef048500a624-system-cni-dir\") pod \"multus-9gqrp\" (UID: \"2368101c-6719-4650-8393-ef048500a624\") " pod="openshift-multus/multus-9gqrp"
Oct 30 06:57:22.747158 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.746537    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a6968152-3c44-4205-a078-f857c413ad91-registration-dir\") pod \"aws-ebs-csi-driver-node-b8kcn\" (UID: \"a6968152-3c44-4205-a078-f857c413ad91\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8kcn"
Oct 30 06:57:22.747158 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.746552    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a6968152-3c44-4205-a078-f857c413ad91-device-dir\") pod \"aws-ebs-csi-driver-node-b8kcn\" (UID: \"a6968152-3c44-4205-a078-f857c413ad91\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8kcn"
Oct 30 06:57:22.747158 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.746571    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a6968152-3c44-4205-a078-f857c413ad91-etc-selinux\") pod \"aws-ebs-csi-driver-node-b8kcn\" (UID: \"a6968152-3c44-4205-a078-f857c413ad91\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8kcn"
Oct 30 06:57:22.747158 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.746575    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a6968152-3c44-4205-a078-f857c413ad91-registration-dir\") pod \"aws-ebs-csi-driver-node-b8kcn\" (UID: \"a6968152-3c44-4205-a078-f857c413ad91\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8kcn"
Oct 30 06:57:22.747158 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.746596    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-var-lib-openvswitch\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:57:22.747158 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.746606    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a6968152-3c44-4205-a078-f857c413ad91-etc-selinux\") pod \"aws-ebs-csi-driver-node-b8kcn\" (UID: \"a6968152-3c44-4205-a078-f857c413ad91\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8kcn"
Oct 30 06:57:22.747158 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.746617    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe292d5d-6d46-4c34-ab24-7991571cd680-etc-kubernetes\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:57:22.747158 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.746639    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a6968152-3c44-4205-a078-f857c413ad91-device-dir\") pod \"aws-ebs-csi-driver-node-b8kcn\" (UID: \"a6968152-3c44-4205-a078-f857c413ad91\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8kcn"
Oct 30 06:57:22.747158 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.746643    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-var-lib-openvswitch\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:57:22.747158 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.746639    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fe292d5d-6d46-4c34-ab24-7991571cd680-etc-sysctl-d\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:57:22.747158 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.746670    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe292d5d-6d46-4c34-ab24-7991571cd680-etc-kubernetes\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:57:22.747158 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.746679    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fe292d5d-6d46-4c34-ab24-7991571cd680-etc-sysctl-conf\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:57:22.747158 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.746702    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d87089b-8b1b-43ea-be1f-72bfb56bf311-host\") pod \"node-ca-l9mfn\" (UID: \"5d87089b-8b1b-43ea-be1f-72bfb56bf311\") " pod="openshift-image-registry/node-ca-l9mfn"
Oct 30 06:57:22.747158 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.746727    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-etc-openvswitch\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:57:22.747158 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.746740    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fe292d5d-6d46-4c34-ab24-7991571cd680-etc-sysctl-d\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:57:22.747158 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.746749    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe292d5d-6d46-4c34-ab24-7991571cd680-var-lib-kubelet\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:57:22.747158 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.746761    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d87089b-8b1b-43ea-be1f-72bfb56bf311-host\") pod \"node-ca-l9mfn\" (UID: \"5d87089b-8b1b-43ea-be1f-72bfb56bf311\") " pod="openshift-image-registry/node-ca-l9mfn"
Oct 30 06:57:22.747158 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.746774    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b321b279-a102-48b9-bd9b-8ed938eed0f7-etc-openvswitch\") pod \"ovnkube-node-r9jjx\" (UID: \"b321b279-a102-48b9-bd9b-8ed938eed0f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:57:22.747158 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.746777    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f7666c14-ff3f-44a5-9cce-21ff2dfda0e2-sys\") pod \"node-exporter-n7zk9\" (UID: \"f7666c14-ff3f-44a5-9cce-21ff2dfda0e2\") " pod="openshift-monitoring/node-exporter-n7zk9"
Oct 30 06:57:22.747158 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.746785    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe292d5d-6d46-4c34-ab24-7991571cd680-var-lib-kubelet\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:57:22.747158 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.746826    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f7666c14-ff3f-44a5-9cce-21ff2dfda0e2-sys\") pod \"node-exporter-n7zk9\" (UID: \"f7666c14-ff3f-44a5-9cce-21ff2dfda0e2\") " pod="openshift-monitoring/node-exporter-n7zk9"
Oct 30 06:57:22.747158 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:22.746865    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fe292d5d-6d46-4c34-ab24-7991571cd680-etc-sysctl-conf\") pod \"tuned-4kfdw\" (UID: \"fe292d5d-6d46-4c34-ab24-7991571cd680\") " pod="openshift-cluster-node-tuning-operator/tuned-4kfdw"
Oct 30 06:57:24.209932 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:24.209899    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-c6trh"
Oct 30 06:57:24.314184 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:24.314156    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:57:24.479851 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:24.479757    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dncbb"
Oct 30 06:57:25.010534 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:25.010509    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:57:25.010764 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:25.010708    6224 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 06:57:25.791499 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:25.791463    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:25.807523 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:25.807488    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:30.838530 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:30.838503    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-f866fbc59-wnrl6"
Oct 30 06:57:31.032853 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:31.032826    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r9jjx"
Oct 30 06:57:31.541343 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:31.541312    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-f866fbc59-wnrl6"
Oct 30 06:57:31.857220 ip-10-0-128-125 kubenswrapper[6224]: E1030 06:57:31.857128    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 06:57:35.222343 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:35.222268    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-769d4f7fd-5sbjs"
Oct 30 06:57:40.843399 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:40.843369    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-f866fbc59-wnrl6"
Oct 30 06:57:40.847100 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:40.847066    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-f866fbc59-wnrl6"
Oct 30 06:57:41.533890 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:41.533862    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wv8tb"
Oct 30 06:57:45.823734 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:45.823703    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 06:57:45.824322 ip-10-0-128-125 kubenswrapper[6224]: E1030 06:57:45.823846    6224 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="36e72f7afaaed1114f7178cf75aadb86bb06d852fa4d40991839fe3c650e7d00" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Oct 30 06:57:45.824322 ip-10-0-128-125 kubenswrapper[6224]: E1030 06:57:45.823920    6224 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="36e72f7afaaed1114f7178cf75aadb86bb06d852fa4d40991839fe3c650e7d00" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Oct 30 06:57:45.824322 ip-10-0-128-125 kubenswrapper[6224]: E1030 06:57:45.823938    6224 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="36e72f7afaaed1114f7178cf75aadb86bb06d852fa4d40991839fe3c650e7d00" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Oct 30 06:57:45.824322 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:45.823962    6224 prober.go:106] "Probe errored" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="12c298a7-e78a-45d7-abe1-7b7ef6f5df03" containerName="prometheus" probeResult="unknown" err="rpc error: code = Canceled desc = context canceled"
Oct 30 06:57:45.824322 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:45.824080    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="12c298a7-e78a-45d7-abe1-7b7ef6f5df03" containerName="prometheus" containerID="cri-o://36e72f7afaaed1114f7178cf75aadb86bb06d852fa4d40991839fe3c650e7d00" gracePeriod=600
Oct 30 06:57:45.824322 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:45.824094    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="12c298a7-e78a-45d7-abe1-7b7ef6f5df03" containerName="kube-rbac-proxy-web" containerID="cri-o://ed4a02af2dab9b59fe33955fd09ad859f10218c846369b3f974b28f274e9e777" gracePeriod=600
Oct 30 06:57:45.824322 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:45.824109    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="12c298a7-e78a-45d7-abe1-7b7ef6f5df03" containerName="kube-rbac-proxy-thanos" containerID="cri-o://82d8cccc3e759d6d5db1f4f50f4fdc6bb433649ad57238011dd1b6f318c733b7" gracePeriod=600
Oct 30 06:57:45.824322 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:45.824188    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="12c298a7-e78a-45d7-abe1-7b7ef6f5df03" containerName="config-reloader" containerID="cri-o://26f5047a4a9d0850c61a1f7288ba0f74492952019ef4ac4beaec3e9e53c98040" gracePeriod=600
Oct 30 06:57:45.824322 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:45.824207    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="12c298a7-e78a-45d7-abe1-7b7ef6f5df03" containerName="thanos-sidecar" containerID="cri-o://88c9314ef41c71965d607688400aa8998adb43abe9255652d83e65e5e9d5e549" gracePeriod=600
Oct 30 06:57:45.824322 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:45.824248    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="12c298a7-e78a-45d7-abe1-7b7ef6f5df03" containerName="kube-rbac-proxy" containerID="cri-o://cecc64cca52be72b9ea23c344fb7d680a37d2d23b00465da4008fd8f08d83587" gracePeriod=600
Oct 30 06:57:46.085609 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.085556    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:46.119247 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.119221    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-secret-grpc-tls\") pod \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") "
Oct 30 06:57:46.119362 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.119254    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") "
Oct 30 06:57:46.119362 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.119278    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-config\" (UniqueName: \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-config\") pod \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") "
Oct 30 06:57:46.119362 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.119295    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-secret-kube-rbac-proxy\") pod \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") "
Oct 30 06:57:46.119362 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.119317    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-prometheus-trusted-ca-bundle\") pod \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") "
Oct 30 06:57:46.119639 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.119445    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-configmap-kubelet-serving-ca-bundle\") pod \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") "
Oct 30 06:57:46.119639 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.119486    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-prometheus-k8s-rulefiles-0\") pod \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") "
Oct 30 06:57:46.119639 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.119538    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-prometheus-k8s-db\") pod \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") "
Oct 30 06:57:46.119639 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.119569    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-kube-api-access-n9bkg\" (UniqueName: \"kubernetes.io/projected/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-kube-api-access-n9bkg\") pod \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") "
Oct 30 06:57:46.119639 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.119600    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-prometheus-k8s-rulefiles-2\") pod \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") "
Oct 30 06:57:46.119925 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.119680    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-web-config\" (UniqueName: \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-web-config\") pod \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") "
Oct 30 06:57:46.119925 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.119714    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-secret-prometheus-k8s-tls\") pod \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") "
Oct 30 06:57:46.119925 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.119745    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-prometheus-k8s-rulefiles-1\") pod \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") "
Oct 30 06:57:46.119925 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.119745    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-prometheus-trusted-ca-bundle" pod "12c298a7-e78a-45d7-abe1-7b7ef6f5df03" (UID: "12c298a7-e78a-45d7-abe1-7b7ef6f5df03"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:57:46.119925 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.119773    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-configmap-serving-certs-ca-bundle\") pod \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") "
Oct 30 06:57:46.119925 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.119837    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-config-out\" (UniqueName: \"kubernetes.io/empty-dir/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-config-out\") pod \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") "
Oct 30 06:57:46.119925 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.119878    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-tls-assets\" (UniqueName: \"kubernetes.io/projected/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-tls-assets\") pod \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") "
Oct 30 06:57:46.119925 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.119882    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-configmap-kubelet-serving-ca-bundle" pod "12c298a7-e78a-45d7-abe1-7b7ef6f5df03" (UID: "12c298a7-e78a-45d7-abe1-7b7ef6f5df03"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:57:46.119925 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.119912    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-thanos-prometheus-http-client-file\") pod \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") "
Oct 30 06:57:46.120376 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.120131    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-prometheus-k8s-rulefiles-2" pod "12c298a7-e78a-45d7-abe1-7b7ef6f5df03" (UID: "12c298a7-e78a-45d7-abe1-7b7ef6f5df03"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:57:46.120435 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.120332    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-prometheus-k8s-rulefiles-1" pod "12c298a7-e78a-45d7-abe1-7b7ef6f5df03" (UID: "12c298a7-e78a-45d7-abe1-7b7ef6f5df03"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:57:46.120435 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.120418    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-secret-metrics-client-certs\") pod \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") "
Oct 30 06:57:46.120541 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.120459    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-configmap-metrics-client-ca\") pod \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") "
Oct 30 06:57:46.120541 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.120503    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\" (UID: \"12c298a7-e78a-45d7-abe1-7b7ef6f5df03\") "
Oct 30 06:57:46.120641 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.120604    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-prometheus-k8s-db" pod "12c298a7-e78a-45d7-abe1-7b7ef6f5df03" (UID: "12c298a7-e78a-45d7-abe1-7b7ef6f5df03"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:57:46.120780 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.120759    6224 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-prometheus-trusted-ca-bundle\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 06:57:46.120857 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.120787    6224 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 06:57:46.120857 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.120823    6224 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-prometheus-k8s-db\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 06:57:46.120857 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.120840    6224 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 06:57:46.120857 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.120854    6224 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 06:57:46.122417 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.122284    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-config" pod "12c298a7-e78a-45d7-abe1-7b7ef6f5df03" (UID: "12c298a7-e78a-45d7-abe1-7b7ef6f5df03"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:57:46.122417 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.122319    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-prometheus-k8s-rulefiles-0" pod "12c298a7-e78a-45d7-abe1-7b7ef6f5df03" (UID: "12c298a7-e78a-45d7-abe1-7b7ef6f5df03"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:57:46.122794 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.122754    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-secret-prometheus-k8s-kube-rbac-proxy-web" pod "12c298a7-e78a-45d7-abe1-7b7ef6f5df03" (UID: "12c298a7-e78a-45d7-abe1-7b7ef6f5df03"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:57:46.123084 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.123010    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-config-out" pod "12c298a7-e78a-45d7-abe1-7b7ef6f5df03" (UID: "12c298a7-e78a-45d7-abe1-7b7ef6f5df03"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:57:46.123223 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.123199    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-configmap-serving-certs-ca-bundle" pod "12c298a7-e78a-45d7-abe1-7b7ef6f5df03" (UID: "12c298a7-e78a-45d7-abe1-7b7ef6f5df03"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:57:46.123356 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.123332    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-configmap-metrics-client-ca" pod "12c298a7-e78a-45d7-abe1-7b7ef6f5df03" (UID: "12c298a7-e78a-45d7-abe1-7b7ef6f5df03"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:57:46.123732 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.123707    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-secret-kube-rbac-proxy" pod "12c298a7-e78a-45d7-abe1-7b7ef6f5df03" (UID: "12c298a7-e78a-45d7-abe1-7b7ef6f5df03"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:57:46.124024 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.123998    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-secret-grpc-tls" pod "12c298a7-e78a-45d7-abe1-7b7ef6f5df03" (UID: "12c298a7-e78a-45d7-abe1-7b7ef6f5df03"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:57:46.124123 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.124089    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-secret-prometheus-k8s-thanos-sidecar-tls" pod "12c298a7-e78a-45d7-abe1-7b7ef6f5df03" (UID: "12c298a7-e78a-45d7-abe1-7b7ef6f5df03"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:57:46.124308 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.124286    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-secret-metrics-client-certs" pod "12c298a7-e78a-45d7-abe1-7b7ef6f5df03" (UID: "12c298a7-e78a-45d7-abe1-7b7ef6f5df03"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:57:46.124471 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.124448    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-kube-api-access-n9bkg" pod "12c298a7-e78a-45d7-abe1-7b7ef6f5df03" (UID: "12c298a7-e78a-45d7-abe1-7b7ef6f5df03"). InnerVolumeSpecName "kube-api-access-n9bkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:57:46.124543 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.124488    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-secret-prometheus-k8s-tls" pod "12c298a7-e78a-45d7-abe1-7b7ef6f5df03" (UID: "12c298a7-e78a-45d7-abe1-7b7ef6f5df03"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:57:46.124994 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.124971    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-thanos-prometheus-http-client-file" pod "12c298a7-e78a-45d7-abe1-7b7ef6f5df03" (UID: "12c298a7-e78a-45d7-abe1-7b7ef6f5df03"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:57:46.125259 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.125244    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-tls-assets" pod "12c298a7-e78a-45d7-abe1-7b7ef6f5df03" (UID: "12c298a7-e78a-45d7-abe1-7b7ef6f5df03"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:57:46.133272 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.133251    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-web-config" pod "12c298a7-e78a-45d7-abe1-7b7ef6f5df03" (UID: "12c298a7-e78a-45d7-abe1-7b7ef6f5df03"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:57:46.222202 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.222167    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9bkg\" (UniqueName: \"kubernetes.io/projected/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-kube-api-access-n9bkg\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 06:57:46.222202 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.222205    6224 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-web-config\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 06:57:46.222415 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.222220    6224 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-secret-prometheus-k8s-tls\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 06:57:46.222415 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.222235    6224 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 06:57:46.222415 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.222251    6224 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-config-out\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 06:57:46.222415 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.222264    6224 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-tls-assets\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 06:57:46.222415 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.222277    6224 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-thanos-prometheus-http-client-file\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 06:57:46.222415 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.222291    6224 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-secret-metrics-client-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 06:57:46.222415 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.222305    6224 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-configmap-metrics-client-ca\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 06:57:46.222415 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.222319    6224 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 06:57:46.222415 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.222333    6224 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-secret-grpc-tls\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 06:57:46.222415 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.222346    6224 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 06:57:46.222415 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.222358    6224 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-config\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 06:57:46.222415 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.222370    6224 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-secret-kube-rbac-proxy\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 06:57:46.222415 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.222383    6224 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/12c298a7-e78a-45d7-abe1-7b7ef6f5df03-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 06:57:46.641605 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.641568    6224 generic.go:356] "Generic (PLEG): container finished" podID="12c298a7-e78a-45d7-abe1-7b7ef6f5df03" containerID="82d8cccc3e759d6d5db1f4f50f4fdc6bb433649ad57238011dd1b6f318c733b7" exitCode=0
Oct 30 06:57:46.641605 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.641594    6224 generic.go:356] "Generic (PLEG): container finished" podID="12c298a7-e78a-45d7-abe1-7b7ef6f5df03" containerID="cecc64cca52be72b9ea23c344fb7d680a37d2d23b00465da4008fd8f08d83587" exitCode=0
Oct 30 06:57:46.641605 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.641601    6224 generic.go:356] "Generic (PLEG): container finished" podID="12c298a7-e78a-45d7-abe1-7b7ef6f5df03" containerID="ed4a02af2dab9b59fe33955fd09ad859f10218c846369b3f974b28f274e9e777" exitCode=0
Oct 30 06:57:46.641605 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.641606    6224 generic.go:356] "Generic (PLEG): container finished" podID="12c298a7-e78a-45d7-abe1-7b7ef6f5df03" containerID="88c9314ef41c71965d607688400aa8998adb43abe9255652d83e65e5e9d5e549" exitCode=0
Oct 30 06:57:46.641605 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.641612    6224 generic.go:356] "Generic (PLEG): container finished" podID="12c298a7-e78a-45d7-abe1-7b7ef6f5df03" containerID="26f5047a4a9d0850c61a1f7288ba0f74492952019ef4ac4beaec3e9e53c98040" exitCode=0
Oct 30 06:57:46.641605 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.641617    6224 generic.go:356] "Generic (PLEG): container finished" podID="12c298a7-e78a-45d7-abe1-7b7ef6f5df03" containerID="36e72f7afaaed1114f7178cf75aadb86bb06d852fa4d40991839fe3c650e7d00" exitCode=0
Oct 30 06:57:46.642065 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.641642    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"12c298a7-e78a-45d7-abe1-7b7ef6f5df03","Type":"ContainerDied","Data":"82d8cccc3e759d6d5db1f4f50f4fdc6bb433649ad57238011dd1b6f318c733b7"}
Oct 30 06:57:46.642065 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.641662    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:46.642065 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.641710    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"12c298a7-e78a-45d7-abe1-7b7ef6f5df03","Type":"ContainerDied","Data":"cecc64cca52be72b9ea23c344fb7d680a37d2d23b00465da4008fd8f08d83587"}
Oct 30 06:57:46.642065 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.641738    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"12c298a7-e78a-45d7-abe1-7b7ef6f5df03","Type":"ContainerDied","Data":"ed4a02af2dab9b59fe33955fd09ad859f10218c846369b3f974b28f274e9e777"}
Oct 30 06:57:46.642065 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.641765    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"12c298a7-e78a-45d7-abe1-7b7ef6f5df03","Type":"ContainerDied","Data":"88c9314ef41c71965d607688400aa8998adb43abe9255652d83e65e5e9d5e549"}
Oct 30 06:57:46.642065 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.641788    6224 scope.go:122] "RemoveContainer" containerID="82d8cccc3e759d6d5db1f4f50f4fdc6bb433649ad57238011dd1b6f318c733b7"
Oct 30 06:57:46.642065 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.641792    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"12c298a7-e78a-45d7-abe1-7b7ef6f5df03","Type":"ContainerDied","Data":"26f5047a4a9d0850c61a1f7288ba0f74492952019ef4ac4beaec3e9e53c98040"}
Oct 30 06:57:46.642065 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.641834    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"12c298a7-e78a-45d7-abe1-7b7ef6f5df03","Type":"ContainerDied","Data":"36e72f7afaaed1114f7178cf75aadb86bb06d852fa4d40991839fe3c650e7d00"}
Oct 30 06:57:46.642065 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.641859    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"12c298a7-e78a-45d7-abe1-7b7ef6f5df03","Type":"ContainerDied","Data":"61eb4fbe80838432dc327dc06a8580e4a1d35a985986b0c2c113de76ac733a9c"}
Oct 30 06:57:46.649661 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.649645    6224 scope.go:122] "RemoveContainer" containerID="cecc64cca52be72b9ea23c344fb7d680a37d2d23b00465da4008fd8f08d83587"
Oct 30 06:57:46.656316 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.656298    6224 scope.go:122] "RemoveContainer" containerID="ed4a02af2dab9b59fe33955fd09ad859f10218c846369b3f974b28f274e9e777"
Oct 30 06:57:46.662465 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.662447    6224 scope.go:122] "RemoveContainer" containerID="88c9314ef41c71965d607688400aa8998adb43abe9255652d83e65e5e9d5e549"
Oct 30 06:57:46.668517 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.668500    6224 scope.go:122] "RemoveContainer" containerID="26f5047a4a9d0850c61a1f7288ba0f74492952019ef4ac4beaec3e9e53c98040"
Oct 30 06:57:46.675165 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.675145    6224 scope.go:122] "RemoveContainer" containerID="36e72f7afaaed1114f7178cf75aadb86bb06d852fa4d40991839fe3c650e7d00"
Oct 30 06:57:46.681488 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.681474    6224 scope.go:122] "RemoveContainer" containerID="7ea8feb9571f349fc2119c3f7c6e1421fe27e22e55f90a4b5370acc7af3d92bb"
Oct 30 06:57:46.687494 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.687479    6224 scope.go:122] "RemoveContainer" containerID="82d8cccc3e759d6d5db1f4f50f4fdc6bb433649ad57238011dd1b6f318c733b7"
Oct 30 06:57:46.687742 ip-10-0-128-125 kubenswrapper[6224]: E1030 06:57:46.687727    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82d8cccc3e759d6d5db1f4f50f4fdc6bb433649ad57238011dd1b6f318c733b7\": container with ID starting with 82d8cccc3e759d6d5db1f4f50f4fdc6bb433649ad57238011dd1b6f318c733b7 not found: ID does not exist" containerID="82d8cccc3e759d6d5db1f4f50f4fdc6bb433649ad57238011dd1b6f318c733b7"
Oct 30 06:57:46.687787 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.687753    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82d8cccc3e759d6d5db1f4f50f4fdc6bb433649ad57238011dd1b6f318c733b7"} err="failed to get container status \"82d8cccc3e759d6d5db1f4f50f4fdc6bb433649ad57238011dd1b6f318c733b7\": rpc error: code = NotFound desc = could not find container \"82d8cccc3e759d6d5db1f4f50f4fdc6bb433649ad57238011dd1b6f318c733b7\": container with ID starting with 82d8cccc3e759d6d5db1f4f50f4fdc6bb433649ad57238011dd1b6f318c733b7 not found: ID does not exist"
Oct 30 06:57:46.687787 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.687781    6224 scope.go:122] "RemoveContainer" containerID="cecc64cca52be72b9ea23c344fb7d680a37d2d23b00465da4008fd8f08d83587"
Oct 30 06:57:46.688020 ip-10-0-128-125 kubenswrapper[6224]: E1030 06:57:46.688003    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cecc64cca52be72b9ea23c344fb7d680a37d2d23b00465da4008fd8f08d83587\": container with ID starting with cecc64cca52be72b9ea23c344fb7d680a37d2d23b00465da4008fd8f08d83587 not found: ID does not exist" containerID="cecc64cca52be72b9ea23c344fb7d680a37d2d23b00465da4008fd8f08d83587"
Oct 30 06:57:46.688065 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.688025    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cecc64cca52be72b9ea23c344fb7d680a37d2d23b00465da4008fd8f08d83587"} err="failed to get container status \"cecc64cca52be72b9ea23c344fb7d680a37d2d23b00465da4008fd8f08d83587\": rpc error: code = NotFound desc = could not find container \"cecc64cca52be72b9ea23c344fb7d680a37d2d23b00465da4008fd8f08d83587\": container with ID starting with cecc64cca52be72b9ea23c344fb7d680a37d2d23b00465da4008fd8f08d83587 not found: ID does not exist"
Oct 30 06:57:46.688065 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.688037    6224 scope.go:122] "RemoveContainer" containerID="ed4a02af2dab9b59fe33955fd09ad859f10218c846369b3f974b28f274e9e777"
Oct 30 06:57:46.688239 ip-10-0-128-125 kubenswrapper[6224]: E1030 06:57:46.688219    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed4a02af2dab9b59fe33955fd09ad859f10218c846369b3f974b28f274e9e777\": container with ID starting with ed4a02af2dab9b59fe33955fd09ad859f10218c846369b3f974b28f274e9e777 not found: ID does not exist" containerID="ed4a02af2dab9b59fe33955fd09ad859f10218c846369b3f974b28f274e9e777"
Oct 30 06:57:46.688276 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.688244    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed4a02af2dab9b59fe33955fd09ad859f10218c846369b3f974b28f274e9e777"} err="failed to get container status \"ed4a02af2dab9b59fe33955fd09ad859f10218c846369b3f974b28f274e9e777\": rpc error: code = NotFound desc = could not find container \"ed4a02af2dab9b59fe33955fd09ad859f10218c846369b3f974b28f274e9e777\": container with ID starting with ed4a02af2dab9b59fe33955fd09ad859f10218c846369b3f974b28f274e9e777 not found: ID does not exist"
Oct 30 06:57:46.688276 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.688256    6224 scope.go:122] "RemoveContainer" containerID="88c9314ef41c71965d607688400aa8998adb43abe9255652d83e65e5e9d5e549"
Oct 30 06:57:46.688450 ip-10-0-128-125 kubenswrapper[6224]: E1030 06:57:46.688433    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88c9314ef41c71965d607688400aa8998adb43abe9255652d83e65e5e9d5e549\": container with ID starting with 88c9314ef41c71965d607688400aa8998adb43abe9255652d83e65e5e9d5e549 not found: ID does not exist" containerID="88c9314ef41c71965d607688400aa8998adb43abe9255652d83e65e5e9d5e549"
Oct 30 06:57:46.688487 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.688455    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88c9314ef41c71965d607688400aa8998adb43abe9255652d83e65e5e9d5e549"} err="failed to get container status \"88c9314ef41c71965d607688400aa8998adb43abe9255652d83e65e5e9d5e549\": rpc error: code = NotFound desc = could not find container \"88c9314ef41c71965d607688400aa8998adb43abe9255652d83e65e5e9d5e549\": container with ID starting with 88c9314ef41c71965d607688400aa8998adb43abe9255652d83e65e5e9d5e549 not found: ID does not exist"
Oct 30 06:57:46.688487 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.688468    6224 scope.go:122] "RemoveContainer" containerID="26f5047a4a9d0850c61a1f7288ba0f74492952019ef4ac4beaec3e9e53c98040"
Oct 30 06:57:46.688701 ip-10-0-128-125 kubenswrapper[6224]: E1030 06:57:46.688684    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26f5047a4a9d0850c61a1f7288ba0f74492952019ef4ac4beaec3e9e53c98040\": container with ID starting with 26f5047a4a9d0850c61a1f7288ba0f74492952019ef4ac4beaec3e9e53c98040 not found: ID does not exist" containerID="26f5047a4a9d0850c61a1f7288ba0f74492952019ef4ac4beaec3e9e53c98040"
Oct 30 06:57:46.688745 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.688704    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26f5047a4a9d0850c61a1f7288ba0f74492952019ef4ac4beaec3e9e53c98040"} err="failed to get container status \"26f5047a4a9d0850c61a1f7288ba0f74492952019ef4ac4beaec3e9e53c98040\": rpc error: code = NotFound desc = could not find container \"26f5047a4a9d0850c61a1f7288ba0f74492952019ef4ac4beaec3e9e53c98040\": container with ID starting with 26f5047a4a9d0850c61a1f7288ba0f74492952019ef4ac4beaec3e9e53c98040 not found: ID does not exist"
Oct 30 06:57:46.688745 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.688715    6224 scope.go:122] "RemoveContainer" containerID="36e72f7afaaed1114f7178cf75aadb86bb06d852fa4d40991839fe3c650e7d00"
Oct 30 06:57:46.688947 ip-10-0-128-125 kubenswrapper[6224]: E1030 06:57:46.688931    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36e72f7afaaed1114f7178cf75aadb86bb06d852fa4d40991839fe3c650e7d00\": container with ID starting with 36e72f7afaaed1114f7178cf75aadb86bb06d852fa4d40991839fe3c650e7d00 not found: ID does not exist" containerID="36e72f7afaaed1114f7178cf75aadb86bb06d852fa4d40991839fe3c650e7d00"
Oct 30 06:57:46.688995 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.688950    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36e72f7afaaed1114f7178cf75aadb86bb06d852fa4d40991839fe3c650e7d00"} err="failed to get container status \"36e72f7afaaed1114f7178cf75aadb86bb06d852fa4d40991839fe3c650e7d00\": rpc error: code = NotFound desc = could not find container \"36e72f7afaaed1114f7178cf75aadb86bb06d852fa4d40991839fe3c650e7d00\": container with ID starting with 36e72f7afaaed1114f7178cf75aadb86bb06d852fa4d40991839fe3c650e7d00 not found: ID does not exist"
Oct 30 06:57:46.688995 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.688961    6224 scope.go:122] "RemoveContainer" containerID="7ea8feb9571f349fc2119c3f7c6e1421fe27e22e55f90a4b5370acc7af3d92bb"
Oct 30 06:57:46.689203 ip-10-0-128-125 kubenswrapper[6224]: E1030 06:57:46.689184    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ea8feb9571f349fc2119c3f7c6e1421fe27e22e55f90a4b5370acc7af3d92bb\": container with ID starting with 7ea8feb9571f349fc2119c3f7c6e1421fe27e22e55f90a4b5370acc7af3d92bb not found: ID does not exist" containerID="7ea8feb9571f349fc2119c3f7c6e1421fe27e22e55f90a4b5370acc7af3d92bb"
Oct 30 06:57:46.689250 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.689208    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ea8feb9571f349fc2119c3f7c6e1421fe27e22e55f90a4b5370acc7af3d92bb"} err="failed to get container status \"7ea8feb9571f349fc2119c3f7c6e1421fe27e22e55f90a4b5370acc7af3d92bb\": rpc error: code = NotFound desc = could not find container \"7ea8feb9571f349fc2119c3f7c6e1421fe27e22e55f90a4b5370acc7af3d92bb\": container with ID starting with 7ea8feb9571f349fc2119c3f7c6e1421fe27e22e55f90a4b5370acc7af3d92bb not found: ID does not exist"
Oct 30 06:57:46.689250 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.689224    6224 scope.go:122] "RemoveContainer" containerID="82d8cccc3e759d6d5db1f4f50f4fdc6bb433649ad57238011dd1b6f318c733b7"
Oct 30 06:57:46.689448 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.689427    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82d8cccc3e759d6d5db1f4f50f4fdc6bb433649ad57238011dd1b6f318c733b7"} err="failed to get container status \"82d8cccc3e759d6d5db1f4f50f4fdc6bb433649ad57238011dd1b6f318c733b7\": rpc error: code = NotFound desc = could not find container \"82d8cccc3e759d6d5db1f4f50f4fdc6bb433649ad57238011dd1b6f318c733b7\": container with ID starting with 82d8cccc3e759d6d5db1f4f50f4fdc6bb433649ad57238011dd1b6f318c733b7 not found: ID does not exist"
Oct 30 06:57:46.689448 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.689448    6224 scope.go:122] "RemoveContainer" containerID="cecc64cca52be72b9ea23c344fb7d680a37d2d23b00465da4008fd8f08d83587"
Oct 30 06:57:46.689658 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.689642    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cecc64cca52be72b9ea23c344fb7d680a37d2d23b00465da4008fd8f08d83587"} err="failed to get container status \"cecc64cca52be72b9ea23c344fb7d680a37d2d23b00465da4008fd8f08d83587\": rpc error: code = NotFound desc = could not find container \"cecc64cca52be72b9ea23c344fb7d680a37d2d23b00465da4008fd8f08d83587\": container with ID starting with cecc64cca52be72b9ea23c344fb7d680a37d2d23b00465da4008fd8f08d83587 not found: ID does not exist"
Oct 30 06:57:46.689727 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.689658    6224 scope.go:122] "RemoveContainer" containerID="ed4a02af2dab9b59fe33955fd09ad859f10218c846369b3f974b28f274e9e777"
Oct 30 06:57:46.689888 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.689873    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed4a02af2dab9b59fe33955fd09ad859f10218c846369b3f974b28f274e9e777"} err="failed to get container status \"ed4a02af2dab9b59fe33955fd09ad859f10218c846369b3f974b28f274e9e777\": rpc error: code = NotFound desc = could not find container \"ed4a02af2dab9b59fe33955fd09ad859f10218c846369b3f974b28f274e9e777\": container with ID starting with ed4a02af2dab9b59fe33955fd09ad859f10218c846369b3f974b28f274e9e777 not found: ID does not exist"
Oct 30 06:57:46.689938 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.689887    6224 scope.go:122] "RemoveContainer" containerID="88c9314ef41c71965d607688400aa8998adb43abe9255652d83e65e5e9d5e549"
Oct 30 06:57:46.690107 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.690089    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88c9314ef41c71965d607688400aa8998adb43abe9255652d83e65e5e9d5e549"} err="failed to get container status \"88c9314ef41c71965d607688400aa8998adb43abe9255652d83e65e5e9d5e549\": rpc error: code = NotFound desc = could not find container \"88c9314ef41c71965d607688400aa8998adb43abe9255652d83e65e5e9d5e549\": container with ID starting with 88c9314ef41c71965d607688400aa8998adb43abe9255652d83e65e5e9d5e549 not found: ID does not exist"
Oct 30 06:57:46.690172 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.690108    6224 scope.go:122] "RemoveContainer" containerID="26f5047a4a9d0850c61a1f7288ba0f74492952019ef4ac4beaec3e9e53c98040"
Oct 30 06:57:46.690335 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.690318    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26f5047a4a9d0850c61a1f7288ba0f74492952019ef4ac4beaec3e9e53c98040"} err="failed to get container status \"26f5047a4a9d0850c61a1f7288ba0f74492952019ef4ac4beaec3e9e53c98040\": rpc error: code = NotFound desc = could not find container \"26f5047a4a9d0850c61a1f7288ba0f74492952019ef4ac4beaec3e9e53c98040\": container with ID starting with 26f5047a4a9d0850c61a1f7288ba0f74492952019ef4ac4beaec3e9e53c98040 not found: ID does not exist"
Oct 30 06:57:46.690393 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.690335    6224 scope.go:122] "RemoveContainer" containerID="36e72f7afaaed1114f7178cf75aadb86bb06d852fa4d40991839fe3c650e7d00"
Oct 30 06:57:46.690578 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.690559    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36e72f7afaaed1114f7178cf75aadb86bb06d852fa4d40991839fe3c650e7d00"} err="failed to get container status \"36e72f7afaaed1114f7178cf75aadb86bb06d852fa4d40991839fe3c650e7d00\": rpc error: code = NotFound desc = could not find container \"36e72f7afaaed1114f7178cf75aadb86bb06d852fa4d40991839fe3c650e7d00\": container with ID starting with 36e72f7afaaed1114f7178cf75aadb86bb06d852fa4d40991839fe3c650e7d00 not found: ID does not exist"
Oct 30 06:57:46.690655 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.690580    6224 scope.go:122] "RemoveContainer" containerID="7ea8feb9571f349fc2119c3f7c6e1421fe27e22e55f90a4b5370acc7af3d92bb"
Oct 30 06:57:46.690899 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.690878    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ea8feb9571f349fc2119c3f7c6e1421fe27e22e55f90a4b5370acc7af3d92bb"} err="failed to get container status \"7ea8feb9571f349fc2119c3f7c6e1421fe27e22e55f90a4b5370acc7af3d92bb\": rpc error: code = NotFound desc = could not find container \"7ea8feb9571f349fc2119c3f7c6e1421fe27e22e55f90a4b5370acc7af3d92bb\": container with ID starting with 7ea8feb9571f349fc2119c3f7c6e1421fe27e22e55f90a4b5370acc7af3d92bb not found: ID does not exist"
Oct 30 06:57:46.690964 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.690900    6224 scope.go:122] "RemoveContainer" containerID="82d8cccc3e759d6d5db1f4f50f4fdc6bb433649ad57238011dd1b6f318c733b7"
Oct 30 06:57:46.691164 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.691147    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82d8cccc3e759d6d5db1f4f50f4fdc6bb433649ad57238011dd1b6f318c733b7"} err="failed to get container status \"82d8cccc3e759d6d5db1f4f50f4fdc6bb433649ad57238011dd1b6f318c733b7\": rpc error: code = NotFound desc = could not find container \"82d8cccc3e759d6d5db1f4f50f4fdc6bb433649ad57238011dd1b6f318c733b7\": container with ID starting with 82d8cccc3e759d6d5db1f4f50f4fdc6bb433649ad57238011dd1b6f318c733b7 not found: ID does not exist"
Oct 30 06:57:46.691239 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.691165    6224 scope.go:122] "RemoveContainer" containerID="cecc64cca52be72b9ea23c344fb7d680a37d2d23b00465da4008fd8f08d83587"
Oct 30 06:57:46.691400 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.691382    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cecc64cca52be72b9ea23c344fb7d680a37d2d23b00465da4008fd8f08d83587"} err="failed to get container status \"cecc64cca52be72b9ea23c344fb7d680a37d2d23b00465da4008fd8f08d83587\": rpc error: code = NotFound desc = could not find container \"cecc64cca52be72b9ea23c344fb7d680a37d2d23b00465da4008fd8f08d83587\": container with ID starting with cecc64cca52be72b9ea23c344fb7d680a37d2d23b00465da4008fd8f08d83587 not found: ID does not exist"
Oct 30 06:57:46.691472 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.691401    6224 scope.go:122] "RemoveContainer" containerID="ed4a02af2dab9b59fe33955fd09ad859f10218c846369b3f974b28f274e9e777"
Oct 30 06:57:46.691627 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.691606    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed4a02af2dab9b59fe33955fd09ad859f10218c846369b3f974b28f274e9e777"} err="failed to get container status \"ed4a02af2dab9b59fe33955fd09ad859f10218c846369b3f974b28f274e9e777\": rpc error: code = NotFound desc = could not find container \"ed4a02af2dab9b59fe33955fd09ad859f10218c846369b3f974b28f274e9e777\": container with ID starting with ed4a02af2dab9b59fe33955fd09ad859f10218c846369b3f974b28f274e9e777 not found: ID does not exist"
Oct 30 06:57:46.691627 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.691623    6224 scope.go:122] "RemoveContainer" containerID="88c9314ef41c71965d607688400aa8998adb43abe9255652d83e65e5e9d5e549"
Oct 30 06:57:46.691957 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.691941    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88c9314ef41c71965d607688400aa8998adb43abe9255652d83e65e5e9d5e549"} err="failed to get container status \"88c9314ef41c71965d607688400aa8998adb43abe9255652d83e65e5e9d5e549\": rpc error: code = NotFound desc = could not find container \"88c9314ef41c71965d607688400aa8998adb43abe9255652d83e65e5e9d5e549\": container with ID starting with 88c9314ef41c71965d607688400aa8998adb43abe9255652d83e65e5e9d5e549 not found: ID does not exist"
Oct 30 06:57:46.691957 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.691957    6224 scope.go:122] "RemoveContainer" containerID="26f5047a4a9d0850c61a1f7288ba0f74492952019ef4ac4beaec3e9e53c98040"
Oct 30 06:57:46.692185 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.692166    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26f5047a4a9d0850c61a1f7288ba0f74492952019ef4ac4beaec3e9e53c98040"} err="failed to get container status \"26f5047a4a9d0850c61a1f7288ba0f74492952019ef4ac4beaec3e9e53c98040\": rpc error: code = NotFound desc = could not find container \"26f5047a4a9d0850c61a1f7288ba0f74492952019ef4ac4beaec3e9e53c98040\": container with ID starting with 26f5047a4a9d0850c61a1f7288ba0f74492952019ef4ac4beaec3e9e53c98040 not found: ID does not exist"
Oct 30 06:57:46.692230 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.692184    6224 scope.go:122] "RemoveContainer" containerID="36e72f7afaaed1114f7178cf75aadb86bb06d852fa4d40991839fe3c650e7d00"
Oct 30 06:57:46.692395 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.692381    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36e72f7afaaed1114f7178cf75aadb86bb06d852fa4d40991839fe3c650e7d00"} err="failed to get container status \"36e72f7afaaed1114f7178cf75aadb86bb06d852fa4d40991839fe3c650e7d00\": rpc error: code = NotFound desc = could not find container \"36e72f7afaaed1114f7178cf75aadb86bb06d852fa4d40991839fe3c650e7d00\": container with ID starting with 36e72f7afaaed1114f7178cf75aadb86bb06d852fa4d40991839fe3c650e7d00 not found: ID does not exist"
Oct 30 06:57:46.692446 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.692395    6224 scope.go:122] "RemoveContainer" containerID="7ea8feb9571f349fc2119c3f7c6e1421fe27e22e55f90a4b5370acc7af3d92bb"
Oct 30 06:57:46.692608 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.692590    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ea8feb9571f349fc2119c3f7c6e1421fe27e22e55f90a4b5370acc7af3d92bb"} err="failed to get container status \"7ea8feb9571f349fc2119c3f7c6e1421fe27e22e55f90a4b5370acc7af3d92bb\": rpc error: code = NotFound desc = could not find container \"7ea8feb9571f349fc2119c3f7c6e1421fe27e22e55f90a4b5370acc7af3d92bb\": container with ID starting with 7ea8feb9571f349fc2119c3f7c6e1421fe27e22e55f90a4b5370acc7af3d92bb not found: ID does not exist"
Oct 30 06:57:46.692654 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.692609    6224 scope.go:122] "RemoveContainer" containerID="82d8cccc3e759d6d5db1f4f50f4fdc6bb433649ad57238011dd1b6f318c733b7"
Oct 30 06:57:46.692835 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.692817    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82d8cccc3e759d6d5db1f4f50f4fdc6bb433649ad57238011dd1b6f318c733b7"} err="failed to get container status \"82d8cccc3e759d6d5db1f4f50f4fdc6bb433649ad57238011dd1b6f318c733b7\": rpc error: code = NotFound desc = could not find container \"82d8cccc3e759d6d5db1f4f50f4fdc6bb433649ad57238011dd1b6f318c733b7\": container with ID starting with 82d8cccc3e759d6d5db1f4f50f4fdc6bb433649ad57238011dd1b6f318c733b7 not found: ID does not exist"
Oct 30 06:57:46.692892 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.692835    6224 scope.go:122] "RemoveContainer" containerID="cecc64cca52be72b9ea23c344fb7d680a37d2d23b00465da4008fd8f08d83587"
Oct 30 06:57:46.693035 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.693017    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cecc64cca52be72b9ea23c344fb7d680a37d2d23b00465da4008fd8f08d83587"} err="failed to get container status \"cecc64cca52be72b9ea23c344fb7d680a37d2d23b00465da4008fd8f08d83587\": rpc error: code = NotFound desc = could not find container \"cecc64cca52be72b9ea23c344fb7d680a37d2d23b00465da4008fd8f08d83587\": container with ID starting with cecc64cca52be72b9ea23c344fb7d680a37d2d23b00465da4008fd8f08d83587 not found: ID does not exist"
Oct 30 06:57:46.693084 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.693037    6224 scope.go:122] "RemoveContainer" containerID="ed4a02af2dab9b59fe33955fd09ad859f10218c846369b3f974b28f274e9e777"
Oct 30 06:57:46.693276 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.693257    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed4a02af2dab9b59fe33955fd09ad859f10218c846369b3f974b28f274e9e777"} err="failed to get container status \"ed4a02af2dab9b59fe33955fd09ad859f10218c846369b3f974b28f274e9e777\": rpc error: code = NotFound desc = could not find container \"ed4a02af2dab9b59fe33955fd09ad859f10218c846369b3f974b28f274e9e777\": container with ID starting with ed4a02af2dab9b59fe33955fd09ad859f10218c846369b3f974b28f274e9e777 not found: ID does not exist"
Oct 30 06:57:46.693346 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.693287    6224 scope.go:122] "RemoveContainer" containerID="88c9314ef41c71965d607688400aa8998adb43abe9255652d83e65e5e9d5e549"
Oct 30 06:57:46.693497 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.693481    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88c9314ef41c71965d607688400aa8998adb43abe9255652d83e65e5e9d5e549"} err="failed to get container status \"88c9314ef41c71965d607688400aa8998adb43abe9255652d83e65e5e9d5e549\": rpc error: code = NotFound desc = could not find container \"88c9314ef41c71965d607688400aa8998adb43abe9255652d83e65e5e9d5e549\": container with ID starting with 88c9314ef41c71965d607688400aa8998adb43abe9255652d83e65e5e9d5e549 not found: ID does not exist"
Oct 30 06:57:46.693539 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.693498    6224 scope.go:122] "RemoveContainer" containerID="26f5047a4a9d0850c61a1f7288ba0f74492952019ef4ac4beaec3e9e53c98040"
Oct 30 06:57:46.693739 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.693722    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26f5047a4a9d0850c61a1f7288ba0f74492952019ef4ac4beaec3e9e53c98040"} err="failed to get container status \"26f5047a4a9d0850c61a1f7288ba0f74492952019ef4ac4beaec3e9e53c98040\": rpc error: code = NotFound desc = could not find container \"26f5047a4a9d0850c61a1f7288ba0f74492952019ef4ac4beaec3e9e53c98040\": container with ID starting with 26f5047a4a9d0850c61a1f7288ba0f74492952019ef4ac4beaec3e9e53c98040 not found: ID does not exist"
Oct 30 06:57:46.693793 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.693739    6224 scope.go:122] "RemoveContainer" containerID="36e72f7afaaed1114f7178cf75aadb86bb06d852fa4d40991839fe3c650e7d00"
Oct 30 06:57:46.693962 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.693941    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36e72f7afaaed1114f7178cf75aadb86bb06d852fa4d40991839fe3c650e7d00"} err="failed to get container status \"36e72f7afaaed1114f7178cf75aadb86bb06d852fa4d40991839fe3c650e7d00\": rpc error: code = NotFound desc = could not find container \"36e72f7afaaed1114f7178cf75aadb86bb06d852fa4d40991839fe3c650e7d00\": container with ID starting with 36e72f7afaaed1114f7178cf75aadb86bb06d852fa4d40991839fe3c650e7d00 not found: ID does not exist"
Oct 30 06:57:46.693962 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.693961    6224 scope.go:122] "RemoveContainer" containerID="7ea8feb9571f349fc2119c3f7c6e1421fe27e22e55f90a4b5370acc7af3d92bb"
Oct 30 06:57:46.694187 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.694160    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ea8feb9571f349fc2119c3f7c6e1421fe27e22e55f90a4b5370acc7af3d92bb"} err="failed to get container status \"7ea8feb9571f349fc2119c3f7c6e1421fe27e22e55f90a4b5370acc7af3d92bb\": rpc error: code = NotFound desc = could not find container \"7ea8feb9571f349fc2119c3f7c6e1421fe27e22e55f90a4b5370acc7af3d92bb\": container with ID starting with 7ea8feb9571f349fc2119c3f7c6e1421fe27e22e55f90a4b5370acc7af3d92bb not found: ID does not exist"
Oct 30 06:57:46.694259 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.694187    6224 scope.go:122] "RemoveContainer" containerID="82d8cccc3e759d6d5db1f4f50f4fdc6bb433649ad57238011dd1b6f318c733b7"
Oct 30 06:57:46.694439 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.694420    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82d8cccc3e759d6d5db1f4f50f4fdc6bb433649ad57238011dd1b6f318c733b7"} err="failed to get container status \"82d8cccc3e759d6d5db1f4f50f4fdc6bb433649ad57238011dd1b6f318c733b7\": rpc error: code = NotFound desc = could not find container \"82d8cccc3e759d6d5db1f4f50f4fdc6bb433649ad57238011dd1b6f318c733b7\": container with ID starting with 82d8cccc3e759d6d5db1f4f50f4fdc6bb433649ad57238011dd1b6f318c733b7 not found: ID does not exist"
Oct 30 06:57:46.694480 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.694441    6224 scope.go:122] "RemoveContainer" containerID="cecc64cca52be72b9ea23c344fb7d680a37d2d23b00465da4008fd8f08d83587"
Oct 30 06:57:46.694666 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.694650    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cecc64cca52be72b9ea23c344fb7d680a37d2d23b00465da4008fd8f08d83587"} err="failed to get container status \"cecc64cca52be72b9ea23c344fb7d680a37d2d23b00465da4008fd8f08d83587\": rpc error: code = NotFound desc = could not find container \"cecc64cca52be72b9ea23c344fb7d680a37d2d23b00465da4008fd8f08d83587\": container with ID starting with cecc64cca52be72b9ea23c344fb7d680a37d2d23b00465da4008fd8f08d83587 not found: ID does not exist"
Oct 30 06:57:46.694701 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.694668    6224 scope.go:122] "RemoveContainer" containerID="ed4a02af2dab9b59fe33955fd09ad859f10218c846369b3f974b28f274e9e777"
Oct 30 06:57:46.694923 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.694904    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed4a02af2dab9b59fe33955fd09ad859f10218c846369b3f974b28f274e9e777"} err="failed to get container status \"ed4a02af2dab9b59fe33955fd09ad859f10218c846369b3f974b28f274e9e777\": rpc error: code = NotFound desc = could not find container \"ed4a02af2dab9b59fe33955fd09ad859f10218c846369b3f974b28f274e9e777\": container with ID starting with ed4a02af2dab9b59fe33955fd09ad859f10218c846369b3f974b28f274e9e777 not found: ID does not exist"
Oct 30 06:57:46.694982 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.694924    6224 scope.go:122] "RemoveContainer" containerID="88c9314ef41c71965d607688400aa8998adb43abe9255652d83e65e5e9d5e549"
Oct 30 06:57:46.695143 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.695126    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88c9314ef41c71965d607688400aa8998adb43abe9255652d83e65e5e9d5e549"} err="failed to get container status \"88c9314ef41c71965d607688400aa8998adb43abe9255652d83e65e5e9d5e549\": rpc error: code = NotFound desc = could not find container \"88c9314ef41c71965d607688400aa8998adb43abe9255652d83e65e5e9d5e549\": container with ID starting with 88c9314ef41c71965d607688400aa8998adb43abe9255652d83e65e5e9d5e549 not found: ID does not exist"
Oct 30 06:57:46.695189 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.695144    6224 scope.go:122] "RemoveContainer" containerID="26f5047a4a9d0850c61a1f7288ba0f74492952019ef4ac4beaec3e9e53c98040"
Oct 30 06:57:46.695391 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.695373    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26f5047a4a9d0850c61a1f7288ba0f74492952019ef4ac4beaec3e9e53c98040"} err="failed to get container status \"26f5047a4a9d0850c61a1f7288ba0f74492952019ef4ac4beaec3e9e53c98040\": rpc error: code = NotFound desc = could not find container \"26f5047a4a9d0850c61a1f7288ba0f74492952019ef4ac4beaec3e9e53c98040\": container with ID starting with 26f5047a4a9d0850c61a1f7288ba0f74492952019ef4ac4beaec3e9e53c98040 not found: ID does not exist"
Oct 30 06:57:46.695448 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.695393    6224 scope.go:122] "RemoveContainer" containerID="36e72f7afaaed1114f7178cf75aadb86bb06d852fa4d40991839fe3c650e7d00"
Oct 30 06:57:46.695604 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.695588    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36e72f7afaaed1114f7178cf75aadb86bb06d852fa4d40991839fe3c650e7d00"} err="failed to get container status \"36e72f7afaaed1114f7178cf75aadb86bb06d852fa4d40991839fe3c650e7d00\": rpc error: code = NotFound desc = could not find container \"36e72f7afaaed1114f7178cf75aadb86bb06d852fa4d40991839fe3c650e7d00\": container with ID starting with 36e72f7afaaed1114f7178cf75aadb86bb06d852fa4d40991839fe3c650e7d00 not found: ID does not exist"
Oct 30 06:57:46.695643 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.695603    6224 scope.go:122] "RemoveContainer" containerID="7ea8feb9571f349fc2119c3f7c6e1421fe27e22e55f90a4b5370acc7af3d92bb"
Oct 30 06:57:46.695842 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.695822    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ea8feb9571f349fc2119c3f7c6e1421fe27e22e55f90a4b5370acc7af3d92bb"} err="failed to get container status \"7ea8feb9571f349fc2119c3f7c6e1421fe27e22e55f90a4b5370acc7af3d92bb\": rpc error: code = NotFound desc = could not find container \"7ea8feb9571f349fc2119c3f7c6e1421fe27e22e55f90a4b5370acc7af3d92bb\": container with ID starting with 7ea8feb9571f349fc2119c3f7c6e1421fe27e22e55f90a4b5370acc7af3d92bb not found: ID does not exist"
Oct 30 06:57:46.695884 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.695843    6224 scope.go:122] "RemoveContainer" containerID="82d8cccc3e759d6d5db1f4f50f4fdc6bb433649ad57238011dd1b6f318c733b7"
Oct 30 06:57:46.696062 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.696046    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82d8cccc3e759d6d5db1f4f50f4fdc6bb433649ad57238011dd1b6f318c733b7"} err="failed to get container status \"82d8cccc3e759d6d5db1f4f50f4fdc6bb433649ad57238011dd1b6f318c733b7\": rpc error: code = NotFound desc = could not find container \"82d8cccc3e759d6d5db1f4f50f4fdc6bb433649ad57238011dd1b6f318c733b7\": container with ID starting with 82d8cccc3e759d6d5db1f4f50f4fdc6bb433649ad57238011dd1b6f318c733b7 not found: ID does not exist"
Oct 30 06:57:46.696100 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.696063    6224 scope.go:122] "RemoveContainer" containerID="cecc64cca52be72b9ea23c344fb7d680a37d2d23b00465da4008fd8f08d83587"
Oct 30 06:57:46.696266 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.696250    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cecc64cca52be72b9ea23c344fb7d680a37d2d23b00465da4008fd8f08d83587"} err="failed to get container status \"cecc64cca52be72b9ea23c344fb7d680a37d2d23b00465da4008fd8f08d83587\": rpc error: code = NotFound desc = could not find container \"cecc64cca52be72b9ea23c344fb7d680a37d2d23b00465da4008fd8f08d83587\": container with ID starting with cecc64cca52be72b9ea23c344fb7d680a37d2d23b00465da4008fd8f08d83587 not found: ID does not exist"
Oct 30 06:57:46.696308 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.696267    6224 scope.go:122] "RemoveContainer" containerID="ed4a02af2dab9b59fe33955fd09ad859f10218c846369b3f974b28f274e9e777"
Oct 30 06:57:46.696481 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.696463    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed4a02af2dab9b59fe33955fd09ad859f10218c846369b3f974b28f274e9e777"} err="failed to get container status \"ed4a02af2dab9b59fe33955fd09ad859f10218c846369b3f974b28f274e9e777\": rpc error: code = NotFound desc = could not find container \"ed4a02af2dab9b59fe33955fd09ad859f10218c846369b3f974b28f274e9e777\": container with ID starting with ed4a02af2dab9b59fe33955fd09ad859f10218c846369b3f974b28f274e9e777 not found: ID does not exist"
Oct 30 06:57:46.696517 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.696482    6224 scope.go:122] "RemoveContainer" containerID="88c9314ef41c71965d607688400aa8998adb43abe9255652d83e65e5e9d5e549"
Oct 30 06:57:46.696705 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.696685    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88c9314ef41c71965d607688400aa8998adb43abe9255652d83e65e5e9d5e549"} err="failed to get container status \"88c9314ef41c71965d607688400aa8998adb43abe9255652d83e65e5e9d5e549\": rpc error: code = NotFound desc = could not find container \"88c9314ef41c71965d607688400aa8998adb43abe9255652d83e65e5e9d5e549\": container with ID starting with 88c9314ef41c71965d607688400aa8998adb43abe9255652d83e65e5e9d5e549 not found: ID does not exist"
Oct 30 06:57:46.696779 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.696707    6224 scope.go:122] "RemoveContainer" containerID="26f5047a4a9d0850c61a1f7288ba0f74492952019ef4ac4beaec3e9e53c98040"
Oct 30 06:57:46.696961 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.696945    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26f5047a4a9d0850c61a1f7288ba0f74492952019ef4ac4beaec3e9e53c98040"} err="failed to get container status \"26f5047a4a9d0850c61a1f7288ba0f74492952019ef4ac4beaec3e9e53c98040\": rpc error: code = NotFound desc = could not find container \"26f5047a4a9d0850c61a1f7288ba0f74492952019ef4ac4beaec3e9e53c98040\": container with ID starting with 26f5047a4a9d0850c61a1f7288ba0f74492952019ef4ac4beaec3e9e53c98040 not found: ID does not exist"
Oct 30 06:57:46.697031 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.696962    6224 scope.go:122] "RemoveContainer" containerID="36e72f7afaaed1114f7178cf75aadb86bb06d852fa4d40991839fe3c650e7d00"
Oct 30 06:57:46.697177 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.697159    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36e72f7afaaed1114f7178cf75aadb86bb06d852fa4d40991839fe3c650e7d00"} err="failed to get container status \"36e72f7afaaed1114f7178cf75aadb86bb06d852fa4d40991839fe3c650e7d00\": rpc error: code = NotFound desc = could not find container \"36e72f7afaaed1114f7178cf75aadb86bb06d852fa4d40991839fe3c650e7d00\": container with ID starting with 36e72f7afaaed1114f7178cf75aadb86bb06d852fa4d40991839fe3c650e7d00 not found: ID does not exist"
Oct 30 06:57:46.697233 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.697185    6224 scope.go:122] "RemoveContainer" containerID="7ea8feb9571f349fc2119c3f7c6e1421fe27e22e55f90a4b5370acc7af3d92bb"
Oct 30 06:57:46.697390 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.697373    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ea8feb9571f349fc2119c3f7c6e1421fe27e22e55f90a4b5370acc7af3d92bb"} err="failed to get container status \"7ea8feb9571f349fc2119c3f7c6e1421fe27e22e55f90a4b5370acc7af3d92bb\": rpc error: code = NotFound desc = could not find container \"7ea8feb9571f349fc2119c3f7c6e1421fe27e22e55f90a4b5370acc7af3d92bb\": container with ID starting with 7ea8feb9571f349fc2119c3f7c6e1421fe27e22e55f90a4b5370acc7af3d92bb not found: ID does not exist"
Oct 30 06:57:46.715117 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.715094    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 06:57:46.729036 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.729014    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 06:57:46.754914 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.754893    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 06:57:46.755129 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.755118    6224 cpu_manager.go:414] "containerMap: removing container" podUID="12c298a7-e78a-45d7-abe1-7b7ef6f5df03" containerName="config-reloader"
Oct 30 06:57:46.755170 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.755130    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12c298a7-e78a-45d7-abe1-7b7ef6f5df03" containerName="config-reloader"
Oct 30 06:57:46.755170 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.755141    6224 cpu_manager.go:414] "containerMap: removing container" podUID="12c298a7-e78a-45d7-abe1-7b7ef6f5df03" containerName="kube-rbac-proxy-web"
Oct 30 06:57:46.755170 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.755146    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12c298a7-e78a-45d7-abe1-7b7ef6f5df03" containerName="kube-rbac-proxy-web"
Oct 30 06:57:46.755170 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.755158    6224 cpu_manager.go:414] "containerMap: removing container" podUID="12c298a7-e78a-45d7-abe1-7b7ef6f5df03" containerName="init-config-reloader"
Oct 30 06:57:46.755170 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.755163    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12c298a7-e78a-45d7-abe1-7b7ef6f5df03" containerName="init-config-reloader"
Oct 30 06:57:46.755170 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.755169    6224 cpu_manager.go:414] "containerMap: removing container" podUID="12c298a7-e78a-45d7-abe1-7b7ef6f5df03" containerName="kube-rbac-proxy"
Oct 30 06:57:46.755335 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.755174    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12c298a7-e78a-45d7-abe1-7b7ef6f5df03" containerName="kube-rbac-proxy"
Oct 30 06:57:46.755335 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.755180    6224 cpu_manager.go:414] "containerMap: removing container" podUID="12c298a7-e78a-45d7-abe1-7b7ef6f5df03" containerName="thanos-sidecar"
Oct 30 06:57:46.755335 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.755184    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12c298a7-e78a-45d7-abe1-7b7ef6f5df03" containerName="thanos-sidecar"
Oct 30 06:57:46.755335 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.755189    6224 cpu_manager.go:414] "containerMap: removing container" podUID="12c298a7-e78a-45d7-abe1-7b7ef6f5df03" containerName="prometheus"
Oct 30 06:57:46.755335 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.755194    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12c298a7-e78a-45d7-abe1-7b7ef6f5df03" containerName="prometheus"
Oct 30 06:57:46.755335 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.755200    6224 cpu_manager.go:414] "containerMap: removing container" podUID="12c298a7-e78a-45d7-abe1-7b7ef6f5df03" containerName="kube-rbac-proxy-thanos"
Oct 30 06:57:46.755335 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.755204    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12c298a7-e78a-45d7-abe1-7b7ef6f5df03" containerName="kube-rbac-proxy-thanos"
Oct 30 06:57:46.755335 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.755240    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="12c298a7-e78a-45d7-abe1-7b7ef6f5df03" containerName="config-reloader"
Oct 30 06:57:46.755335 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.755246    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="12c298a7-e78a-45d7-abe1-7b7ef6f5df03" containerName="kube-rbac-proxy-web"
Oct 30 06:57:46.755335 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.755254    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="12c298a7-e78a-45d7-abe1-7b7ef6f5df03" containerName="init-config-reloader"
Oct 30 06:57:46.755335 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.755259    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="12c298a7-e78a-45d7-abe1-7b7ef6f5df03" containerName="kube-rbac-proxy"
Oct 30 06:57:46.755335 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.755264    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="12c298a7-e78a-45d7-abe1-7b7ef6f5df03" containerName="thanos-sidecar"
Oct 30 06:57:46.755335 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.755269    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="12c298a7-e78a-45d7-abe1-7b7ef6f5df03" containerName="prometheus"
Oct 30 06:57:46.755335 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.755274    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="12c298a7-e78a-45d7-abe1-7b7ef6f5df03" containerName="kube-rbac-proxy-thanos"
Oct 30 06:57:46.787539 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.787519    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:46.787843 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.787821    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 06:57:46.805174 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.805155    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 06:57:46.805317 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.805298    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 06:57:46.805385 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.805308    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 06:57:46.805385 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.805331    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 06:57:46.805385 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.805350    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 06:57:46.805501 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.805420    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 06:57:46.805610 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.805595    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-6rnqp\""
Oct 30 06:57:46.805667 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.805624    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 06:57:46.805667 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.805625    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 06:57:46.805667 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.805640    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-e9mv09h28asd\""
Oct 30 06:57:46.805667 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.805664    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 06:57:46.805832 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.805733    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 06:57:46.805895 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.805868    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 06:57:46.805895 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.805882    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 06:57:46.820559 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.820541    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 06:57:46.820638 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.820565    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 06:57:46.825705 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.825685    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3054a645-d5e5-4c52-b795-5e278e2d42d6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3054a645-d5e5-4c52-b795-5e278e2d42d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:46.826056 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.825716    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3054a645-d5e5-4c52-b795-5e278e2d42d6-web-config\") pod \"prometheus-k8s-0\" (UID: \"3054a645-d5e5-4c52-b795-5e278e2d42d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:46.826056 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.825744    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3054a645-d5e5-4c52-b795-5e278e2d42d6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3054a645-d5e5-4c52-b795-5e278e2d42d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:46.826056 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.825886    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbd54\" (UniqueName: \"kubernetes.io/projected/3054a645-d5e5-4c52-b795-5e278e2d42d6-kube-api-access-vbd54\") pod \"prometheus-k8s-0\" (UID: \"3054a645-d5e5-4c52-b795-5e278e2d42d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:46.826056 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.825912    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3054a645-d5e5-4c52-b795-5e278e2d42d6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3054a645-d5e5-4c52-b795-5e278e2d42d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:46.826056 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.825942    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3054a645-d5e5-4c52-b795-5e278e2d42d6-config-out\") pod \"prometheus-k8s-0\" (UID: \"3054a645-d5e5-4c52-b795-5e278e2d42d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:46.826056 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.826014    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3054a645-d5e5-4c52-b795-5e278e2d42d6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3054a645-d5e5-4c52-b795-5e278e2d42d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:46.826286 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.826067    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3054a645-d5e5-4c52-b795-5e278e2d42d6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3054a645-d5e5-4c52-b795-5e278e2d42d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:46.826286 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.826095    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3054a645-d5e5-4c52-b795-5e278e2d42d6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3054a645-d5e5-4c52-b795-5e278e2d42d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:46.826286 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.826117    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3054a645-d5e5-4c52-b795-5e278e2d42d6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3054a645-d5e5-4c52-b795-5e278e2d42d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:46.826286 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.826132    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3054a645-d5e5-4c52-b795-5e278e2d42d6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3054a645-d5e5-4c52-b795-5e278e2d42d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:46.826286 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.826181    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3054a645-d5e5-4c52-b795-5e278e2d42d6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3054a645-d5e5-4c52-b795-5e278e2d42d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:46.826286 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.826243    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3054a645-d5e5-4c52-b795-5e278e2d42d6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3054a645-d5e5-4c52-b795-5e278e2d42d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:46.826286 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.826257    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3054a645-d5e5-4c52-b795-5e278e2d42d6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3054a645-d5e5-4c52-b795-5e278e2d42d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:46.826286 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.826273    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3054a645-d5e5-4c52-b795-5e278e2d42d6-config\") pod \"prometheus-k8s-0\" (UID: \"3054a645-d5e5-4c52-b795-5e278e2d42d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:46.826574 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.826290    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3054a645-d5e5-4c52-b795-5e278e2d42d6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3054a645-d5e5-4c52-b795-5e278e2d42d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:46.826574 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.826331    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3054a645-d5e5-4c52-b795-5e278e2d42d6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3054a645-d5e5-4c52-b795-5e278e2d42d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:46.826574 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.826376    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3054a645-d5e5-4c52-b795-5e278e2d42d6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3054a645-d5e5-4c52-b795-5e278e2d42d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:46.826574 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.826427    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3054a645-d5e5-4c52-b795-5e278e2d42d6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3054a645-d5e5-4c52-b795-5e278e2d42d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:46.826574 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.826450    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3054a645-d5e5-4c52-b795-5e278e2d42d6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3054a645-d5e5-4c52-b795-5e278e2d42d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:46.927557 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.927492    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3054a645-d5e5-4c52-b795-5e278e2d42d6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3054a645-d5e5-4c52-b795-5e278e2d42d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:46.927557 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.927527    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3054a645-d5e5-4c52-b795-5e278e2d42d6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3054a645-d5e5-4c52-b795-5e278e2d42d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:46.927739 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.927558    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3054a645-d5e5-4c52-b795-5e278e2d42d6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3054a645-d5e5-4c52-b795-5e278e2d42d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:46.927739 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.927665    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3054a645-d5e5-4c52-b795-5e278e2d42d6-web-config\") pod \"prometheus-k8s-0\" (UID: \"3054a645-d5e5-4c52-b795-5e278e2d42d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:46.927739 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.927697    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3054a645-d5e5-4c52-b795-5e278e2d42d6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3054a645-d5e5-4c52-b795-5e278e2d42d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:46.927739 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.927721    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbd54\" (UniqueName: \"kubernetes.io/projected/3054a645-d5e5-4c52-b795-5e278e2d42d6-kube-api-access-vbd54\") pod \"prometheus-k8s-0\" (UID: \"3054a645-d5e5-4c52-b795-5e278e2d42d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:46.927739 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.927738    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3054a645-d5e5-4c52-b795-5e278e2d42d6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3054a645-d5e5-4c52-b795-5e278e2d42d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:46.928025 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.927769    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3054a645-d5e5-4c52-b795-5e278e2d42d6-config-out\") pod \"prometheus-k8s-0\" (UID: \"3054a645-d5e5-4c52-b795-5e278e2d42d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:46.928025 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.927799    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3054a645-d5e5-4c52-b795-5e278e2d42d6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3054a645-d5e5-4c52-b795-5e278e2d42d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:46.928025 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.927865    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3054a645-d5e5-4c52-b795-5e278e2d42d6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3054a645-d5e5-4c52-b795-5e278e2d42d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:46.928025 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.927898    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3054a645-d5e5-4c52-b795-5e278e2d42d6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3054a645-d5e5-4c52-b795-5e278e2d42d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:46.928025 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.927922    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3054a645-d5e5-4c52-b795-5e278e2d42d6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3054a645-d5e5-4c52-b795-5e278e2d42d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:46.928025 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.927945    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3054a645-d5e5-4c52-b795-5e278e2d42d6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3054a645-d5e5-4c52-b795-5e278e2d42d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:46.928025 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.927935    6224 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 06:57:46.928025 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.927974    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3054a645-d5e5-4c52-b795-5e278e2d42d6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3054a645-d5e5-4c52-b795-5e278e2d42d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:46.928411 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.928039    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3054a645-d5e5-4c52-b795-5e278e2d42d6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3054a645-d5e5-4c52-b795-5e278e2d42d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:46.928411 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.928087    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3054a645-d5e5-4c52-b795-5e278e2d42d6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3054a645-d5e5-4c52-b795-5e278e2d42d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:46.928527 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.928505    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3054a645-d5e5-4c52-b795-5e278e2d42d6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3054a645-d5e5-4c52-b795-5e278e2d42d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:46.928595 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.928548    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3054a645-d5e5-4c52-b795-5e278e2d42d6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3054a645-d5e5-4c52-b795-5e278e2d42d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:46.929305 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.929278    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3054a645-d5e5-4c52-b795-5e278e2d42d6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3054a645-d5e5-4c52-b795-5e278e2d42d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:46.930789 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.930763    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3054a645-d5e5-4c52-b795-5e278e2d42d6-config-out\") pod \"prometheus-k8s-0\" (UID: \"3054a645-d5e5-4c52-b795-5e278e2d42d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:46.930907 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.930839    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3054a645-d5e5-4c52-b795-5e278e2d42d6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3054a645-d5e5-4c52-b795-5e278e2d42d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:46.930907 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.930874    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3054a645-d5e5-4c52-b795-5e278e2d42d6-config\") pod \"prometheus-k8s-0\" (UID: \"3054a645-d5e5-4c52-b795-5e278e2d42d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:46.931039 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.930905    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3054a645-d5e5-4c52-b795-5e278e2d42d6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3054a645-d5e5-4c52-b795-5e278e2d42d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:46.931039 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.930937    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3054a645-d5e5-4c52-b795-5e278e2d42d6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3054a645-d5e5-4c52-b795-5e278e2d42d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:46.931039 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.930965    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3054a645-d5e5-4c52-b795-5e278e2d42d6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3054a645-d5e5-4c52-b795-5e278e2d42d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:46.931202 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.931177    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3054a645-d5e5-4c52-b795-5e278e2d42d6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3054a645-d5e5-4c52-b795-5e278e2d42d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:46.931314 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.931279    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3054a645-d5e5-4c52-b795-5e278e2d42d6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3054a645-d5e5-4c52-b795-5e278e2d42d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:46.932572 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.932376    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3054a645-d5e5-4c52-b795-5e278e2d42d6-web-config\") pod \"prometheus-k8s-0\" (UID: \"3054a645-d5e5-4c52-b795-5e278e2d42d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:46.932572 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.932417    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3054a645-d5e5-4c52-b795-5e278e2d42d6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3054a645-d5e5-4c52-b795-5e278e2d42d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:46.932572 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.932546    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3054a645-d5e5-4c52-b795-5e278e2d42d6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3054a645-d5e5-4c52-b795-5e278e2d42d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:46.932794 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.932615    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3054a645-d5e5-4c52-b795-5e278e2d42d6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3054a645-d5e5-4c52-b795-5e278e2d42d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:46.932794 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.932699    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3054a645-d5e5-4c52-b795-5e278e2d42d6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3054a645-d5e5-4c52-b795-5e278e2d42d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:46.933629 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.933400    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3054a645-d5e5-4c52-b795-5e278e2d42d6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3054a645-d5e5-4c52-b795-5e278e2d42d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:46.933629 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.933590    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3054a645-d5e5-4c52-b795-5e278e2d42d6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3054a645-d5e5-4c52-b795-5e278e2d42d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:46.934722 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.933933    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3054a645-d5e5-4c52-b795-5e278e2d42d6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3054a645-d5e5-4c52-b795-5e278e2d42d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:46.934722 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.934233    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3054a645-d5e5-4c52-b795-5e278e2d42d6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3054a645-d5e5-4c52-b795-5e278e2d42d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:46.934722 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.934306    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/3054a645-d5e5-4c52-b795-5e278e2d42d6-config\") pod \"prometheus-k8s-0\" (UID: \"3054a645-d5e5-4c52-b795-5e278e2d42d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:46.937296 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.935095    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3054a645-d5e5-4c52-b795-5e278e2d42d6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3054a645-d5e5-4c52-b795-5e278e2d42d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:46.938325 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.938301    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3054a645-d5e5-4c52-b795-5e278e2d42d6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3054a645-d5e5-4c52-b795-5e278e2d42d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:46.953146 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:46.953127    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbd54\" (UniqueName: \"kubernetes.io/projected/3054a645-d5e5-4c52-b795-5e278e2d42d6-kube-api-access-vbd54\") pod \"prometheus-k8s-0\" (UID: \"3054a645-d5e5-4c52-b795-5e278e2d42d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:47.191108 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:47.191041    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:47.502989 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:47.502915    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12c298a7-e78a-45d7-abe1-7b7ef6f5df03" path="/var/lib/kubelet/pods/12c298a7-e78a-45d7-abe1-7b7ef6f5df03/volumes"
Oct 30 06:57:47.544638 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:47.544612    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 06:57:47.556326 ip-10-0-128-125 kubenswrapper[6224]: W1030 06:57:47.556300    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3054a645_d5e5_4c52_b795_5e278e2d42d6.slice/crio-4b70c285291fc2229e913ecee8e00628c25a9deb61039cb85256be57584b3ca3 WatchSource:0}: Error finding container 4b70c285291fc2229e913ecee8e00628c25a9deb61039cb85256be57584b3ca3: Status 404 returned error can't find the container with id 4b70c285291fc2229e913ecee8e00628c25a9deb61039cb85256be57584b3ca3
Oct 30 06:57:47.647280 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:47.647238    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3054a645-d5e5-4c52-b795-5e278e2d42d6","Type":"ContainerStarted","Data":"81490ff8c39380af4113303b21773455bd3aaf2e889a3da1dd0c40dadf1f9e64"}
Oct 30 06:57:47.647401 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:47.647322    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3054a645-d5e5-4c52-b795-5e278e2d42d6","Type":"ContainerStarted","Data":"4b70c285291fc2229e913ecee8e00628c25a9deb61039cb85256be57584b3ca3"}
Oct 30 06:57:48.581704 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:48.581659    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5c7875b857-wv9lf" podUID="9d24d508-ec6f-4165-8394-f17b09134b34" containerName="console" containerID="cri-o://49e9a6769599b7b0985a13e2cb66627a350ba5f98575e7facdc0ab191db03837" gracePeriod=15
Oct 30 06:57:48.652091 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:48.652058    6224 generic.go:356] "Generic (PLEG): container finished" podID="3054a645-d5e5-4c52-b795-5e278e2d42d6" containerID="81490ff8c39380af4113303b21773455bd3aaf2e889a3da1dd0c40dadf1f9e64" exitCode=0
Oct 30 06:57:48.652218 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:48.652150    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3054a645-d5e5-4c52-b795-5e278e2d42d6","Type":"ContainerDied","Data":"81490ff8c39380af4113303b21773455bd3aaf2e889a3da1dd0c40dadf1f9e64"}
Oct 30 06:57:48.837802 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:48.837780    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c7875b857-wv9lf"
Oct 30 06:57:48.950644 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:48.950621    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d24d508-ec6f-4165-8394-f17b09134b34-kube-api-access-x67rl\" (UniqueName: \"kubernetes.io/projected/9d24d508-ec6f-4165-8394-f17b09134b34-kube-api-access-x67rl\") pod \"9d24d508-ec6f-4165-8394-f17b09134b34\" (UID: \"9d24d508-ec6f-4165-8394-f17b09134b34\") "
Oct 30 06:57:48.950773 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:48.950677    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d24d508-ec6f-4165-8394-f17b09134b34-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9d24d508-ec6f-4165-8394-f17b09134b34-console-serving-cert\") pod \"9d24d508-ec6f-4165-8394-f17b09134b34\" (UID: \"9d24d508-ec6f-4165-8394-f17b09134b34\") "
Oct 30 06:57:48.950773 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:48.950736    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d24d508-ec6f-4165-8394-f17b09134b34-tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d24d508-ec6f-4165-8394-f17b09134b34-tmp\") pod \"9d24d508-ec6f-4165-8394-f17b09134b34\" (UID: \"9d24d508-ec6f-4165-8394-f17b09134b34\") "
Oct 30 06:57:48.950773 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:48.950767    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d24d508-ec6f-4165-8394-f17b09134b34-console-config\" (UniqueName: \"kubernetes.io/configmap/9d24d508-ec6f-4165-8394-f17b09134b34-console-config\") pod \"9d24d508-ec6f-4165-8394-f17b09134b34\" (UID: \"9d24d508-ec6f-4165-8394-f17b09134b34\") "
Oct 30 06:57:48.950948 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:48.950798    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d24d508-ec6f-4165-8394-f17b09134b34-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9d24d508-ec6f-4165-8394-f17b09134b34-console-oauth-config\") pod \"9d24d508-ec6f-4165-8394-f17b09134b34\" (UID: \"9d24d508-ec6f-4165-8394-f17b09134b34\") "
Oct 30 06:57:48.950948 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:48.950859    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d24d508-ec6f-4165-8394-f17b09134b34-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9d24d508-ec6f-4165-8394-f17b09134b34-oauth-serving-cert\") pod \"9d24d508-ec6f-4165-8394-f17b09134b34\" (UID: \"9d24d508-ec6f-4165-8394-f17b09134b34\") "
Oct 30 06:57:48.950948 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:48.950909    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d24d508-ec6f-4165-8394-f17b09134b34-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d24d508-ec6f-4165-8394-f17b09134b34-service-ca\") pod \"9d24d508-ec6f-4165-8394-f17b09134b34\" (UID: \"9d24d508-ec6f-4165-8394-f17b09134b34\") "
Oct 30 06:57:48.951561 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:48.951492    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d24d508-ec6f-4165-8394-f17b09134b34-service-ca" pod "9d24d508-ec6f-4165-8394-f17b09134b34" (UID: "9d24d508-ec6f-4165-8394-f17b09134b34"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:57:48.951858 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:48.951823    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d24d508-ec6f-4165-8394-f17b09134b34-tmp" pod "9d24d508-ec6f-4165-8394-f17b09134b34" (UID: "9d24d508-ec6f-4165-8394-f17b09134b34"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:57:48.951945 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:48.951872    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d24d508-ec6f-4165-8394-f17b09134b34-oauth-serving-cert" pod "9d24d508-ec6f-4165-8394-f17b09134b34" (UID: "9d24d508-ec6f-4165-8394-f17b09134b34"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:57:48.951945 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:48.951908    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d24d508-ec6f-4165-8394-f17b09134b34-console-config" pod "9d24d508-ec6f-4165-8394-f17b09134b34" (UID: "9d24d508-ec6f-4165-8394-f17b09134b34"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:57:48.954363 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:48.954298    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d24d508-ec6f-4165-8394-f17b09134b34-console-serving-cert" pod "9d24d508-ec6f-4165-8394-f17b09134b34" (UID: "9d24d508-ec6f-4165-8394-f17b09134b34"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:57:48.954453 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:48.954413    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d24d508-ec6f-4165-8394-f17b09134b34-console-oauth-config" pod "9d24d508-ec6f-4165-8394-f17b09134b34" (UID: "9d24d508-ec6f-4165-8394-f17b09134b34"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:57:48.954754 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:48.954729    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d24d508-ec6f-4165-8394-f17b09134b34-kube-api-access-x67rl" pod "9d24d508-ec6f-4165-8394-f17b09134b34" (UID: "9d24d508-ec6f-4165-8394-f17b09134b34"). InnerVolumeSpecName "kube-api-access-x67rl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:57:49.052399 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:49.052379    6224 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9d24d508-ec6f-4165-8394-f17b09134b34-console-oauth-config\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 06:57:49.052399 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:49.052400    6224 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9d24d508-ec6f-4165-8394-f17b09134b34-oauth-serving-cert\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 06:57:49.052540 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:49.052412    6224 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9d24d508-ec6f-4165-8394-f17b09134b34-service-ca\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 06:57:49.052540 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:49.052425    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x67rl\" (UniqueName: \"kubernetes.io/projected/9d24d508-ec6f-4165-8394-f17b09134b34-kube-api-access-x67rl\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 06:57:49.052540 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:49.052433    6224 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9d24d508-ec6f-4165-8394-f17b09134b34-console-serving-cert\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 06:57:49.052540 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:49.052465    6224 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d24d508-ec6f-4165-8394-f17b09134b34-tmp\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 06:57:49.052540 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:49.052476    6224 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9d24d508-ec6f-4165-8394-f17b09134b34-console-config\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 06:57:49.660920 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:49.660880    6224 generic.go:356] "Generic (PLEG): container finished" podID="9d24d508-ec6f-4165-8394-f17b09134b34" containerID="49e9a6769599b7b0985a13e2cb66627a350ba5f98575e7facdc0ab191db03837" exitCode=0
Oct 30 06:57:49.661351 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:49.660950    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c7875b857-wv9lf"
Oct 30 06:57:49.661351 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:49.660965    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c7875b857-wv9lf" event={"ID":"9d24d508-ec6f-4165-8394-f17b09134b34","Type":"ContainerDied","Data":"49e9a6769599b7b0985a13e2cb66627a350ba5f98575e7facdc0ab191db03837"}
Oct 30 06:57:49.661351 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:49.661051    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c7875b857-wv9lf" event={"ID":"9d24d508-ec6f-4165-8394-f17b09134b34","Type":"ContainerDied","Data":"3af572320fb94341a428cb0d1ff52bf19e935a13aa08043036427d1aed82f43b"}
Oct 30 06:57:49.661351 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:49.661078    6224 scope.go:122] "RemoveContainer" containerID="49e9a6769599b7b0985a13e2cb66627a350ba5f98575e7facdc0ab191db03837"
Oct 30 06:57:49.664601 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:49.664581    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3054a645-d5e5-4c52-b795-5e278e2d42d6","Type":"ContainerStarted","Data":"c433037819662864d8fad0d94a09d1352ec747989dff328f9e09b3fc3bc87fd2"}
Oct 30 06:57:49.664696 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:49.664635    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3054a645-d5e5-4c52-b795-5e278e2d42d6","Type":"ContainerStarted","Data":"ad3b5b4cd24733c35c5b701612e6426bd3f7192cd35050f3dea9d6f65cb7eb0a"}
Oct 30 06:57:49.664696 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:49.664660    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3054a645-d5e5-4c52-b795-5e278e2d42d6","Type":"ContainerStarted","Data":"4bc511bc2fc9a5e6f59b1f29eccecb53cd21897f5897699864e5aeff801fbdd6"}
Oct 30 06:57:49.664696 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:49.664684    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3054a645-d5e5-4c52-b795-5e278e2d42d6","Type":"ContainerStarted","Data":"8f8076496c3d7d4ce015b5351defcd9d91b02bf7d3cf756cbe4c611dab7930d3"}
Oct 30 06:57:49.664826 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:49.664705    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3054a645-d5e5-4c52-b795-5e278e2d42d6","Type":"ContainerStarted","Data":"21f56d9f6ddb16d5ae40a9119b9debe8172a4f6a6157a060fda7fe6ceec3849f"}
Oct 30 06:57:49.664826 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:49.664727    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3054a645-d5e5-4c52-b795-5e278e2d42d6","Type":"ContainerStarted","Data":"3f55174dc526b15505dc1900e8a81bae6a290a505fabd5dc8ecd28bf34653378"}
Oct 30 06:57:49.669134 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:49.669110    6224 scope.go:122] "RemoveContainer" containerID="49e9a6769599b7b0985a13e2cb66627a350ba5f98575e7facdc0ab191db03837"
Oct 30 06:57:49.669399 ip-10-0-128-125 kubenswrapper[6224]: E1030 06:57:49.669382    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49e9a6769599b7b0985a13e2cb66627a350ba5f98575e7facdc0ab191db03837\": container with ID starting with 49e9a6769599b7b0985a13e2cb66627a350ba5f98575e7facdc0ab191db03837 not found: ID does not exist" containerID="49e9a6769599b7b0985a13e2cb66627a350ba5f98575e7facdc0ab191db03837"
Oct 30 06:57:49.669456 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:49.669407    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49e9a6769599b7b0985a13e2cb66627a350ba5f98575e7facdc0ab191db03837"} err="failed to get container status \"49e9a6769599b7b0985a13e2cb66627a350ba5f98575e7facdc0ab191db03837\": rpc error: code = NotFound desc = could not find container \"49e9a6769599b7b0985a13e2cb66627a350ba5f98575e7facdc0ab191db03837\": container with ID starting with 49e9a6769599b7b0985a13e2cb66627a350ba5f98575e7facdc0ab191db03837 not found: ID does not exist"
Oct 30 06:57:49.782160 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:49.782101    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.761625015 podStartE2EDuration="3.782085458s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.460443ms" isStatefulPod=false podCreationTimestamp="2026-10-30 06:57:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:57:49.767458232 +0000 UTC m=+28.376865201" watchObservedRunningTime="2026-10-30 06:57:49.782085458 +0000 UTC m=+28.391492431"
Oct 30 06:57:49.805732 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:49.805709    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c7875b857-wv9lf"]
Oct 30 06:57:49.809455 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:49.809436    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5c7875b857-wv9lf"]
Oct 30 06:57:49.952356 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:49.952327    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f866fbc59-wnrl6"]
Oct 30 06:57:51.500018 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:51.499979    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d24d508-ec6f-4165-8394-f17b09134b34" path="/var/lib/kubelet/pods/9d24d508-ec6f-4165-8394-f17b09134b34/volumes"
Oct 30 06:57:52.191738 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:52.191698    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:57:54.212662 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:57:54.212633    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-c6trh"
Oct 30 06:58:01.855391 ip-10-0-128-125 kubenswrapper[6224]: E1030 06:58:01.855355    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 06:58:14.971197 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:58:14.971135    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-f866fbc59-wnrl6" podUID="bdc803a1-bbb2-407b-baa5-1aa3a54b7021" containerName="console" containerID="cri-o://63d5d853ed7c50fbf974b4fdbf7dcbdfff2c42308ef84ec2c05e635f404d02cc" gracePeriod=15
Oct 30 06:58:15.208820 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:58:15.208787    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f866fbc59-wnrl6"
Oct 30 06:58:15.337289 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:58:15.337250    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-oauth-serving-cert\") pod \"bdc803a1-bbb2-407b-baa5-1aa3a54b7021\" (UID: \"bdc803a1-bbb2-407b-baa5-1aa3a54b7021\") "
Oct 30 06:58:15.337452 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:58:15.337308    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-tmp\" (UniqueName: \"kubernetes.io/empty-dir/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-tmp\") pod \"bdc803a1-bbb2-407b-baa5-1aa3a54b7021\" (UID: \"bdc803a1-bbb2-407b-baa5-1aa3a54b7021\") "
Oct 30 06:58:15.337452 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:58:15.337341    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-service-ca\") pod \"bdc803a1-bbb2-407b-baa5-1aa3a54b7021\" (UID: \"bdc803a1-bbb2-407b-baa5-1aa3a54b7021\") "
Oct 30 06:58:15.337452 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:58:15.337386    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-kube-api-access-q7cdq\" (UniqueName: \"kubernetes.io/projected/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-kube-api-access-q7cdq\") pod \"bdc803a1-bbb2-407b-baa5-1aa3a54b7021\" (UID: \"bdc803a1-bbb2-407b-baa5-1aa3a54b7021\") "
Oct 30 06:58:15.337452 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:58:15.337417    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-console-serving-cert\") pod \"bdc803a1-bbb2-407b-baa5-1aa3a54b7021\" (UID: \"bdc803a1-bbb2-407b-baa5-1aa3a54b7021\") "
Oct 30 06:58:15.337452 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:58:15.337448    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-console-config\" (UniqueName: \"kubernetes.io/configmap/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-console-config\") pod \"bdc803a1-bbb2-407b-baa5-1aa3a54b7021\" (UID: \"bdc803a1-bbb2-407b-baa5-1aa3a54b7021\") "
Oct 30 06:58:15.337717 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:58:15.337471    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-trusted-ca-bundle\") pod \"bdc803a1-bbb2-407b-baa5-1aa3a54b7021\" (UID: \"bdc803a1-bbb2-407b-baa5-1aa3a54b7021\") "
Oct 30 06:58:15.337717 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:58:15.337496    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-console-oauth-config\") pod \"bdc803a1-bbb2-407b-baa5-1aa3a54b7021\" (UID: \"bdc803a1-bbb2-407b-baa5-1aa3a54b7021\") "
Oct 30 06:58:15.337717 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:58:15.337636    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-tmp" pod "bdc803a1-bbb2-407b-baa5-1aa3a54b7021" (UID: "bdc803a1-bbb2-407b-baa5-1aa3a54b7021"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 06:58:15.337717 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:58:15.337710    6224 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-tmp\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 06:58:15.337947 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:58:15.337732    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-oauth-serving-cert" pod "bdc803a1-bbb2-407b-baa5-1aa3a54b7021" (UID: "bdc803a1-bbb2-407b-baa5-1aa3a54b7021"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:58:15.337947 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:58:15.337836    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-service-ca" pod "bdc803a1-bbb2-407b-baa5-1aa3a54b7021" (UID: "bdc803a1-bbb2-407b-baa5-1aa3a54b7021"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:58:15.338527 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:58:15.338495    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-console-config" pod "bdc803a1-bbb2-407b-baa5-1aa3a54b7021" (UID: "bdc803a1-bbb2-407b-baa5-1aa3a54b7021"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:58:15.338527 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:58:15.337956    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-trusted-ca-bundle" pod "bdc803a1-bbb2-407b-baa5-1aa3a54b7021" (UID: "bdc803a1-bbb2-407b-baa5-1aa3a54b7021"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 06:58:15.343441 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:58:15.343415    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-console-oauth-config" pod "bdc803a1-bbb2-407b-baa5-1aa3a54b7021" (UID: "bdc803a1-bbb2-407b-baa5-1aa3a54b7021"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:58:15.343560 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:58:15.343453    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-kube-api-access-q7cdq" pod "bdc803a1-bbb2-407b-baa5-1aa3a54b7021" (UID: "bdc803a1-bbb2-407b-baa5-1aa3a54b7021"). InnerVolumeSpecName "kube-api-access-q7cdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 06:58:15.343560 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:58:15.343477    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-console-serving-cert" pod "bdc803a1-bbb2-407b-baa5-1aa3a54b7021" (UID: "bdc803a1-bbb2-407b-baa5-1aa3a54b7021"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 06:58:15.438432 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:58:15.438409    6224 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-service-ca\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 06:58:15.438432 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:58:15.438430    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7cdq\" (UniqueName: \"kubernetes.io/projected/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-kube-api-access-q7cdq\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 06:58:15.438567 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:58:15.438441    6224 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-console-serving-cert\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 06:58:15.438567 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:58:15.438449    6224 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-console-config\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 06:58:15.438567 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:58:15.438458    6224 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-trusted-ca-bundle\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 06:58:15.438567 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:58:15.438466    6224 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-console-oauth-config\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 06:58:15.438567 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:58:15.438474    6224 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bdc803a1-bbb2-407b-baa5-1aa3a54b7021-oauth-serving-cert\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 06:58:15.748964 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:58:15.748886    6224 generic.go:356] "Generic (PLEG): container finished" podID="bdc803a1-bbb2-407b-baa5-1aa3a54b7021" containerID="63d5d853ed7c50fbf974b4fdbf7dcbdfff2c42308ef84ec2c05e635f404d02cc" exitCode=0
Oct 30 06:58:15.748964 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:58:15.748946    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f866fbc59-wnrl6"
Oct 30 06:58:15.748964 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:58:15.748960    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f866fbc59-wnrl6" event={"ID":"bdc803a1-bbb2-407b-baa5-1aa3a54b7021","Type":"ContainerDied","Data":"63d5d853ed7c50fbf974b4fdbf7dcbdfff2c42308ef84ec2c05e635f404d02cc"}
Oct 30 06:58:15.749168 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:58:15.749002    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f866fbc59-wnrl6" event={"ID":"bdc803a1-bbb2-407b-baa5-1aa3a54b7021","Type":"ContainerDied","Data":"12c2c077b5e8d30e4257b0ea0ae956cbbab35ee764ee7f301a9dc1e01d073c97"}
Oct 30 06:58:15.749168 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:58:15.749024    6224 scope.go:122] "RemoveContainer" containerID="63d5d853ed7c50fbf974b4fdbf7dcbdfff2c42308ef84ec2c05e635f404d02cc"
Oct 30 06:58:15.756508 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:58:15.756488    6224 scope.go:122] "RemoveContainer" containerID="63d5d853ed7c50fbf974b4fdbf7dcbdfff2c42308ef84ec2c05e635f404d02cc"
Oct 30 06:58:15.756763 ip-10-0-128-125 kubenswrapper[6224]: E1030 06:58:15.756741    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63d5d853ed7c50fbf974b4fdbf7dcbdfff2c42308ef84ec2c05e635f404d02cc\": container with ID starting with 63d5d853ed7c50fbf974b4fdbf7dcbdfff2c42308ef84ec2c05e635f404d02cc not found: ID does not exist" containerID="63d5d853ed7c50fbf974b4fdbf7dcbdfff2c42308ef84ec2c05e635f404d02cc"
Oct 30 06:58:15.756845 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:58:15.756771    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63d5d853ed7c50fbf974b4fdbf7dcbdfff2c42308ef84ec2c05e635f404d02cc"} err="failed to get container status \"63d5d853ed7c50fbf974b4fdbf7dcbdfff2c42308ef84ec2c05e635f404d02cc\": rpc error: code = NotFound desc = could not find container \"63d5d853ed7c50fbf974b4fdbf7dcbdfff2c42308ef84ec2c05e635f404d02cc\": container with ID starting with 63d5d853ed7c50fbf974b4fdbf7dcbdfff2c42308ef84ec2c05e635f404d02cc not found: ID does not exist"
Oct 30 06:58:15.804762 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:58:15.804741    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f866fbc59-wnrl6"]
Oct 30 06:58:15.818771 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:58:15.818750    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-f866fbc59-wnrl6"]
Oct 30 06:58:17.501657 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:58:17.501626    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdc803a1-bbb2-407b-baa5-1aa3a54b7021" path="/var/lib/kubelet/pods/bdc803a1-bbb2-407b-baa5-1aa3a54b7021/volumes"
Oct 30 06:58:31.854727 ip-10-0-128-125 kubenswrapper[6224]: E1030 06:58:31.854691    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 06:58:47.409372 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:58:47.409337    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:58:47.424517 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:58:47.424490    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:58:47.867519 ip-10-0-128-125 kubenswrapper[6224]: I1030 06:58:47.867490    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 06:59:01.856916 ip-10-0-128-125 kubenswrapper[6224]: E1030 06:59:01.856875    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 06:59:31.855229 ip-10-0-128-125 kubenswrapper[6224]: E1030 06:59:31.855193    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:00:01.855396 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:00:01.855358    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:00:04.154256 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:04.154218    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vs8c7"]
Oct 30 07:00:04.154641 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:04.154498    6224 cpu_manager.go:414] "containerMap: removing container" podUID="bdc803a1-bbb2-407b-baa5-1aa3a54b7021" containerName="console"
Oct 30 07:00:04.154641 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:04.154509    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdc803a1-bbb2-407b-baa5-1aa3a54b7021" containerName="console"
Oct 30 07:00:04.154641 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:04.154528    6224 cpu_manager.go:414] "containerMap: removing container" podUID="9d24d508-ec6f-4165-8394-f17b09134b34" containerName="console"
Oct 30 07:00:04.154641 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:04.154533    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d24d508-ec6f-4165-8394-f17b09134b34" containerName="console"
Oct 30 07:00:04.154641 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:04.154580    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d24d508-ec6f-4165-8394-f17b09134b34" containerName="console"
Oct 30 07:00:04.154641 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:04.154588    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdc803a1-bbb2-407b-baa5-1aa3a54b7021" containerName="console"
Oct 30 07:00:04.157459 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:04.157437    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vs8c7"
Oct 30 07:00:04.187905 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:04.187863    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 07:00:04.187905 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:04.187862    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 07:00:04.187905 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:04.187902    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-9znvq\""
Oct 30 07:00:04.188544 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:04.188528    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vs8c7"]
Oct 30 07:00:04.313981 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:04.313942    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kls4c\" (UniqueName: \"kubernetes.io/projected/1398f6ad-9bcf-47a7-ae51-8827d4cfe5f3-kube-api-access-kls4c\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vs8c7\" (UID: \"1398f6ad-9bcf-47a7-ae51-8827d4cfe5f3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vs8c7"
Oct 30 07:00:04.314154 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:04.314018    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1398f6ad-9bcf-47a7-ae51-8827d4cfe5f3-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vs8c7\" (UID: \"1398f6ad-9bcf-47a7-ae51-8827d4cfe5f3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vs8c7"
Oct 30 07:00:04.314214 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:04.314151    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1398f6ad-9bcf-47a7-ae51-8827d4cfe5f3-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vs8c7\" (UID: \"1398f6ad-9bcf-47a7-ae51-8827d4cfe5f3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vs8c7"
Oct 30 07:00:04.415042 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:04.414963    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1398f6ad-9bcf-47a7-ae51-8827d4cfe5f3-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vs8c7\" (UID: \"1398f6ad-9bcf-47a7-ae51-8827d4cfe5f3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vs8c7"
Oct 30 07:00:04.415042 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:04.415020    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1398f6ad-9bcf-47a7-ae51-8827d4cfe5f3-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vs8c7\" (UID: \"1398f6ad-9bcf-47a7-ae51-8827d4cfe5f3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vs8c7"
Oct 30 07:00:04.415217 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:04.415043    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kls4c\" (UniqueName: \"kubernetes.io/projected/1398f6ad-9bcf-47a7-ae51-8827d4cfe5f3-kube-api-access-kls4c\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vs8c7\" (UID: \"1398f6ad-9bcf-47a7-ae51-8827d4cfe5f3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vs8c7"
Oct 30 07:00:04.415342 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:04.415323    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1398f6ad-9bcf-47a7-ae51-8827d4cfe5f3-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vs8c7\" (UID: \"1398f6ad-9bcf-47a7-ae51-8827d4cfe5f3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vs8c7"
Oct 30 07:00:04.415382 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:04.415354    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1398f6ad-9bcf-47a7-ae51-8827d4cfe5f3-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vs8c7\" (UID: \"1398f6ad-9bcf-47a7-ae51-8827d4cfe5f3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vs8c7"
Oct 30 07:00:04.440189 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:04.440156    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kls4c\" (UniqueName: \"kubernetes.io/projected/1398f6ad-9bcf-47a7-ae51-8827d4cfe5f3-kube-api-access-kls4c\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vs8c7\" (UID: \"1398f6ad-9bcf-47a7-ae51-8827d4cfe5f3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vs8c7"
Oct 30 07:00:04.473249 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:04.473220    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vs8c7"
Oct 30 07:00:04.764390 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:04.764366    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vs8c7"]
Oct 30 07:00:04.766320 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:00:04.766290    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1398f6ad_9bcf_47a7_ae51_8827d4cfe5f3.slice/crio-b636e0b638d639cb4725525506c5969cc49a0940d19dfdaa18dfdb28d5f4d826 WatchSource:0}: Error finding container b636e0b638d639cb4725525506c5969cc49a0940d19dfdaa18dfdb28d5f4d826: Status 404 returned error can't find the container with id b636e0b638d639cb4725525506c5969cc49a0940d19dfdaa18dfdb28d5f4d826
Oct 30 07:00:04.768424 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:04.768405    6224 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 07:00:05.014020 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:05.013946    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p6bhn"]
Oct 30 07:00:05.018306 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:05.018284    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p6bhn"
Oct 30 07:00:05.038252 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:05.038226    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-bmqcm\""
Oct 30 07:00:05.039038 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:05.039005    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p6bhn"]
Oct 30 07:00:05.097580 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:05.097549    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vs8c7" event={"ID":"1398f6ad-9bcf-47a7-ae51-8827d4cfe5f3","Type":"ContainerStarted","Data":"b636e0b638d639cb4725525506c5969cc49a0940d19dfdaa18dfdb28d5f4d826"}
Oct 30 07:00:05.115264 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:05.115234    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqf6wc"]
Oct 30 07:00:05.118416 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:05.118399    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqf6wc"
Oct 30 07:00:05.120062 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:05.120033    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9kck\" (UniqueName: \"kubernetes.io/projected/461b314c-252b-4479-9006-aca5c0d0f5e5-kube-api-access-f9kck\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p6bhn\" (UID: \"461b314c-252b-4479-9006-aca5c0d0f5e5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p6bhn"
Oct 30 07:00:05.120190 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:05.120066    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/461b314c-252b-4479-9006-aca5c0d0f5e5-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p6bhn\" (UID: \"461b314c-252b-4479-9006-aca5c0d0f5e5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p6bhn"
Oct 30 07:00:05.120190 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:05.120091    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/461b314c-252b-4479-9006-aca5c0d0f5e5-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p6bhn\" (UID: \"461b314c-252b-4479-9006-aca5c0d0f5e5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p6bhn"
Oct 30 07:00:05.137276 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:05.137253    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-7wfx8\""
Oct 30 07:00:05.138376 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:05.138355    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqf6wc"]
Oct 30 07:00:05.211512 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:05.211480    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9w6r8"]
Oct 30 07:00:05.214756 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:05.214740    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9w6r8"
Oct 30 07:00:05.221054 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:05.220633    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9kck\" (UniqueName: \"kubernetes.io/projected/461b314c-252b-4479-9006-aca5c0d0f5e5-kube-api-access-f9kck\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p6bhn\" (UID: \"461b314c-252b-4479-9006-aca5c0d0f5e5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p6bhn"
Oct 30 07:00:05.221054 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:05.220675    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/461b314c-252b-4479-9006-aca5c0d0f5e5-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p6bhn\" (UID: \"461b314c-252b-4479-9006-aca5c0d0f5e5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p6bhn"
Oct 30 07:00:05.221054 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:05.220730    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/461b314c-252b-4479-9006-aca5c0d0f5e5-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p6bhn\" (UID: \"461b314c-252b-4479-9006-aca5c0d0f5e5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p6bhn"
Oct 30 07:00:05.221054 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:05.220758    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/419faae2-2f0c-4c9e-863c-a5968d8e0813-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqf6wc\" (UID: \"419faae2-2f0c-4c9e-863c-a5968d8e0813\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqf6wc"
Oct 30 07:00:05.221054 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:05.220791    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/419faae2-2f0c-4c9e-863c-a5968d8e0813-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqf6wc\" (UID: \"419faae2-2f0c-4c9e-863c-a5968d8e0813\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqf6wc"
Oct 30 07:00:05.221054 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:05.220863    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7r9m\" (UniqueName: \"kubernetes.io/projected/419faae2-2f0c-4c9e-863c-a5968d8e0813-kube-api-access-l7r9m\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqf6wc\" (UID: \"419faae2-2f0c-4c9e-863c-a5968d8e0813\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqf6wc"
Oct 30 07:00:05.221375 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:05.221292    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/461b314c-252b-4479-9006-aca5c0d0f5e5-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p6bhn\" (UID: \"461b314c-252b-4479-9006-aca5c0d0f5e5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p6bhn"
Oct 30 07:00:05.221375 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:05.221298    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/461b314c-252b-4479-9006-aca5c0d0f5e5-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p6bhn\" (UID: \"461b314c-252b-4479-9006-aca5c0d0f5e5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p6bhn"
Oct 30 07:00:05.236145 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:05.236104    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-7jl6t\""
Oct 30 07:00:05.237588 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:05.237560    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9w6r8"]
Oct 30 07:00:05.246222 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:05.246196    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9kck\" (UniqueName: \"kubernetes.io/projected/461b314c-252b-4479-9006-aca5c0d0f5e5-kube-api-access-f9kck\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p6bhn\" (UID: \"461b314c-252b-4479-9006-aca5c0d0f5e5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p6bhn"
Oct 30 07:00:05.316230 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:05.316199    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k7sr9"]
Oct 30 07:00:05.319587 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:05.319563    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k7sr9"
Oct 30 07:00:05.321980 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:05.321957    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/419faae2-2f0c-4c9e-863c-a5968d8e0813-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqf6wc\" (UID: \"419faae2-2f0c-4c9e-863c-a5968d8e0813\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqf6wc"
Oct 30 07:00:05.322104 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:05.321999    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/419faae2-2f0c-4c9e-863c-a5968d8e0813-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqf6wc\" (UID: \"419faae2-2f0c-4c9e-863c-a5968d8e0813\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqf6wc"
Oct 30 07:00:05.322104 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:05.322041    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a04c40e2-2a8e-47eb-8551-a7fcdd0467a9-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9w6r8\" (UID: \"a04c40e2-2a8e-47eb-8551-a7fcdd0467a9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9w6r8"
Oct 30 07:00:05.322104 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:05.322082    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7r9m\" (UniqueName: \"kubernetes.io/projected/419faae2-2f0c-4c9e-863c-a5968d8e0813-kube-api-access-l7r9m\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqf6wc\" (UID: \"419faae2-2f0c-4c9e-863c-a5968d8e0813\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqf6wc"
Oct 30 07:00:05.322269 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:05.322114    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a04c40e2-2a8e-47eb-8551-a7fcdd0467a9-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9w6r8\" (UID: \"a04c40e2-2a8e-47eb-8551-a7fcdd0467a9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9w6r8"
Oct 30 07:00:05.322269 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:05.322182    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln9r2\" (UniqueName: \"kubernetes.io/projected/a04c40e2-2a8e-47eb-8551-a7fcdd0467a9-kube-api-access-ln9r2\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9w6r8\" (UID: \"a04c40e2-2a8e-47eb-8551-a7fcdd0467a9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9w6r8"
Oct 30 07:00:05.322366 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:05.322333    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/419faae2-2f0c-4c9e-863c-a5968d8e0813-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqf6wc\" (UID: \"419faae2-2f0c-4c9e-863c-a5968d8e0813\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqf6wc"
Oct 30 07:00:05.322560 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:05.322522    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/419faae2-2f0c-4c9e-863c-a5968d8e0813-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqf6wc\" (UID: \"419faae2-2f0c-4c9e-863c-a5968d8e0813\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqf6wc"
Oct 30 07:00:05.333397 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:05.333378    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p6bhn"
Oct 30 07:00:05.336184 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:05.336164    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-dqnmq\""
Oct 30 07:00:05.340081 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:05.340055    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k7sr9"]
Oct 30 07:00:05.348143 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:05.348118    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7r9m\" (UniqueName: \"kubernetes.io/projected/419faae2-2f0c-4c9e-863c-a5968d8e0813-kube-api-access-l7r9m\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqf6wc\" (UID: \"419faae2-2f0c-4c9e-863c-a5968d8e0813\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqf6wc"
Oct 30 07:00:05.423422 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:05.423387    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln9r2\" (UniqueName: \"kubernetes.io/projected/a04c40e2-2a8e-47eb-8551-a7fcdd0467a9-kube-api-access-ln9r2\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9w6r8\" (UID: \"a04c40e2-2a8e-47eb-8551-a7fcdd0467a9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9w6r8"
Oct 30 07:00:05.423583 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:05.423563    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9b5a6fc4-47e1-4595-8ecd-216a575703a1-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k7sr9\" (UID: \"9b5a6fc4-47e1-4595-8ecd-216a575703a1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k7sr9"
Oct 30 07:00:05.423643 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:05.423602    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgtnf\" (UniqueName: \"kubernetes.io/projected/9b5a6fc4-47e1-4595-8ecd-216a575703a1-kube-api-access-rgtnf\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k7sr9\" (UID: \"9b5a6fc4-47e1-4595-8ecd-216a575703a1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k7sr9"
Oct 30 07:00:05.423733 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:05.423716    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a04c40e2-2a8e-47eb-8551-a7fcdd0467a9-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9w6r8\" (UID: \"a04c40e2-2a8e-47eb-8551-a7fcdd0467a9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9w6r8"
Oct 30 07:00:05.423796 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:05.423750    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9b5a6fc4-47e1-4595-8ecd-216a575703a1-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k7sr9\" (UID: \"9b5a6fc4-47e1-4595-8ecd-216a575703a1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k7sr9"
Oct 30 07:00:05.423879 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:05.423831    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a04c40e2-2a8e-47eb-8551-a7fcdd0467a9-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9w6r8\" (UID: \"a04c40e2-2a8e-47eb-8551-a7fcdd0467a9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9w6r8"
Oct 30 07:00:05.424121 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:05.424101    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a04c40e2-2a8e-47eb-8551-a7fcdd0467a9-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9w6r8\" (UID: \"a04c40e2-2a8e-47eb-8551-a7fcdd0467a9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9w6r8"
Oct 30 07:00:05.424219 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:05.424198    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a04c40e2-2a8e-47eb-8551-a7fcdd0467a9-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9w6r8\" (UID: \"a04c40e2-2a8e-47eb-8551-a7fcdd0467a9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9w6r8"
Oct 30 07:00:05.434358 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:05.434330    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqf6wc"
Oct 30 07:00:05.449223 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:05.449203    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln9r2\" (UniqueName: \"kubernetes.io/projected/a04c40e2-2a8e-47eb-8551-a7fcdd0467a9-kube-api-access-ln9r2\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9w6r8\" (UID: \"a04c40e2-2a8e-47eb-8551-a7fcdd0467a9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9w6r8"
Oct 30 07:00:05.524358 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:05.524323    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9b5a6fc4-47e1-4595-8ecd-216a575703a1-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k7sr9\" (UID: \"9b5a6fc4-47e1-4595-8ecd-216a575703a1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k7sr9"
Oct 30 07:00:05.524518 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:05.524448    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgtnf\" (UniqueName: \"kubernetes.io/projected/9b5a6fc4-47e1-4595-8ecd-216a575703a1-kube-api-access-rgtnf\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k7sr9\" (UID: \"9b5a6fc4-47e1-4595-8ecd-216a575703a1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k7sr9"
Oct 30 07:00:05.524518 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:05.524499    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9b5a6fc4-47e1-4595-8ecd-216a575703a1-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k7sr9\" (UID: \"9b5a6fc4-47e1-4595-8ecd-216a575703a1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k7sr9"
Oct 30 07:00:05.524686 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:05.524662    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9b5a6fc4-47e1-4595-8ecd-216a575703a1-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k7sr9\" (UID: \"9b5a6fc4-47e1-4595-8ecd-216a575703a1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k7sr9"
Oct 30 07:00:05.524986 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:05.524953    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9b5a6fc4-47e1-4595-8ecd-216a575703a1-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k7sr9\" (UID: \"9b5a6fc4-47e1-4595-8ecd-216a575703a1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k7sr9"
Oct 30 07:00:05.530343 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:05.530322    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9w6r8"
Oct 30 07:00:05.550337 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:05.550312    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgtnf\" (UniqueName: \"kubernetes.io/projected/9b5a6fc4-47e1-4595-8ecd-216a575703a1-kube-api-access-rgtnf\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k7sr9\" (UID: \"9b5a6fc4-47e1-4595-8ecd-216a575703a1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k7sr9"
Oct 30 07:00:05.633998 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:05.633974    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p6bhn"]
Oct 30 07:00:05.635499 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:00:05.635472    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod461b314c_252b_4479_9006_aca5c0d0f5e5.slice/crio-def5facd684844f15d4be7209a2884967a67d7fe61e7f78a05005ac83687ae32 WatchSource:0}: Error finding container def5facd684844f15d4be7209a2884967a67d7fe61e7f78a05005ac83687ae32: Status 404 returned error can't find the container with id def5facd684844f15d4be7209a2884967a67d7fe61e7f78a05005ac83687ae32
Oct 30 07:00:05.636685 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:05.636580    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k7sr9"
Oct 30 07:00:05.738056 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:05.737508    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqf6wc"]
Oct 30 07:00:05.739762 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:00:05.739734    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod419faae2_2f0c_4c9e_863c_a5968d8e0813.slice/crio-33dfd35813a82026b6b75c77f65b7d6f3075df08d82c38debefafb03e04b3d5c WatchSource:0}: Error finding container 33dfd35813a82026b6b75c77f65b7d6f3075df08d82c38debefafb03e04b3d5c: Status 404 returned error can't find the container with id 33dfd35813a82026b6b75c77f65b7d6f3075df08d82c38debefafb03e04b3d5c
Oct 30 07:00:05.837512 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:05.837403    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9w6r8"]
Oct 30 07:00:05.839958 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:00:05.839897    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda04c40e2_2a8e_47eb_8551_a7fcdd0467a9.slice/crio-e6e9938c17b53f2e13ec97afca85bd10ec5e56c1d5fda65ac0f50334e44e5ad8 WatchSource:0}: Error finding container e6e9938c17b53f2e13ec97afca85bd10ec5e56c1d5fda65ac0f50334e44e5ad8: Status 404 returned error can't find the container with id e6e9938c17b53f2e13ec97afca85bd10ec5e56c1d5fda65ac0f50334e44e5ad8
Oct 30 07:00:05.947169 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:05.947144    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k7sr9"]
Oct 30 07:00:05.949236 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:00:05.949205    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b5a6fc4_47e1_4595_8ecd_216a575703a1.slice/crio-1e5f2d948990cb92f459e0eced2ec65f5ac3cb3a08a0a1899611f53e0d33d6f7 WatchSource:0}: Error finding container 1e5f2d948990cb92f459e0eced2ec65f5ac3cb3a08a0a1899611f53e0d33d6f7: Status 404 returned error can't find the container with id 1e5f2d948990cb92f459e0eced2ec65f5ac3cb3a08a0a1899611f53e0d33d6f7
Oct 30 07:00:06.103430 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:06.103359    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqf6wc" event={"ID":"419faae2-2f0c-4c9e-863c-a5968d8e0813","Type":"ContainerStarted","Data":"33dfd35813a82026b6b75c77f65b7d6f3075df08d82c38debefafb03e04b3d5c"}
Oct 30 07:00:06.105074 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:06.105044    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k7sr9" event={"ID":"9b5a6fc4-47e1-4595-8ecd-216a575703a1","Type":"ContainerStarted","Data":"1e5f2d948990cb92f459e0eced2ec65f5ac3cb3a08a0a1899611f53e0d33d6f7"}
Oct 30 07:00:06.106447 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:06.106422    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9w6r8" event={"ID":"a04c40e2-2a8e-47eb-8551-a7fcdd0467a9","Type":"ContainerStarted","Data":"e6e9938c17b53f2e13ec97afca85bd10ec5e56c1d5fda65ac0f50334e44e5ad8"}
Oct 30 07:00:06.107881 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:06.107855    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p6bhn" event={"ID":"461b314c-252b-4479-9006-aca5c0d0f5e5","Type":"ContainerStarted","Data":"def5facd684844f15d4be7209a2884967a67d7fe61e7f78a05005ac83687ae32"}
Oct 30 07:00:06.971902 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:06.970061    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pbc6d"]
Oct 30 07:00:06.974560 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:06.974530    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pbc6d"
Oct 30 07:00:07.005213 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:07.004944    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-ktnjf\""
Oct 30 07:00:07.005611 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:07.005587    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pbc6d"]
Oct 30 07:00:07.139881 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:07.139738    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/44ad92ac-b90f-4d09-be1e-86be52cae40d-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pbc6d\" (UID: \"44ad92ac-b90f-4d09-be1e-86be52cae40d\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pbc6d"
Oct 30 07:00:07.140457 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:07.139794    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/44ad92ac-b90f-4d09-be1e-86be52cae40d-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pbc6d\" (UID: \"44ad92ac-b90f-4d09-be1e-86be52cae40d\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pbc6d"
Oct 30 07:00:07.140457 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:07.140268    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6pjp\" (UniqueName: \"kubernetes.io/projected/44ad92ac-b90f-4d09-be1e-86be52cae40d-kube-api-access-w6pjp\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pbc6d\" (UID: \"44ad92ac-b90f-4d09-be1e-86be52cae40d\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pbc6d"
Oct 30 07:00:07.241629 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:07.241542    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/44ad92ac-b90f-4d09-be1e-86be52cae40d-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pbc6d\" (UID: \"44ad92ac-b90f-4d09-be1e-86be52cae40d\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pbc6d"
Oct 30 07:00:07.241629 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:07.241588    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/44ad92ac-b90f-4d09-be1e-86be52cae40d-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pbc6d\" (UID: \"44ad92ac-b90f-4d09-be1e-86be52cae40d\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pbc6d"
Oct 30 07:00:07.241629 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:07.241622    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6pjp\" (UniqueName: \"kubernetes.io/projected/44ad92ac-b90f-4d09-be1e-86be52cae40d-kube-api-access-w6pjp\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pbc6d\" (UID: \"44ad92ac-b90f-4d09-be1e-86be52cae40d\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pbc6d"
Oct 30 07:00:07.242374 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:07.242302    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/44ad92ac-b90f-4d09-be1e-86be52cae40d-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pbc6d\" (UID: \"44ad92ac-b90f-4d09-be1e-86be52cae40d\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pbc6d"
Oct 30 07:00:07.242374 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:07.242348    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/44ad92ac-b90f-4d09-be1e-86be52cae40d-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pbc6d\" (UID: \"44ad92ac-b90f-4d09-be1e-86be52cae40d\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pbc6d"
Oct 30 07:00:07.280125 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:07.280063    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6pjp\" (UniqueName: \"kubernetes.io/projected/44ad92ac-b90f-4d09-be1e-86be52cae40d-kube-api-access-w6pjp\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pbc6d\" (UID: \"44ad92ac-b90f-4d09-be1e-86be52cae40d\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pbc6d"
Oct 30 07:00:07.298098 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:07.297996    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pbc6d"
Oct 30 07:00:07.628227 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:07.627865    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pbc6d"]
Oct 30 07:00:07.630691 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:00:07.630661    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44ad92ac_b90f_4d09_be1e_86be52cae40d.slice/crio-f907a1cde502e972a96d80bf77175a43513eebb000386c7fa9e4c078db023cfa WatchSource:0}: Error finding container f907a1cde502e972a96d80bf77175a43513eebb000386c7fa9e4c078db023cfa: Status 404 returned error can't find the container with id f907a1cde502e972a96d80bf77175a43513eebb000386c7fa9e4c078db023cfa
Oct 30 07:00:08.123989 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:08.123953    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pbc6d" event={"ID":"44ad92ac-b90f-4d09-be1e-86be52cae40d","Type":"ContainerStarted","Data":"f907a1cde502e972a96d80bf77175a43513eebb000386c7fa9e4c078db023cfa"}
Oct 30 07:00:08.196181 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:08.196146    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5mjkn"]
Oct 30 07:00:08.199784 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:08.199756    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5mjkn"
Oct 30 07:00:08.230764 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:08.230738    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5mjkn"]
Oct 30 07:00:08.230914 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:08.230890    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-clvkk\""
Oct 30 07:00:08.351891 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:08.351853    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/10da4324-a9c4-46ac-b9d7-6df46dd60905-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5mjkn\" (UID: \"10da4324-a9c4-46ac-b9d7-6df46dd60905\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5mjkn"
Oct 30 07:00:08.352075 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:08.351969    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hk2m\" (UniqueName: \"kubernetes.io/projected/10da4324-a9c4-46ac-b9d7-6df46dd60905-kube-api-access-6hk2m\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5mjkn\" (UID: \"10da4324-a9c4-46ac-b9d7-6df46dd60905\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5mjkn"
Oct 30 07:00:08.352075 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:08.352006    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/10da4324-a9c4-46ac-b9d7-6df46dd60905-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5mjkn\" (UID: \"10da4324-a9c4-46ac-b9d7-6df46dd60905\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5mjkn"
Oct 30 07:00:08.453508 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:08.453426    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hk2m\" (UniqueName: \"kubernetes.io/projected/10da4324-a9c4-46ac-b9d7-6df46dd60905-kube-api-access-6hk2m\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5mjkn\" (UID: \"10da4324-a9c4-46ac-b9d7-6df46dd60905\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5mjkn"
Oct 30 07:00:08.453508 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:08.453486    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/10da4324-a9c4-46ac-b9d7-6df46dd60905-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5mjkn\" (UID: \"10da4324-a9c4-46ac-b9d7-6df46dd60905\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5mjkn"
Oct 30 07:00:08.453728 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:08.453525    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/10da4324-a9c4-46ac-b9d7-6df46dd60905-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5mjkn\" (UID: \"10da4324-a9c4-46ac-b9d7-6df46dd60905\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5mjkn"
Oct 30 07:00:08.454046 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:08.454020    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/10da4324-a9c4-46ac-b9d7-6df46dd60905-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5mjkn\" (UID: \"10da4324-a9c4-46ac-b9d7-6df46dd60905\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5mjkn"
Oct 30 07:00:08.454615 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:08.454591    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/10da4324-a9c4-46ac-b9d7-6df46dd60905-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5mjkn\" (UID: \"10da4324-a9c4-46ac-b9d7-6df46dd60905\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5mjkn"
Oct 30 07:00:08.491859 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:08.484867    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hk2m\" (UniqueName: \"kubernetes.io/projected/10da4324-a9c4-46ac-b9d7-6df46dd60905-kube-api-access-6hk2m\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5mjkn\" (UID: \"10da4324-a9c4-46ac-b9d7-6df46dd60905\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5mjkn"
Oct 30 07:00:08.521476 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:08.521445    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5mjkn"
Oct 30 07:00:08.858795 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:08.858395    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5mjkn"]
Oct 30 07:00:08.865581 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:00:08.865521    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10da4324_a9c4_46ac_b9d7_6df46dd60905.slice/crio-de563955c450a6bda033500784bd48a703a331680b732c9eb385af1893c8be75 WatchSource:0}: Error finding container de563955c450a6bda033500784bd48a703a331680b732c9eb385af1893c8be75: Status 404 returned error can't find the container with id de563955c450a6bda033500784bd48a703a331680b732c9eb385af1893c8be75
Oct 30 07:00:09.131322 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:09.131237    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5mjkn" event={"ID":"10da4324-a9c4-46ac-b9d7-6df46dd60905","Type":"ContainerStarted","Data":"de563955c450a6bda033500784bd48a703a331680b732c9eb385af1893c8be75"}
Oct 30 07:00:17.214378 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:17.212947    6224 generic.go:356] "Generic (PLEG): container finished" podID="1398f6ad-9bcf-47a7-ae51-8827d4cfe5f3" containerID="ff7f59e8da3b49ff291a5b61f0a584f2bf5a1d19cc29f00c35a87b873c00503d" exitCode=0
Oct 30 07:00:17.214378 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:17.213067    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vs8c7" event={"ID":"1398f6ad-9bcf-47a7-ae51-8827d4cfe5f3","Type":"ContainerDied","Data":"ff7f59e8da3b49ff291a5b61f0a584f2bf5a1d19cc29f00c35a87b873c00503d"}
Oct 30 07:00:18.223141 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:18.223102    6224 generic.go:356] "Generic (PLEG): container finished" podID="10da4324-a9c4-46ac-b9d7-6df46dd60905" containerID="e32d687dba8668a6b6da2973e50ead966a96aaaf7939ba2eecf4ca8f0ac6e788" exitCode=0
Oct 30 07:00:18.223686 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:18.223184    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5mjkn" event={"ID":"10da4324-a9c4-46ac-b9d7-6df46dd60905","Type":"ContainerDied","Data":"e32d687dba8668a6b6da2973e50ead966a96aaaf7939ba2eecf4ca8f0ac6e788"}
Oct 30 07:00:19.230794 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:19.229712    6224 generic.go:356] "Generic (PLEG): container finished" podID="1398f6ad-9bcf-47a7-ae51-8827d4cfe5f3" containerID="e195d6aa61a7221e8556c3b1d963abac44883380c29896731c7de6398c21c209" exitCode=0
Oct 30 07:00:19.230794 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:19.229827    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vs8c7" event={"ID":"1398f6ad-9bcf-47a7-ae51-8827d4cfe5f3","Type":"ContainerDied","Data":"e195d6aa61a7221e8556c3b1d963abac44883380c29896731c7de6398c21c209"}
Oct 30 07:00:19.231563 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:19.231497    6224 generic.go:356] "Generic (PLEG): container finished" podID="461b314c-252b-4479-9006-aca5c0d0f5e5" containerID="3c135524c3cefca2348f81fb82776fd2043d2c505d8b2c607335dc7efd6f3e4d" exitCode=0
Oct 30 07:00:19.231563 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:19.231551    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p6bhn" event={"ID":"461b314c-252b-4479-9006-aca5c0d0f5e5","Type":"ContainerDied","Data":"3c135524c3cefca2348f81fb82776fd2043d2c505d8b2c607335dc7efd6f3e4d"}
Oct 30 07:00:20.236978 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:20.236951    6224 generic.go:356] "Generic (PLEG): container finished" podID="419faae2-2f0c-4c9e-863c-a5968d8e0813" containerID="5586c1217ba00922f9b3671697401e4e98549d769e1d77e20356e7cd26f77e86" exitCode=0
Oct 30 07:00:20.237361 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:20.237048    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqf6wc" event={"ID":"419faae2-2f0c-4c9e-863c-a5968d8e0813","Type":"ContainerDied","Data":"5586c1217ba00922f9b3671697401e4e98549d769e1d77e20356e7cd26f77e86"}
Oct 30 07:00:20.239469 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:20.239390    6224 generic.go:356] "Generic (PLEG): container finished" podID="9b5a6fc4-47e1-4595-8ecd-216a575703a1" containerID="9f2c5cdf5fdddd4d58696f8080290e93d116e63923b2fb4116badac9fcebca32" exitCode=0
Oct 30 07:00:20.239581 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:20.239470    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k7sr9" event={"ID":"9b5a6fc4-47e1-4595-8ecd-216a575703a1","Type":"ContainerDied","Data":"9f2c5cdf5fdddd4d58696f8080290e93d116e63923b2fb4116badac9fcebca32"}
Oct 30 07:00:20.241294 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:20.241272    6224 generic.go:356] "Generic (PLEG): container finished" podID="a04c40e2-2a8e-47eb-8551-a7fcdd0467a9" containerID="8dd228bdc35027f0fb57129c1a4a317d88428df809035d781baa343ad155f37f" exitCode=0
Oct 30 07:00:20.241535 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:20.241511    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9w6r8" event={"ID":"a04c40e2-2a8e-47eb-8551-a7fcdd0467a9","Type":"ContainerDied","Data":"8dd228bdc35027f0fb57129c1a4a317d88428df809035d781baa343ad155f37f"}
Oct 30 07:00:21.249570 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:21.249535    6224 generic.go:356] "Generic (PLEG): container finished" podID="44ad92ac-b90f-4d09-be1e-86be52cae40d" containerID="3c28ceee02c19ff309176d6d6eb1cc0ee8e2eaedd18b8cbc93b94faa26647bc4" exitCode=0
Oct 30 07:00:21.250058 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:21.249637    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pbc6d" event={"ID":"44ad92ac-b90f-4d09-be1e-86be52cae40d","Type":"ContainerDied","Data":"3c28ceee02c19ff309176d6d6eb1cc0ee8e2eaedd18b8cbc93b94faa26647bc4"}
Oct 30 07:00:21.252281 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:21.252255    6224 generic.go:356] "Generic (PLEG): container finished" podID="461b314c-252b-4479-9006-aca5c0d0f5e5" containerID="ab1c04ca9d38c05aba2904a66191ed9d850ae40b65021445075b3013097a983b" exitCode=0
Oct 30 07:00:21.252373 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:21.252354    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p6bhn" event={"ID":"461b314c-252b-4479-9006-aca5c0d0f5e5","Type":"ContainerDied","Data":"ab1c04ca9d38c05aba2904a66191ed9d850ae40b65021445075b3013097a983b"}
Oct 30 07:00:22.260530 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:22.260493    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5mjkn" event={"ID":"10da4324-a9c4-46ac-b9d7-6df46dd60905","Type":"ContainerStarted","Data":"e5a343fb6f9450fd3bd8f5180e4fda9887b363051d6aeab1941429e765c6d6fc"}
Oct 30 07:00:22.265091 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:22.265067    6224 generic.go:356] "Generic (PLEG): container finished" podID="9b5a6fc4-47e1-4595-8ecd-216a575703a1" containerID="d523e6091699984949e6a01491484475fcd921fe75b464d2c4a3f042d6b6dc8b" exitCode=0
Oct 30 07:00:22.265195 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:22.265135    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k7sr9" event={"ID":"9b5a6fc4-47e1-4595-8ecd-216a575703a1","Type":"ContainerDied","Data":"d523e6091699984949e6a01491484475fcd921fe75b464d2c4a3f042d6b6dc8b"}
Oct 30 07:00:22.268764 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:22.268739    6224 generic.go:356] "Generic (PLEG): container finished" podID="a04c40e2-2a8e-47eb-8551-a7fcdd0467a9" containerID="22158903b925047a17a33d41f44bdf7329a79dc554714d96e0171f42dc524fff" exitCode=0
Oct 30 07:00:22.268875 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:22.268779    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9w6r8" event={"ID":"a04c40e2-2a8e-47eb-8551-a7fcdd0467a9","Type":"ContainerDied","Data":"22158903b925047a17a33d41f44bdf7329a79dc554714d96e0171f42dc524fff"}
Oct 30 07:00:23.279457 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:23.278425    6224 generic.go:356] "Generic (PLEG): container finished" podID="10da4324-a9c4-46ac-b9d7-6df46dd60905" containerID="e5a343fb6f9450fd3bd8f5180e4fda9887b363051d6aeab1941429e765c6d6fc" exitCode=0
Oct 30 07:00:23.279457 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:23.278540    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5mjkn" event={"ID":"10da4324-a9c4-46ac-b9d7-6df46dd60905","Type":"ContainerDied","Data":"e5a343fb6f9450fd3bd8f5180e4fda9887b363051d6aeab1941429e765c6d6fc"}
Oct 30 07:00:23.283068 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:23.283039    6224 generic.go:356] "Generic (PLEG): container finished" podID="419faae2-2f0c-4c9e-863c-a5968d8e0813" containerID="ace06af864246e15be961c22ea54e5671c3dcceee7f28a33c445948a37dda31a" exitCode=0
Oct 30 07:00:23.283165 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:23.283150    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqf6wc" event={"ID":"419faae2-2f0c-4c9e-863c-a5968d8e0813","Type":"ContainerDied","Data":"ace06af864246e15be961c22ea54e5671c3dcceee7f28a33c445948a37dda31a"}
Oct 30 07:00:25.307679 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:25.307640    6224 generic.go:356] "Generic (PLEG): container finished" podID="44ad92ac-b90f-4d09-be1e-86be52cae40d" containerID="9a13d02258f540366add972050a1f37813d86c3c87dbc154cd57561bf8a7de87" exitCode=0
Oct 30 07:00:25.308192 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:25.307761    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pbc6d" event={"ID":"44ad92ac-b90f-4d09-be1e-86be52cae40d","Type":"ContainerDied","Data":"9a13d02258f540366add972050a1f37813d86c3c87dbc154cd57561bf8a7de87"}
Oct 30 07:00:31.358823 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:31.358711    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vs8c7" event={"ID":"1398f6ad-9bcf-47a7-ae51-8827d4cfe5f3","Type":"ContainerStarted","Data":"ce7b6ac4a36b1c2d0a826bb74f8603f6bcc7b6a5b24fe0ba357f61c9e66e9233"}
Oct 30 07:00:31.406314 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:31.406254    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vs8c7" podStartSLOduration=2.14844389 podStartE2EDuration="27.406235844s" totalImagesPullingTime="25.257791954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:00:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:00:31.406016753 +0000 UTC m=+190.015423724" watchObservedRunningTime="2026-10-30 07:00:31.406235844 +0000 UTC m=+190.015642815"
Oct 30 07:00:31.866322 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:00:31.866214    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:00:32.367171 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:32.367141    6224 generic.go:356] "Generic (PLEG): container finished" podID="1398f6ad-9bcf-47a7-ae51-8827d4cfe5f3" containerID="ce7b6ac4a36b1c2d0a826bb74f8603f6bcc7b6a5b24fe0ba357f61c9e66e9233" exitCode=0
Oct 30 07:00:32.367634 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:32.367215    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vs8c7" event={"ID":"1398f6ad-9bcf-47a7-ae51-8827d4cfe5f3","Type":"ContainerDied","Data":"ce7b6ac4a36b1c2d0a826bb74f8603f6bcc7b6a5b24fe0ba357f61c9e66e9233"}
Oct 30 07:00:32.372765 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:32.372733    6224 generic.go:356] "Generic (PLEG): container finished" podID="419faae2-2f0c-4c9e-863c-a5968d8e0813" containerID="8c0eb86f66fc3f29ed278b1f48d7ca2c1ea681a849661b2903d93449c64a1852" exitCode=0
Oct 30 07:00:32.372907 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:32.372891    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqf6wc" event={"ID":"419faae2-2f0c-4c9e-863c-a5968d8e0813","Type":"ContainerDied","Data":"8c0eb86f66fc3f29ed278b1f48d7ca2c1ea681a849661b2903d93449c64a1852"}
Oct 30 07:00:32.376085 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:32.376059    6224 generic.go:356] "Generic (PLEG): container finished" podID="10da4324-a9c4-46ac-b9d7-6df46dd60905" containerID="24898e9b63c830636e16b7b38c75d64e69bd0a1642df2bc603c59872bc5e79d4" exitCode=0
Oct 30 07:00:32.376177 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:32.376099    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5mjkn" event={"ID":"10da4324-a9c4-46ac-b9d7-6df46dd60905","Type":"ContainerDied","Data":"24898e9b63c830636e16b7b38c75d64e69bd0a1642df2bc603c59872bc5e79d4"}
Oct 30 07:00:33.578345 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:33.578009    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vs8c7"
Oct 30 07:00:33.666979 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:33.666950    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqf6wc"
Oct 30 07:00:33.675379 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:33.675294    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5mjkn"
Oct 30 07:00:33.706829 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:33.706359    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1398f6ad-9bcf-47a7-ae51-8827d4cfe5f3-util\" (UniqueName: \"kubernetes.io/empty-dir/1398f6ad-9bcf-47a7-ae51-8827d4cfe5f3-util\") pod \"1398f6ad-9bcf-47a7-ae51-8827d4cfe5f3\" (UID: \"1398f6ad-9bcf-47a7-ae51-8827d4cfe5f3\") "
Oct 30 07:00:33.706829 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:33.706419    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1398f6ad-9bcf-47a7-ae51-8827d4cfe5f3-kube-api-access-kls4c\" (UniqueName: \"kubernetes.io/projected/1398f6ad-9bcf-47a7-ae51-8827d4cfe5f3-kube-api-access-kls4c\") pod \"1398f6ad-9bcf-47a7-ae51-8827d4cfe5f3\" (UID: \"1398f6ad-9bcf-47a7-ae51-8827d4cfe5f3\") "
Oct 30 07:00:33.706829 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:33.706451    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1398f6ad-9bcf-47a7-ae51-8827d4cfe5f3-bundle\" (UniqueName: \"kubernetes.io/empty-dir/1398f6ad-9bcf-47a7-ae51-8827d4cfe5f3-bundle\") pod \"1398f6ad-9bcf-47a7-ae51-8827d4cfe5f3\" (UID: \"1398f6ad-9bcf-47a7-ae51-8827d4cfe5f3\") "
Oct 30 07:00:33.707599 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:33.707565    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1398f6ad-9bcf-47a7-ae51-8827d4cfe5f3-bundle" pod "1398f6ad-9bcf-47a7-ae51-8827d4cfe5f3" (UID: "1398f6ad-9bcf-47a7-ae51-8827d4cfe5f3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:00:33.709351 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:33.709306    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1398f6ad-9bcf-47a7-ae51-8827d4cfe5f3-kube-api-access-kls4c" pod "1398f6ad-9bcf-47a7-ae51-8827d4cfe5f3" (UID: "1398f6ad-9bcf-47a7-ae51-8827d4cfe5f3"). InnerVolumeSpecName "kube-api-access-kls4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:00:33.713249 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:33.713218    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1398f6ad-9bcf-47a7-ae51-8827d4cfe5f3-util" pod "1398f6ad-9bcf-47a7-ae51-8827d4cfe5f3" (UID: "1398f6ad-9bcf-47a7-ae51-8827d4cfe5f3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:00:33.807038 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:33.806963    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/419faae2-2f0c-4c9e-863c-a5968d8e0813-bundle\" (UniqueName: \"kubernetes.io/empty-dir/419faae2-2f0c-4c9e-863c-a5968d8e0813-bundle\") pod \"419faae2-2f0c-4c9e-863c-a5968d8e0813\" (UID: \"419faae2-2f0c-4c9e-863c-a5968d8e0813\") "
Oct 30 07:00:33.807381 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:33.807329    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10da4324-a9c4-46ac-b9d7-6df46dd60905-bundle\" (UniqueName: \"kubernetes.io/empty-dir/10da4324-a9c4-46ac-b9d7-6df46dd60905-bundle\") pod \"10da4324-a9c4-46ac-b9d7-6df46dd60905\" (UID: \"10da4324-a9c4-46ac-b9d7-6df46dd60905\") "
Oct 30 07:00:33.807493 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:33.807365    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10da4324-a9c4-46ac-b9d7-6df46dd60905-util\" (UniqueName: \"kubernetes.io/empty-dir/10da4324-a9c4-46ac-b9d7-6df46dd60905-util\") pod \"10da4324-a9c4-46ac-b9d7-6df46dd60905\" (UID: \"10da4324-a9c4-46ac-b9d7-6df46dd60905\") "
Oct 30 07:00:33.807646 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:33.807541    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/419faae2-2f0c-4c9e-863c-a5968d8e0813-kube-api-access-l7r9m\" (UniqueName: \"kubernetes.io/projected/419faae2-2f0c-4c9e-863c-a5968d8e0813-kube-api-access-l7r9m\") pod \"419faae2-2f0c-4c9e-863c-a5968d8e0813\" (UID: \"419faae2-2f0c-4c9e-863c-a5968d8e0813\") "
Oct 30 07:00:33.807646 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:33.807576    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/419faae2-2f0c-4c9e-863c-a5968d8e0813-util\" (UniqueName: \"kubernetes.io/empty-dir/419faae2-2f0c-4c9e-863c-a5968d8e0813-util\") pod \"419faae2-2f0c-4c9e-863c-a5968d8e0813\" (UID: \"419faae2-2f0c-4c9e-863c-a5968d8e0813\") "
Oct 30 07:00:33.807646 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:33.807611    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10da4324-a9c4-46ac-b9d7-6df46dd60905-kube-api-access-6hk2m\" (UniqueName: \"kubernetes.io/projected/10da4324-a9c4-46ac-b9d7-6df46dd60905-kube-api-access-6hk2m\") pod \"10da4324-a9c4-46ac-b9d7-6df46dd60905\" (UID: \"10da4324-a9c4-46ac-b9d7-6df46dd60905\") "
Oct 30 07:00:33.807824 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:33.807747    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/419faae2-2f0c-4c9e-863c-a5968d8e0813-bundle" pod "419faae2-2f0c-4c9e-863c-a5968d8e0813" (UID: "419faae2-2f0c-4c9e-863c-a5968d8e0813"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:00:33.807901 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:33.807883    6224 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1398f6ad-9bcf-47a7-ae51-8827d4cfe5f3-bundle\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:00:33.807954 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:33.807908    6224 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/419faae2-2f0c-4c9e-863c-a5968d8e0813-bundle\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:00:33.807954 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:33.807922    6224 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1398f6ad-9bcf-47a7-ae51-8827d4cfe5f3-util\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:00:33.807954 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:33.807936    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kls4c\" (UniqueName: \"kubernetes.io/projected/1398f6ad-9bcf-47a7-ae51-8827d4cfe5f3-kube-api-access-kls4c\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:00:33.808324 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:33.808291    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10da4324-a9c4-46ac-b9d7-6df46dd60905-bundle" pod "10da4324-a9c4-46ac-b9d7-6df46dd60905" (UID: "10da4324-a9c4-46ac-b9d7-6df46dd60905"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:00:33.810335 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:33.810281    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10da4324-a9c4-46ac-b9d7-6df46dd60905-kube-api-access-6hk2m" pod "10da4324-a9c4-46ac-b9d7-6df46dd60905" (UID: "10da4324-a9c4-46ac-b9d7-6df46dd60905"). InnerVolumeSpecName "kube-api-access-6hk2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:00:33.814173 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:33.814140    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/419faae2-2f0c-4c9e-863c-a5968d8e0813-util" pod "419faae2-2f0c-4c9e-863c-a5968d8e0813" (UID: "419faae2-2f0c-4c9e-863c-a5968d8e0813"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:00:33.815578 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:33.815552    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10da4324-a9c4-46ac-b9d7-6df46dd60905-util" pod "10da4324-a9c4-46ac-b9d7-6df46dd60905" (UID: "10da4324-a9c4-46ac-b9d7-6df46dd60905"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:00:33.816219 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:33.816195    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/419faae2-2f0c-4c9e-863c-a5968d8e0813-kube-api-access-l7r9m" pod "419faae2-2f0c-4c9e-863c-a5968d8e0813" (UID: "419faae2-2f0c-4c9e-863c-a5968d8e0813"). InnerVolumeSpecName "kube-api-access-l7r9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:00:33.908948 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:33.908845    6224 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/10da4324-a9c4-46ac-b9d7-6df46dd60905-bundle\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:00:33.908948 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:33.908881    6224 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/10da4324-a9c4-46ac-b9d7-6df46dd60905-util\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:00:33.908948 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:33.908898    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7r9m\" (UniqueName: \"kubernetes.io/projected/419faae2-2f0c-4c9e-863c-a5968d8e0813-kube-api-access-l7r9m\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:00:33.908948 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:33.908918    6224 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/419faae2-2f0c-4c9e-863c-a5968d8e0813-util\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:00:33.908948 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:33.908927    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hk2m\" (UniqueName: \"kubernetes.io/projected/10da4324-a9c4-46ac-b9d7-6df46dd60905-kube-api-access-6hk2m\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:00:34.387069 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:34.387031    6224 generic.go:356] "Generic (PLEG): container finished" podID="44ad92ac-b90f-4d09-be1e-86be52cae40d" containerID="cb85df33b1dfc108bb73bf2257a4c0f5870cb299322e176b3bc1df076b616cc2" exitCode=0
Oct 30 07:00:34.387224 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:34.387131    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pbc6d" event={"ID":"44ad92ac-b90f-4d09-be1e-86be52cae40d","Type":"ContainerDied","Data":"cb85df33b1dfc108bb73bf2257a4c0f5870cb299322e176b3bc1df076b616cc2"}
Oct 30 07:00:34.389658 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:34.389631    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5mjkn"
Oct 30 07:00:34.389773 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:34.389630    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5mjkn" event={"ID":"10da4324-a9c4-46ac-b9d7-6df46dd60905","Type":"ContainerDied","Data":"de563955c450a6bda033500784bd48a703a331680b732c9eb385af1893c8be75"}
Oct 30 07:00:34.389867 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:34.389829    6224 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de563955c450a6bda033500784bd48a703a331680b732c9eb385af1893c8be75"
Oct 30 07:00:34.394448 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:34.394428    6224 generic.go:356] "Generic (PLEG): container finished" podID="461b314c-252b-4479-9006-aca5c0d0f5e5" containerID="68fbf8f41507a8332f225ae1d33f034f45bbb6414c6081a5416b3e72bb3182f2" exitCode=0
Oct 30 07:00:34.394547 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:34.394515    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p6bhn" event={"ID":"461b314c-252b-4479-9006-aca5c0d0f5e5","Type":"ContainerDied","Data":"68fbf8f41507a8332f225ae1d33f034f45bbb6414c6081a5416b3e72bb3182f2"}
Oct 30 07:00:34.396937 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:34.396917    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vs8c7"
Oct 30 07:00:34.397239 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:34.397215    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vs8c7" event={"ID":"1398f6ad-9bcf-47a7-ae51-8827d4cfe5f3","Type":"ContainerDied","Data":"b636e0b638d639cb4725525506c5969cc49a0940d19dfdaa18dfdb28d5f4d826"}
Oct 30 07:00:34.397330 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:34.397259    6224 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b636e0b638d639cb4725525506c5969cc49a0940d19dfdaa18dfdb28d5f4d826"
Oct 30 07:00:34.399762 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:34.399739    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqf6wc" event={"ID":"419faae2-2f0c-4c9e-863c-a5968d8e0813","Type":"ContainerDied","Data":"33dfd35813a82026b6b75c77f65b7d6f3075df08d82c38debefafb03e04b3d5c"}
Oct 30 07:00:34.399884 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:34.399780    6224 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33dfd35813a82026b6b75c77f65b7d6f3075df08d82c38debefafb03e04b3d5c"
Oct 30 07:00:34.399884 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:34.399866    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqf6wc"
Oct 30 07:00:35.404716 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:35.404688    6224 generic.go:356] "Generic (PLEG): container finished" podID="9b5a6fc4-47e1-4595-8ecd-216a575703a1" containerID="130595ff56b4e8173c3a2f72fbcb5fb169186655be7d069387c8eb17950112a5" exitCode=0
Oct 30 07:00:35.405065 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:35.404771    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k7sr9" event={"ID":"9b5a6fc4-47e1-4595-8ecd-216a575703a1","Type":"ContainerDied","Data":"130595ff56b4e8173c3a2f72fbcb5fb169186655be7d069387c8eb17950112a5"}
Oct 30 07:00:35.406761 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:35.406742    6224 generic.go:356] "Generic (PLEG): container finished" podID="a04c40e2-2a8e-47eb-8551-a7fcdd0467a9" containerID="e1da104152ba5c8b17a9a5fdf1ee2cd3ae58ac6605e51a79a05f7d42a7847ff8" exitCode=0
Oct 30 07:00:35.406864 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:35.406832    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9w6r8" event={"ID":"a04c40e2-2a8e-47eb-8551-a7fcdd0467a9","Type":"ContainerDied","Data":"e1da104152ba5c8b17a9a5fdf1ee2cd3ae58ac6605e51a79a05f7d42a7847ff8"}
Oct 30 07:00:35.559981 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:35.559959    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p6bhn"
Oct 30 07:00:35.572057 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:35.572037    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pbc6d"
Oct 30 07:00:35.622034 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:35.622002    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/461b314c-252b-4479-9006-aca5c0d0f5e5-kube-api-access-f9kck\" (UniqueName: \"kubernetes.io/projected/461b314c-252b-4479-9006-aca5c0d0f5e5-kube-api-access-f9kck\") pod \"461b314c-252b-4479-9006-aca5c0d0f5e5\" (UID: \"461b314c-252b-4479-9006-aca5c0d0f5e5\") "
Oct 30 07:00:35.622190 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:35.622089    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/461b314c-252b-4479-9006-aca5c0d0f5e5-util\" (UniqueName: \"kubernetes.io/empty-dir/461b314c-252b-4479-9006-aca5c0d0f5e5-util\") pod \"461b314c-252b-4479-9006-aca5c0d0f5e5\" (UID: \"461b314c-252b-4479-9006-aca5c0d0f5e5\") "
Oct 30 07:00:35.622190 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:35.622143    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/461b314c-252b-4479-9006-aca5c0d0f5e5-bundle\" (UniqueName: \"kubernetes.io/empty-dir/461b314c-252b-4479-9006-aca5c0d0f5e5-bundle\") pod \"461b314c-252b-4479-9006-aca5c0d0f5e5\" (UID: \"461b314c-252b-4479-9006-aca5c0d0f5e5\") "
Oct 30 07:00:35.622997 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:35.622965    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/461b314c-252b-4479-9006-aca5c0d0f5e5-bundle" pod "461b314c-252b-4479-9006-aca5c0d0f5e5" (UID: "461b314c-252b-4479-9006-aca5c0d0f5e5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:00:35.624413 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:35.624390    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/461b314c-252b-4479-9006-aca5c0d0f5e5-kube-api-access-f9kck" pod "461b314c-252b-4479-9006-aca5c0d0f5e5" (UID: "461b314c-252b-4479-9006-aca5c0d0f5e5"). InnerVolumeSpecName "kube-api-access-f9kck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:00:35.628593 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:35.628566    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/461b314c-252b-4479-9006-aca5c0d0f5e5-util" pod "461b314c-252b-4479-9006-aca5c0d0f5e5" (UID: "461b314c-252b-4479-9006-aca5c0d0f5e5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:00:35.723428 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:35.723403    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44ad92ac-b90f-4d09-be1e-86be52cae40d-bundle\" (UniqueName: \"kubernetes.io/empty-dir/44ad92ac-b90f-4d09-be1e-86be52cae40d-bundle\") pod \"44ad92ac-b90f-4d09-be1e-86be52cae40d\" (UID: \"44ad92ac-b90f-4d09-be1e-86be52cae40d\") "
Oct 30 07:00:35.723557 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:35.723480    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44ad92ac-b90f-4d09-be1e-86be52cae40d-kube-api-access-w6pjp\" (UniqueName: \"kubernetes.io/projected/44ad92ac-b90f-4d09-be1e-86be52cae40d-kube-api-access-w6pjp\") pod \"44ad92ac-b90f-4d09-be1e-86be52cae40d\" (UID: \"44ad92ac-b90f-4d09-be1e-86be52cae40d\") "
Oct 30 07:00:35.723557 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:35.723519    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44ad92ac-b90f-4d09-be1e-86be52cae40d-util\" (UniqueName: \"kubernetes.io/empty-dir/44ad92ac-b90f-4d09-be1e-86be52cae40d-util\") pod \"44ad92ac-b90f-4d09-be1e-86be52cae40d\" (UID: \"44ad92ac-b90f-4d09-be1e-86be52cae40d\") "
Oct 30 07:00:35.723721 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:35.723705    6224 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/461b314c-252b-4479-9006-aca5c0d0f5e5-util\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:00:35.723785 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:35.723728    6224 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/461b314c-252b-4479-9006-aca5c0d0f5e5-bundle\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:00:35.723785 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:35.723743    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9kck\" (UniqueName: \"kubernetes.io/projected/461b314c-252b-4479-9006-aca5c0d0f5e5-kube-api-access-f9kck\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:00:35.723912 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:35.723887    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44ad92ac-b90f-4d09-be1e-86be52cae40d-bundle" pod "44ad92ac-b90f-4d09-be1e-86be52cae40d" (UID: "44ad92ac-b90f-4d09-be1e-86be52cae40d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:00:35.725586 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:35.725563    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44ad92ac-b90f-4d09-be1e-86be52cae40d-kube-api-access-w6pjp" pod "44ad92ac-b90f-4d09-be1e-86be52cae40d" (UID: "44ad92ac-b90f-4d09-be1e-86be52cae40d"). InnerVolumeSpecName "kube-api-access-w6pjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:00:35.727390 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:35.727369    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44ad92ac-b90f-4d09-be1e-86be52cae40d-util" pod "44ad92ac-b90f-4d09-be1e-86be52cae40d" (UID: "44ad92ac-b90f-4d09-be1e-86be52cae40d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:00:35.824354 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:35.824334    6224 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/44ad92ac-b90f-4d09-be1e-86be52cae40d-bundle\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:00:35.824354 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:35.824355    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6pjp\" (UniqueName: \"kubernetes.io/projected/44ad92ac-b90f-4d09-be1e-86be52cae40d-kube-api-access-w6pjp\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:00:35.824477 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:35.824365    6224 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/44ad92ac-b90f-4d09-be1e-86be52cae40d-util\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:00:36.412325 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:36.412301    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pbc6d"
Oct 30 07:00:36.412772 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:36.412295    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pbc6d" event={"ID":"44ad92ac-b90f-4d09-be1e-86be52cae40d","Type":"ContainerDied","Data":"f907a1cde502e972a96d80bf77175a43513eebb000386c7fa9e4c078db023cfa"}
Oct 30 07:00:36.412772 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:36.412438    6224 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f907a1cde502e972a96d80bf77175a43513eebb000386c7fa9e4c078db023cfa"
Oct 30 07:00:36.414140 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:36.414122    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p6bhn"
Oct 30 07:00:36.414235 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:36.414154    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p6bhn" event={"ID":"461b314c-252b-4479-9006-aca5c0d0f5e5","Type":"ContainerDied","Data":"def5facd684844f15d4be7209a2884967a67d7fe61e7f78a05005ac83687ae32"}
Oct 30 07:00:36.414235 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:36.414190    6224 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="def5facd684844f15d4be7209a2884967a67d7fe61e7f78a05005ac83687ae32"
Oct 30 07:00:36.565683 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:36.565661    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9w6r8"
Oct 30 07:00:36.589747 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:36.589721    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k7sr9"
Oct 30 07:00:36.630934 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:36.630909    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a04c40e2-2a8e-47eb-8551-a7fcdd0467a9-kube-api-access-ln9r2\" (UniqueName: \"kubernetes.io/projected/a04c40e2-2a8e-47eb-8551-a7fcdd0467a9-kube-api-access-ln9r2\") pod \"a04c40e2-2a8e-47eb-8551-a7fcdd0467a9\" (UID: \"a04c40e2-2a8e-47eb-8551-a7fcdd0467a9\") "
Oct 30 07:00:36.631076 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:36.630942    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a04c40e2-2a8e-47eb-8551-a7fcdd0467a9-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a04c40e2-2a8e-47eb-8551-a7fcdd0467a9-bundle\") pod \"a04c40e2-2a8e-47eb-8551-a7fcdd0467a9\" (UID: \"a04c40e2-2a8e-47eb-8551-a7fcdd0467a9\") "
Oct 30 07:00:36.631076 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:36.631045    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a04c40e2-2a8e-47eb-8551-a7fcdd0467a9-util\" (UniqueName: \"kubernetes.io/empty-dir/a04c40e2-2a8e-47eb-8551-a7fcdd0467a9-util\") pod \"a04c40e2-2a8e-47eb-8551-a7fcdd0467a9\" (UID: \"a04c40e2-2a8e-47eb-8551-a7fcdd0467a9\") "
Oct 30 07:00:36.631625 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:36.631603    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a04c40e2-2a8e-47eb-8551-a7fcdd0467a9-bundle" pod "a04c40e2-2a8e-47eb-8551-a7fcdd0467a9" (UID: "a04c40e2-2a8e-47eb-8551-a7fcdd0467a9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:00:36.633028 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:36.633001    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a04c40e2-2a8e-47eb-8551-a7fcdd0467a9-kube-api-access-ln9r2" pod "a04c40e2-2a8e-47eb-8551-a7fcdd0467a9" (UID: "a04c40e2-2a8e-47eb-8551-a7fcdd0467a9"). InnerVolumeSpecName "kube-api-access-ln9r2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:00:36.637131 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:36.637108    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a04c40e2-2a8e-47eb-8551-a7fcdd0467a9-util" pod "a04c40e2-2a8e-47eb-8551-a7fcdd0467a9" (UID: "a04c40e2-2a8e-47eb-8551-a7fcdd0467a9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:00:36.731643 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:36.731585    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b5a6fc4-47e1-4595-8ecd-216a575703a1-kube-api-access-rgtnf\" (UniqueName: \"kubernetes.io/projected/9b5a6fc4-47e1-4595-8ecd-216a575703a1-kube-api-access-rgtnf\") pod \"9b5a6fc4-47e1-4595-8ecd-216a575703a1\" (UID: \"9b5a6fc4-47e1-4595-8ecd-216a575703a1\") "
Oct 30 07:00:36.731643 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:36.731619    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b5a6fc4-47e1-4595-8ecd-216a575703a1-util\" (UniqueName: \"kubernetes.io/empty-dir/9b5a6fc4-47e1-4595-8ecd-216a575703a1-util\") pod \"9b5a6fc4-47e1-4595-8ecd-216a575703a1\" (UID: \"9b5a6fc4-47e1-4595-8ecd-216a575703a1\") "
Oct 30 07:00:36.731798 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:36.731644    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b5a6fc4-47e1-4595-8ecd-216a575703a1-bundle\" (UniqueName: \"kubernetes.io/empty-dir/9b5a6fc4-47e1-4595-8ecd-216a575703a1-bundle\") pod \"9b5a6fc4-47e1-4595-8ecd-216a575703a1\" (UID: \"9b5a6fc4-47e1-4595-8ecd-216a575703a1\") "
Oct 30 07:00:36.731798 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:36.731759    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ln9r2\" (UniqueName: \"kubernetes.io/projected/a04c40e2-2a8e-47eb-8551-a7fcdd0467a9-kube-api-access-ln9r2\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:00:36.731798 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:36.731769    6224 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a04c40e2-2a8e-47eb-8551-a7fcdd0467a9-bundle\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:00:36.731798 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:36.731778    6224 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a04c40e2-2a8e-47eb-8551-a7fcdd0467a9-util\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:00:36.732169 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:36.732142    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b5a6fc4-47e1-4595-8ecd-216a575703a1-bundle" pod "9b5a6fc4-47e1-4595-8ecd-216a575703a1" (UID: "9b5a6fc4-47e1-4595-8ecd-216a575703a1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:00:36.733592 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:36.733570    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b5a6fc4-47e1-4595-8ecd-216a575703a1-kube-api-access-rgtnf" pod "9b5a6fc4-47e1-4595-8ecd-216a575703a1" (UID: "9b5a6fc4-47e1-4595-8ecd-216a575703a1"). InnerVolumeSpecName "kube-api-access-rgtnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:00:36.735857 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:36.735837    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b5a6fc4-47e1-4595-8ecd-216a575703a1-util" pod "9b5a6fc4-47e1-4595-8ecd-216a575703a1" (UID: "9b5a6fc4-47e1-4595-8ecd-216a575703a1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:00:36.832568 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:36.832542    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgtnf\" (UniqueName: \"kubernetes.io/projected/9b5a6fc4-47e1-4595-8ecd-216a575703a1-kube-api-access-rgtnf\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:00:36.832568 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:36.832569    6224 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9b5a6fc4-47e1-4595-8ecd-216a575703a1-util\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:00:36.832716 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:36.832583    6224 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9b5a6fc4-47e1-4595-8ecd-216a575703a1-bundle\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:00:37.420066 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.420039    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9w6r8"
Oct 30 07:00:37.420444 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.420064    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9w6r8" event={"ID":"a04c40e2-2a8e-47eb-8551-a7fcdd0467a9","Type":"ContainerDied","Data":"e6e9938c17b53f2e13ec97afca85bd10ec5e56c1d5fda65ac0f50334e44e5ad8"}
Oct 30 07:00:37.420444 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.420115    6224 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6e9938c17b53f2e13ec97afca85bd10ec5e56c1d5fda65ac0f50334e44e5ad8"
Oct 30 07:00:37.422035 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.422016    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k7sr9" event={"ID":"9b5a6fc4-47e1-4595-8ecd-216a575703a1","Type":"ContainerDied","Data":"1e5f2d948990cb92f459e0eced2ec65f5ac3cb3a08a0a1899611f53e0d33d6f7"}
Oct 30 07:00:37.422154 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.422046    6224 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e5f2d948990cb92f459e0eced2ec65f5ac3cb3a08a0a1899611f53e0d33d6f7"
Oct 30 07:00:37.422154 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.422061    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k7sr9"
Oct 30 07:00:37.475450 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.475421    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-85f89f7bd7-72d8h"]
Oct 30 07:00:37.475666 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.475655    6224 cpu_manager.go:414] "containerMap: removing container" podUID="10da4324-a9c4-46ac-b9d7-6df46dd60905" containerName="util"
Oct 30 07:00:37.475714 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.475667    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10da4324-a9c4-46ac-b9d7-6df46dd60905" containerName="util"
Oct 30 07:00:37.475714 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.475675    6224 cpu_manager.go:414] "containerMap: removing container" podUID="461b314c-252b-4479-9006-aca5c0d0f5e5" containerName="util"
Oct 30 07:00:37.475714 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.475680    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="461b314c-252b-4479-9006-aca5c0d0f5e5" containerName="util"
Oct 30 07:00:37.475714 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.475690    6224 cpu_manager.go:414] "containerMap: removing container" podUID="9b5a6fc4-47e1-4595-8ecd-216a575703a1" containerName="pull"
Oct 30 07:00:37.475714 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.475695    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b5a6fc4-47e1-4595-8ecd-216a575703a1" containerName="pull"
Oct 30 07:00:37.475714 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.475702    6224 cpu_manager.go:414] "containerMap: removing container" podUID="419faae2-2f0c-4c9e-863c-a5968d8e0813" containerName="pull"
Oct 30 07:00:37.475714 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.475706    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="419faae2-2f0c-4c9e-863c-a5968d8e0813" containerName="pull"
Oct 30 07:00:37.475714 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.475712    6224 cpu_manager.go:414] "containerMap: removing container" podUID="1398f6ad-9bcf-47a7-ae51-8827d4cfe5f3" containerName="extract"
Oct 30 07:00:37.475714 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.475716    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1398f6ad-9bcf-47a7-ae51-8827d4cfe5f3" containerName="extract"
Oct 30 07:00:37.475978 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.475724    6224 cpu_manager.go:414] "containerMap: removing container" podUID="10da4324-a9c4-46ac-b9d7-6df46dd60905" containerName="extract"
Oct 30 07:00:37.475978 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.475729    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10da4324-a9c4-46ac-b9d7-6df46dd60905" containerName="extract"
Oct 30 07:00:37.475978 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.475737    6224 cpu_manager.go:414] "containerMap: removing container" podUID="44ad92ac-b90f-4d09-be1e-86be52cae40d" containerName="pull"
Oct 30 07:00:37.475978 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.475741    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44ad92ac-b90f-4d09-be1e-86be52cae40d" containerName="pull"
Oct 30 07:00:37.475978 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.475747    6224 cpu_manager.go:414] "containerMap: removing container" podUID="a04c40e2-2a8e-47eb-8551-a7fcdd0467a9" containerName="extract"
Oct 30 07:00:37.475978 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.475751    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a04c40e2-2a8e-47eb-8551-a7fcdd0467a9" containerName="extract"
Oct 30 07:00:37.475978 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.475759    6224 cpu_manager.go:414] "containerMap: removing container" podUID="419faae2-2f0c-4c9e-863c-a5968d8e0813" containerName="util"
Oct 30 07:00:37.475978 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.475763    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="419faae2-2f0c-4c9e-863c-a5968d8e0813" containerName="util"
Oct 30 07:00:37.475978 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.475769    6224 cpu_manager.go:414] "containerMap: removing container" podUID="10da4324-a9c4-46ac-b9d7-6df46dd60905" containerName="pull"
Oct 30 07:00:37.475978 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.475773    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10da4324-a9c4-46ac-b9d7-6df46dd60905" containerName="pull"
Oct 30 07:00:37.475978 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.475784    6224 cpu_manager.go:414] "containerMap: removing container" podUID="44ad92ac-b90f-4d09-be1e-86be52cae40d" containerName="util"
Oct 30 07:00:37.475978 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.475788    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44ad92ac-b90f-4d09-be1e-86be52cae40d" containerName="util"
Oct 30 07:00:37.475978 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.475794    6224 cpu_manager.go:414] "containerMap: removing container" podUID="461b314c-252b-4479-9006-aca5c0d0f5e5" containerName="pull"
Oct 30 07:00:37.475978 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.475797    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="461b314c-252b-4479-9006-aca5c0d0f5e5" containerName="pull"
Oct 30 07:00:37.475978 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.475820    6224 cpu_manager.go:414] "containerMap: removing container" podUID="44ad92ac-b90f-4d09-be1e-86be52cae40d" containerName="extract"
Oct 30 07:00:37.475978 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.475826    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44ad92ac-b90f-4d09-be1e-86be52cae40d" containerName="extract"
Oct 30 07:00:37.475978 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.475835    6224 cpu_manager.go:414] "containerMap: removing container" podUID="1398f6ad-9bcf-47a7-ae51-8827d4cfe5f3" containerName="util"
Oct 30 07:00:37.475978 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.475839    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1398f6ad-9bcf-47a7-ae51-8827d4cfe5f3" containerName="util"
Oct 30 07:00:37.475978 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.475845    6224 cpu_manager.go:414] "containerMap: removing container" podUID="1398f6ad-9bcf-47a7-ae51-8827d4cfe5f3" containerName="pull"
Oct 30 07:00:37.475978 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.475849    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1398f6ad-9bcf-47a7-ae51-8827d4cfe5f3" containerName="pull"
Oct 30 07:00:37.475978 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.475855    6224 cpu_manager.go:414] "containerMap: removing container" podUID="a04c40e2-2a8e-47eb-8551-a7fcdd0467a9" containerName="util"
Oct 30 07:00:37.475978 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.475859    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a04c40e2-2a8e-47eb-8551-a7fcdd0467a9" containerName="util"
Oct 30 07:00:37.475978 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.475865    6224 cpu_manager.go:414] "containerMap: removing container" podUID="9b5a6fc4-47e1-4595-8ecd-216a575703a1" containerName="extract"
Oct 30 07:00:37.475978 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.475869    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b5a6fc4-47e1-4595-8ecd-216a575703a1" containerName="extract"
Oct 30 07:00:37.475978 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.475874    6224 cpu_manager.go:414] "containerMap: removing container" podUID="a04c40e2-2a8e-47eb-8551-a7fcdd0467a9" containerName="pull"
Oct 30 07:00:37.475978 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.475878    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a04c40e2-2a8e-47eb-8551-a7fcdd0467a9" containerName="pull"
Oct 30 07:00:37.475978 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.475887    6224 cpu_manager.go:414] "containerMap: removing container" podUID="9b5a6fc4-47e1-4595-8ecd-216a575703a1" containerName="util"
Oct 30 07:00:37.475978 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.475892    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b5a6fc4-47e1-4595-8ecd-216a575703a1" containerName="util"
Oct 30 07:00:37.475978 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.475898    6224 cpu_manager.go:414] "containerMap: removing container" podUID="419faae2-2f0c-4c9e-863c-a5968d8e0813" containerName="extract"
Oct 30 07:00:37.475978 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.475902    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="419faae2-2f0c-4c9e-863c-a5968d8e0813" containerName="extract"
Oct 30 07:00:37.475978 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.475907    6224 cpu_manager.go:414] "containerMap: removing container" podUID="461b314c-252b-4479-9006-aca5c0d0f5e5" containerName="extract"
Oct 30 07:00:37.475978 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.475913    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="461b314c-252b-4479-9006-aca5c0d0f5e5" containerName="extract"
Oct 30 07:00:37.475978 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.475952    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="10da4324-a9c4-46ac-b9d7-6df46dd60905" containerName="extract"
Oct 30 07:00:37.475978 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.475960    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="1398f6ad-9bcf-47a7-ae51-8827d4cfe5f3" containerName="extract"
Oct 30 07:00:37.475978 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.475965    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="a04c40e2-2a8e-47eb-8551-a7fcdd0467a9" containerName="extract"
Oct 30 07:00:37.475978 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.475974    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="44ad92ac-b90f-4d09-be1e-86be52cae40d" containerName="extract"
Oct 30 07:00:37.475978 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.475979    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b5a6fc4-47e1-4595-8ecd-216a575703a1" containerName="extract"
Oct 30 07:00:37.475978 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.475986    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="419faae2-2f0c-4c9e-863c-a5968d8e0813" containerName="extract"
Oct 30 07:00:37.475978 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.475992    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="461b314c-252b-4479-9006-aca5c0d0f5e5" containerName="extract"
Oct 30 07:00:37.478201 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.478186    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85f89f7bd7-72d8h"
Oct 30 07:00:37.496068 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.496043    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 07:00:37.496068 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.496059    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 30 07:00:37.496214 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.496086    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 30 07:00:37.496214 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.496111    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-q568p\""
Oct 30 07:00:37.496343 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.496327    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 30 07:00:37.496392 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.496354    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 07:00:37.531145 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.531120    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-85f89f7bd7-72d8h"]
Oct 30 07:00:37.638969 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.638943    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7949fdce-8f66-4515-81e0-69e575ac9c25-manifests\") pod \"opendatahub-operator-controller-manager-85f89f7bd7-72d8h\" (UID: \"7949fdce-8f66-4515-81e0-69e575ac9c25\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85f89f7bd7-72d8h"
Oct 30 07:00:37.639102 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.638974    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7949fdce-8f66-4515-81e0-69e575ac9c25-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85f89f7bd7-72d8h\" (UID: \"7949fdce-8f66-4515-81e0-69e575ac9c25\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85f89f7bd7-72d8h"
Oct 30 07:00:37.639102 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.639010    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wbgc\" (UniqueName: \"kubernetes.io/projected/7949fdce-8f66-4515-81e0-69e575ac9c25-kube-api-access-8wbgc\") pod \"opendatahub-operator-controller-manager-85f89f7bd7-72d8h\" (UID: \"7949fdce-8f66-4515-81e0-69e575ac9c25\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85f89f7bd7-72d8h"
Oct 30 07:00:37.639173 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.639158    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7949fdce-8f66-4515-81e0-69e575ac9c25-webhook-cert\") pod \"opendatahub-operator-controller-manager-85f89f7bd7-72d8h\" (UID: \"7949fdce-8f66-4515-81e0-69e575ac9c25\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85f89f7bd7-72d8h"
Oct 30 07:00:37.639279 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.639214    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7949fdce-8f66-4515-81e0-69e575ac9c25-tmp\") pod \"opendatahub-operator-controller-manager-85f89f7bd7-72d8h\" (UID: \"7949fdce-8f66-4515-81e0-69e575ac9c25\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85f89f7bd7-72d8h"
Oct 30 07:00:37.639279 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.639231    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7949fdce-8f66-4515-81e0-69e575ac9c25-metrics-cert\") pod \"opendatahub-operator-controller-manager-85f89f7bd7-72d8h\" (UID: \"7949fdce-8f66-4515-81e0-69e575ac9c25\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85f89f7bd7-72d8h"
Oct 30 07:00:37.739706 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.739631    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7949fdce-8f66-4515-81e0-69e575ac9c25-webhook-cert\") pod \"opendatahub-operator-controller-manager-85f89f7bd7-72d8h\" (UID: \"7949fdce-8f66-4515-81e0-69e575ac9c25\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85f89f7bd7-72d8h"
Oct 30 07:00:37.739706 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.739667    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7949fdce-8f66-4515-81e0-69e575ac9c25-tmp\") pod \"opendatahub-operator-controller-manager-85f89f7bd7-72d8h\" (UID: \"7949fdce-8f66-4515-81e0-69e575ac9c25\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85f89f7bd7-72d8h"
Oct 30 07:00:37.739706 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.739688    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7949fdce-8f66-4515-81e0-69e575ac9c25-metrics-cert\") pod \"opendatahub-operator-controller-manager-85f89f7bd7-72d8h\" (UID: \"7949fdce-8f66-4515-81e0-69e575ac9c25\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85f89f7bd7-72d8h"
Oct 30 07:00:37.739967 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.739721    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7949fdce-8f66-4515-81e0-69e575ac9c25-manifests\") pod \"opendatahub-operator-controller-manager-85f89f7bd7-72d8h\" (UID: \"7949fdce-8f66-4515-81e0-69e575ac9c25\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85f89f7bd7-72d8h"
Oct 30 07:00:37.739967 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.739772    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7949fdce-8f66-4515-81e0-69e575ac9c25-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85f89f7bd7-72d8h\" (UID: \"7949fdce-8f66-4515-81e0-69e575ac9c25\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85f89f7bd7-72d8h"
Oct 30 07:00:37.739967 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.739855    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wbgc\" (UniqueName: \"kubernetes.io/projected/7949fdce-8f66-4515-81e0-69e575ac9c25-kube-api-access-8wbgc\") pod \"opendatahub-operator-controller-manager-85f89f7bd7-72d8h\" (UID: \"7949fdce-8f66-4515-81e0-69e575ac9c25\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85f89f7bd7-72d8h"
Oct 30 07:00:37.740275 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.740248    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7949fdce-8f66-4515-81e0-69e575ac9c25-manifests\") pod \"opendatahub-operator-controller-manager-85f89f7bd7-72d8h\" (UID: \"7949fdce-8f66-4515-81e0-69e575ac9c25\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85f89f7bd7-72d8h"
Oct 30 07:00:37.742071 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.742047    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7949fdce-8f66-4515-81e0-69e575ac9c25-tmp\") pod \"opendatahub-operator-controller-manager-85f89f7bd7-72d8h\" (UID: \"7949fdce-8f66-4515-81e0-69e575ac9c25\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85f89f7bd7-72d8h"
Oct 30 07:00:37.742366 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.742347    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7949fdce-8f66-4515-81e0-69e575ac9c25-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85f89f7bd7-72d8h\" (UID: \"7949fdce-8f66-4515-81e0-69e575ac9c25\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85f89f7bd7-72d8h"
Oct 30 07:00:37.742581 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.742567    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7949fdce-8f66-4515-81e0-69e575ac9c25-metrics-cert\") pod \"opendatahub-operator-controller-manager-85f89f7bd7-72d8h\" (UID: \"7949fdce-8f66-4515-81e0-69e575ac9c25\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85f89f7bd7-72d8h"
Oct 30 07:00:37.742883 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.742870    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7949fdce-8f66-4515-81e0-69e575ac9c25-webhook-cert\") pod \"opendatahub-operator-controller-manager-85f89f7bd7-72d8h\" (UID: \"7949fdce-8f66-4515-81e0-69e575ac9c25\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85f89f7bd7-72d8h"
Oct 30 07:00:37.765555 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.765530    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wbgc\" (UniqueName: \"kubernetes.io/projected/7949fdce-8f66-4515-81e0-69e575ac9c25-kube-api-access-8wbgc\") pod \"opendatahub-operator-controller-manager-85f89f7bd7-72d8h\" (UID: \"7949fdce-8f66-4515-81e0-69e575ac9c25\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85f89f7bd7-72d8h"
Oct 30 07:00:37.790351 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:37.790328    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85f89f7bd7-72d8h"
Oct 30 07:00:38.120115 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:38.120089    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-85f89f7bd7-72d8h"]
Oct 30 07:00:38.120687 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:00:38.120664    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7949fdce_8f66_4515_81e0_69e575ac9c25.slice/crio-5cab9023ebee60e7220567a795efdbb6b50185639c19d32ee87a395d3e8d2f22 WatchSource:0}: Error finding container 5cab9023ebee60e7220567a795efdbb6b50185639c19d32ee87a395d3e8d2f22: Status 404 returned error can't find the container with id 5cab9023ebee60e7220567a795efdbb6b50185639c19d32ee87a395d3e8d2f22
Oct 30 07:00:38.427594 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:38.427510    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85f89f7bd7-72d8h" event={"ID":"7949fdce-8f66-4515-81e0-69e575ac9c25","Type":"ContainerStarted","Data":"5cab9023ebee60e7220567a795efdbb6b50185639c19d32ee87a395d3e8d2f22"}
Oct 30 07:00:41.443171 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:41.443137    6224 generic.go:356] "Generic (PLEG): container finished" podID="7949fdce-8f66-4515-81e0-69e575ac9c25" containerID="7fa87009830fc185c4dc23eeaaef48ea2c1567b12483a31ff58a74288f81e3f6" exitCode=0
Oct 30 07:00:41.443555 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:41.443227    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85f89f7bd7-72d8h" event={"ID":"7949fdce-8f66-4515-81e0-69e575ac9c25","Type":"ContainerDied","Data":"7fa87009830fc185c4dc23eeaaef48ea2c1567b12483a31ff58a74288f81e3f6"}
Oct 30 07:00:42.042096 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:42.042068    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-724n7"]
Oct 30 07:00:42.044023 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:42.044009    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-724n7"
Oct 30 07:00:42.066457 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:42.066430    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Oct 30 07:00:42.067418 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:42.067399    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Oct 30 07:00:42.068318 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:42.067054    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-c94fc\""
Oct 30 07:00:42.072148 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:42.072125    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-724n7"]
Oct 30 07:00:42.173433 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:42.173404    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6a3d87c-603a-4159-a297-c24d4caaa437-tmp\") pod \"jobset-operator-674cd59dc5-724n7\" (UID: \"f6a3d87c-603a-4159-a297-c24d4caaa437\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-724n7"
Oct 30 07:00:42.173562 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:42.173440    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4w5k\" (UniqueName: \"kubernetes.io/projected/f6a3d87c-603a-4159-a297-c24d4caaa437-kube-api-access-f4w5k\") pod \"jobset-operator-674cd59dc5-724n7\" (UID: \"f6a3d87c-603a-4159-a297-c24d4caaa437\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-724n7"
Oct 30 07:00:42.274555 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:42.274514    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6a3d87c-603a-4159-a297-c24d4caaa437-tmp\") pod \"jobset-operator-674cd59dc5-724n7\" (UID: \"f6a3d87c-603a-4159-a297-c24d4caaa437\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-724n7"
Oct 30 07:00:42.274755 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:42.274572    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4w5k\" (UniqueName: \"kubernetes.io/projected/f6a3d87c-603a-4159-a297-c24d4caaa437-kube-api-access-f4w5k\") pod \"jobset-operator-674cd59dc5-724n7\" (UID: \"f6a3d87c-603a-4159-a297-c24d4caaa437\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-724n7"
Oct 30 07:00:42.274981 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:42.274956    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6a3d87c-603a-4159-a297-c24d4caaa437-tmp\") pod \"jobset-operator-674cd59dc5-724n7\" (UID: \"f6a3d87c-603a-4159-a297-c24d4caaa437\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-724n7"
Oct 30 07:00:42.301381 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:42.301319    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4w5k\" (UniqueName: \"kubernetes.io/projected/f6a3d87c-603a-4159-a297-c24d4caaa437-kube-api-access-f4w5k\") pod \"jobset-operator-674cd59dc5-724n7\" (UID: \"f6a3d87c-603a-4159-a297-c24d4caaa437\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-724n7"
Oct 30 07:00:42.367614 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:42.367583    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-724n7"
Oct 30 07:00:42.450419 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:42.450222    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85f89f7bd7-72d8h" event={"ID":"7949fdce-8f66-4515-81e0-69e575ac9c25","Type":"ContainerStarted","Data":"9d02ebda3fc30066dc45fd7e7282599ee7e80546ed2aee9fe74ef5204a0ee0d4"}
Oct 30 07:00:42.450419 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:42.450395    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85f89f7bd7-72d8h"
Oct 30 07:00:42.517787 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:42.517737    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85f89f7bd7-72d8h" podStartSLOduration=2.032694577 podStartE2EDuration="5.517723882s" totalImagesPullingTime="2.994943698s" totalInitContainerRuntime="490.085607ms" isStatefulPod=false podCreationTimestamp="2026-10-30 07:00:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:00:42.516734675 +0000 UTC m=+201.126141648" watchObservedRunningTime="2026-10-30 07:00:42.517723882 +0000 UTC m=+201.127130852"
Oct 30 07:00:42.667301 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:42.667274    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-724n7"]
Oct 30 07:00:42.668408 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:00:42.668364    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf6a3d87c_603a_4159_a297_c24d4caaa437.slice/crio-e491c72f4ee7d7c6b8115d097f1ede161dc118005c4f6e8067d3ee8939a3f4b9 WatchSource:0}: Error finding container e491c72f4ee7d7c6b8115d097f1ede161dc118005c4f6e8067d3ee8939a3f4b9: Status 404 returned error can't find the container with id e491c72f4ee7d7c6b8115d097f1ede161dc118005c4f6e8067d3ee8939a3f4b9
Oct 30 07:00:43.455033 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:43.454994    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-724n7" event={"ID":"f6a3d87c-603a-4159-a297-c24d4caaa437","Type":"ContainerStarted","Data":"e491c72f4ee7d7c6b8115d097f1ede161dc118005c4f6e8067d3ee8939a3f4b9"}
Oct 30 07:00:45.466229 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:45.466145    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-724n7" event={"ID":"f6a3d87c-603a-4159-a297-c24d4caaa437","Type":"ContainerStarted","Data":"a2a9485393250e95f26925a84391eb4d0a8097c4c4813e9a9393e75b65411b7b"}
Oct 30 07:00:45.517990 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:45.517941    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-724n7" podStartSLOduration=1.10951279 podStartE2EDuration="3.517928229s" totalImagesPullingTime="2.408415439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:00:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:00:45.516714618 +0000 UTC m=+204.126121589" watchObservedRunningTime="2026-10-30 07:00:45.517928229 +0000 UTC m=+204.127335196"
Oct 30 07:00:52.750339 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:52.750304    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-8stfv"]
Oct 30 07:00:52.754223 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:52.754206    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-8stfv"
Oct 30 07:00:52.788198 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:52.788172    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 07:00:52.788198 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:52.788187    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-gqcbv\""
Oct 30 07:00:52.788198 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:52.788174    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 07:00:52.788947 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:52.788928    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-8stfv"]
Oct 30 07:00:52.864221 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:52.864198    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wlkk\" (UniqueName: \"kubernetes.io/projected/1270d903-7234-4ffb-825c-b4c71a4e9b86-kube-api-access-8wlkk\") pod \"limitador-operator-controller-manager-5778bb4449-8stfv\" (UID: \"1270d903-7234-4ffb-825c-b4c71a4e9b86\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-8stfv"
Oct 30 07:00:52.964563 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:52.964534    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wlkk\" (UniqueName: \"kubernetes.io/projected/1270d903-7234-4ffb-825c-b4c71a4e9b86-kube-api-access-8wlkk\") pod \"limitador-operator-controller-manager-5778bb4449-8stfv\" (UID: \"1270d903-7234-4ffb-825c-b4c71a4e9b86\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-8stfv"
Oct 30 07:00:52.991858 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:52.991833    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wlkk\" (UniqueName: \"kubernetes.io/projected/1270d903-7234-4ffb-825c-b4c71a4e9b86-kube-api-access-8wlkk\") pod \"limitador-operator-controller-manager-5778bb4449-8stfv\" (UID: \"1270d903-7234-4ffb-825c-b4c71a4e9b86\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-8stfv"
Oct 30 07:00:53.065656 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:53.065633    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-8stfv"
Oct 30 07:00:53.359874 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:53.359181    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-8stfv"]
Oct 30 07:00:53.360393 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:00:53.360364    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1270d903_7234_4ffb_825c_b4c71a4e9b86.slice/crio-f2bad08c4e1ac25c97055cb2956e715be45cd9dbe803cd23f37e86c5317b40cb WatchSource:0}: Error finding container f2bad08c4e1ac25c97055cb2956e715be45cd9dbe803cd23f37e86c5317b40cb: Status 404 returned error can't find the container with id f2bad08c4e1ac25c97055cb2956e715be45cd9dbe803cd23f37e86c5317b40cb
Oct 30 07:00:53.457565 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:53.457543    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85f89f7bd7-72d8h"
Oct 30 07:00:53.499493 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:53.499466    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-8stfv" event={"ID":"1270d903-7234-4ffb-825c-b4c71a4e9b86","Type":"ContainerStarted","Data":"f2bad08c4e1ac25c97055cb2956e715be45cd9dbe803cd23f37e86c5317b40cb"}
Oct 30 07:00:56.515416 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:56.515373    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-8stfv" event={"ID":"1270d903-7234-4ffb-825c-b4c71a4e9b86","Type":"ContainerStarted","Data":"601cadecf9360050668ae0767e08885aa3198afc09cca69bb07b185c0b554346"}
Oct 30 07:00:56.515796 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:56.515498    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-8stfv"
Oct 30 07:00:56.579023 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:56.578978    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-8stfv" podStartSLOduration=2.051487609 podStartE2EDuration="4.578967403s" totalImagesPullingTime="2.527479794s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:00:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:00:56.578210996 +0000 UTC m=+215.187617965" watchObservedRunningTime="2026-10-30 07:00:56.578967403 +0000 UTC m=+215.188374373"
Oct 30 07:00:57.169682 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:57.169650    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-llwcb"]
Oct 30 07:00:57.171719 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:57.171704    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-llwcb"
Oct 30 07:00:57.200098 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:57.200074    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee14649b-e03c-42bb-b0e5-e9697f336bdb-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-llwcb\" (UID: \"ee14649b-e03c-42bb-b0e5-e9697f336bdb\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-llwcb"
Oct 30 07:00:57.200208 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:57.200115    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjckc\" (UniqueName: \"kubernetes.io/projected/ee14649b-e03c-42bb-b0e5-e9697f336bdb-kube-api-access-sjckc\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-llwcb\" (UID: \"ee14649b-e03c-42bb-b0e5-e9697f336bdb\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-llwcb"
Oct 30 07:00:57.200664 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:57.200641    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-llwcb"]
Oct 30 07:00:57.212857 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:57.212832    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Oct 30 07:00:57.212857 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:57.212848    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Oct 30 07:00:57.213015 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:57.212834    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-g8jk2\""
Oct 30 07:00:57.300848 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:57.300791    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjckc\" (UniqueName: \"kubernetes.io/projected/ee14649b-e03c-42bb-b0e5-e9697f336bdb-kube-api-access-sjckc\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-llwcb\" (UID: \"ee14649b-e03c-42bb-b0e5-e9697f336bdb\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-llwcb"
Oct 30 07:00:57.301012 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:57.300902    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee14649b-e03c-42bb-b0e5-e9697f336bdb-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-llwcb\" (UID: \"ee14649b-e03c-42bb-b0e5-e9697f336bdb\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-llwcb"
Oct 30 07:00:57.301264 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:57.301245    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee14649b-e03c-42bb-b0e5-e9697f336bdb-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-llwcb\" (UID: \"ee14649b-e03c-42bb-b0e5-e9697f336bdb\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-llwcb"
Oct 30 07:00:57.331418 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:57.331392    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjckc\" (UniqueName: \"kubernetes.io/projected/ee14649b-e03c-42bb-b0e5-e9697f336bdb-kube-api-access-sjckc\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-llwcb\" (UID: \"ee14649b-e03c-42bb-b0e5-e9697f336bdb\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-llwcb"
Oct 30 07:00:57.481535 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:57.481466    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-llwcb"
Oct 30 07:00:57.800800 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:57.800772    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-llwcb"]
Oct 30 07:00:57.801526 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:00:57.801502    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee14649b_e03c_42bb_b0e5_e9697f336bdb.slice/crio-895fe29fd9623a4c83315470e772ad441a13c9d6c917ca7420eb8b7e94af2e78 WatchSource:0}: Error finding container 895fe29fd9623a4c83315470e772ad441a13c9d6c917ca7420eb8b7e94af2e78: Status 404 returned error can't find the container with id 895fe29fd9623a4c83315470e772ad441a13c9d6c917ca7420eb8b7e94af2e78
Oct 30 07:00:58.525412 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:58.525376    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-llwcb" event={"ID":"ee14649b-e03c-42bb-b0e5-e9697f336bdb","Type":"ContainerStarted","Data":"895fe29fd9623a4c83315470e772ad441a13c9d6c917ca7420eb8b7e94af2e78"}
Oct 30 07:00:59.536587 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:59.536549    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-llwcb" event={"ID":"ee14649b-e03c-42bb-b0e5-e9697f336bdb","Type":"ContainerStarted","Data":"53f9a14912b06b8823ee370ecb30315163b74e28118bf510ff59c797a0c7507d"}
Oct 30 07:00:59.615353 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:00:59.615299    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-llwcb" podStartSLOduration=1.278386113 podStartE2EDuration="2.615280989s" totalImagesPullingTime="1.336894876s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:00:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:00:59.598136671 +0000 UTC m=+218.207543643" watchObservedRunningTime="2026-10-30 07:00:59.615280989 +0000 UTC m=+218.224687964"
Oct 30 07:01:01.857276 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:01:01.857241    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:01:02.500917 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:02.500880    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-krwg6"]
Oct 30 07:01:02.502932 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:02.502916    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-krwg6"
Oct 30 07:01:02.523408 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:02.523385    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-krwg6"]
Oct 30 07:01:02.536408 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:02.536382    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 30 07:01:02.536513 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:02.536409    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 30 07:01:02.539814 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:02.539782    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf4ps\" (UniqueName: \"kubernetes.io/projected/18179f43-73ec-471d-bfe1-d096c4f6e2f8-kube-api-access-gf4ps\") pod \"cert-manager-webhook-5dbbbf75cc-krwg6\" (UID: \"18179f43-73ec-471d-bfe1-d096c4f6e2f8\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-krwg6"
Oct 30 07:01:02.539904 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:02.539843    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/18179f43-73ec-471d-bfe1-d096c4f6e2f8-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-krwg6\" (UID: \"18179f43-73ec-471d-bfe1-d096c4f6e2f8\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-krwg6"
Oct 30 07:01:02.564236 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:02.564214    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-b6czf\""
Oct 30 07:01:02.640550 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:02.640508    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf4ps\" (UniqueName: \"kubernetes.io/projected/18179f43-73ec-471d-bfe1-d096c4f6e2f8-kube-api-access-gf4ps\") pod \"cert-manager-webhook-5dbbbf75cc-krwg6\" (UID: \"18179f43-73ec-471d-bfe1-d096c4f6e2f8\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-krwg6"
Oct 30 07:01:02.640706 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:02.640572    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/18179f43-73ec-471d-bfe1-d096c4f6e2f8-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-krwg6\" (UID: \"18179f43-73ec-471d-bfe1-d096c4f6e2f8\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-krwg6"
Oct 30 07:01:02.665547 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:02.665510    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/18179f43-73ec-471d-bfe1-d096c4f6e2f8-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-krwg6\" (UID: \"18179f43-73ec-471d-bfe1-d096c4f6e2f8\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-krwg6"
Oct 30 07:01:02.665679 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:02.665565    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf4ps\" (UniqueName: \"kubernetes.io/projected/18179f43-73ec-471d-bfe1-d096c4f6e2f8-kube-api-access-gf4ps\") pod \"cert-manager-webhook-5dbbbf75cc-krwg6\" (UID: \"18179f43-73ec-471d-bfe1-d096c4f6e2f8\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-krwg6"
Oct 30 07:01:02.812611 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:02.812574    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-krwg6"
Oct 30 07:01:03.107013 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:03.106986    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-krwg6"]
Oct 30 07:01:03.108163 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:01:03.108135    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18179f43_73ec_471d_bfe1_d096c4f6e2f8.slice/crio-6b62dd30bb019d10fbb733253fc0e79fa8e68a1c3aa1040766762fa602b235d2 WatchSource:0}: Error finding container 6b62dd30bb019d10fbb733253fc0e79fa8e68a1c3aa1040766762fa602b235d2: Status 404 returned error can't find the container with id 6b62dd30bb019d10fbb733253fc0e79fa8e68a1c3aa1040766762fa602b235d2
Oct 30 07:01:03.553040 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:03.553008    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-krwg6" event={"ID":"18179f43-73ec-471d-bfe1-d096c4f6e2f8","Type":"ContainerStarted","Data":"6b62dd30bb019d10fbb733253fc0e79fa8e68a1c3aa1040766762fa602b235d2"}
Oct 30 07:01:03.852314 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:03.852215    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-gwz2f"]
Oct 30 07:01:03.855330 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:03.855306    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-gwz2f"
Oct 30 07:01:03.889595 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:03.889471    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Oct 30 07:01:03.889595 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:03.889571    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-7ms8x\""
Oct 30 07:01:03.891332 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:03.891301    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-gwz2f"]
Oct 30 07:01:03.952472 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:03.952437    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-9g5ss"]
Oct 30 07:01:03.953146 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:03.953124    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th726\" (UniqueName: \"kubernetes.io/projected/c84bb85f-b5d5-4934-b67f-0589d63fc861-kube-api-access-th726\") pod \"dns-operator-controller-manager-8557cbdb84-gwz2f\" (UID: \"c84bb85f-b5d5-4934-b67f-0589d63fc861\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-gwz2f"
Oct 30 07:01:03.956038 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:03.956020    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9g5ss"
Oct 30 07:01:03.982684 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:03.982657    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-qgrvn\""
Oct 30 07:01:03.984416 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:03.984391    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-9g5ss"]
Oct 30 07:01:04.053956 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:04.053915    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th726\" (UniqueName: \"kubernetes.io/projected/c84bb85f-b5d5-4934-b67f-0589d63fc861-kube-api-access-th726\") pod \"dns-operator-controller-manager-8557cbdb84-gwz2f\" (UID: \"c84bb85f-b5d5-4934-b67f-0589d63fc861\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-gwz2f"
Oct 30 07:01:04.054129 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:04.053977    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f1d7b3f9-d4c3-4a7c-96fc-29d73b8128c4-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-9g5ss\" (UID: \"f1d7b3f9-d4c3-4a7c-96fc-29d73b8128c4\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9g5ss"
Oct 30 07:01:04.054129 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:04.054094    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k27h6\" (UniqueName: \"kubernetes.io/projected/f1d7b3f9-d4c3-4a7c-96fc-29d73b8128c4-kube-api-access-k27h6\") pod \"cert-manager-cainjector-5c57bbd4fc-9g5ss\" (UID: \"f1d7b3f9-d4c3-4a7c-96fc-29d73b8128c4\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9g5ss"
Oct 30 07:01:04.082837 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:04.082794    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-th726\" (UniqueName: \"kubernetes.io/projected/c84bb85f-b5d5-4934-b67f-0589d63fc861-kube-api-access-th726\") pod \"dns-operator-controller-manager-8557cbdb84-gwz2f\" (UID: \"c84bb85f-b5d5-4934-b67f-0589d63fc861\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-gwz2f"
Oct 30 07:01:04.155227 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:04.155129    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f1d7b3f9-d4c3-4a7c-96fc-29d73b8128c4-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-9g5ss\" (UID: \"f1d7b3f9-d4c3-4a7c-96fc-29d73b8128c4\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9g5ss"
Oct 30 07:01:04.155703 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:04.155263    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k27h6\" (UniqueName: \"kubernetes.io/projected/f1d7b3f9-d4c3-4a7c-96fc-29d73b8128c4-kube-api-access-k27h6\") pod \"cert-manager-cainjector-5c57bbd4fc-9g5ss\" (UID: \"f1d7b3f9-d4c3-4a7c-96fc-29d73b8128c4\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9g5ss"
Oct 30 07:01:04.168481 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:04.168452    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-gwz2f"
Oct 30 07:01:04.180934 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:04.180904    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k27h6\" (UniqueName: \"kubernetes.io/projected/f1d7b3f9-d4c3-4a7c-96fc-29d73b8128c4-kube-api-access-k27h6\") pod \"cert-manager-cainjector-5c57bbd4fc-9g5ss\" (UID: \"f1d7b3f9-d4c3-4a7c-96fc-29d73b8128c4\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9g5ss"
Oct 30 07:01:04.181067 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:04.180983    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f1d7b3f9-d4c3-4a7c-96fc-29d73b8128c4-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-9g5ss\" (UID: \"f1d7b3f9-d4c3-4a7c-96fc-29d73b8128c4\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9g5ss"
Oct 30 07:01:04.268704 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:04.268668    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9g5ss"
Oct 30 07:01:04.482307 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:04.482279    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-gwz2f"]
Oct 30 07:01:04.483603 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:01:04.483560    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc84bb85f_b5d5_4934_b67f_0589d63fc861.slice/crio-d355005e69f83c9ab62504bb9498b737a3dfaa6804ad00484609a988f5259467 WatchSource:0}: Error finding container d355005e69f83c9ab62504bb9498b737a3dfaa6804ad00484609a988f5259467: Status 404 returned error can't find the container with id d355005e69f83c9ab62504bb9498b737a3dfaa6804ad00484609a988f5259467
Oct 30 07:01:04.560177 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:04.560136    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-gwz2f" event={"ID":"c84bb85f-b5d5-4934-b67f-0589d63fc861","Type":"ContainerStarted","Data":"d355005e69f83c9ab62504bb9498b737a3dfaa6804ad00484609a988f5259467"}
Oct 30 07:01:04.576512 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:04.576451    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-9g5ss"]
Oct 30 07:01:04.579324 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:01:04.579065    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1d7b3f9_d4c3_4a7c_96fc_29d73b8128c4.slice/crio-8ffd1391c7a01b5d4989c7b859578fda47bc609a050aa51325c84179bf4d36d2 WatchSource:0}: Error finding container 8ffd1391c7a01b5d4989c7b859578fda47bc609a050aa51325c84179bf4d36d2: Status 404 returned error can't find the container with id 8ffd1391c7a01b5d4989c7b859578fda47bc609a050aa51325c84179bf4d36d2
Oct 30 07:01:05.567750 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:05.567713    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9g5ss" event={"ID":"f1d7b3f9-d4c3-4a7c-96fc-29d73b8128c4","Type":"ContainerStarted","Data":"8ffd1391c7a01b5d4989c7b859578fda47bc609a050aa51325c84179bf4d36d2"}
Oct 30 07:01:06.574412 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:06.574370    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-krwg6" event={"ID":"18179f43-73ec-471d-bfe1-d096c4f6e2f8","Type":"ContainerStarted","Data":"527ba487e7cc64076b3eb6424545fd30df0cf4695ed00679ac90dc0a508b4c54"}
Oct 30 07:01:06.574911 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:06.574439    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-krwg6"
Oct 30 07:01:06.576398 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:06.576364    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9g5ss" event={"ID":"f1d7b3f9-d4c3-4a7c-96fc-29d73b8128c4","Type":"ContainerStarted","Data":"19d789419f67d09e8e05fd4477f5ffa007ac0ca6ba045ae448ab5c21723e8033"}
Oct 30 07:01:06.623248 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:06.623200    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-krwg6" podStartSLOduration=1.686442167 podStartE2EDuration="4.623187611s" totalImagesPullingTime="2.936745444s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:01:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:01:06.622595092 +0000 UTC m=+225.232002082" watchObservedRunningTime="2026-10-30 07:01:06.623187611 +0000 UTC m=+225.232594580"
Oct 30 07:01:06.677337 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:06.677290    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9g5ss" podStartSLOduration=2.082061738 podStartE2EDuration="3.677277671s" totalImagesPullingTime="1.595215933s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:01:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:01:06.676208849 +0000 UTC m=+225.285615819" watchObservedRunningTime="2026-10-30 07:01:06.677277671 +0000 UTC m=+225.286684641"
Oct 30 07:01:07.501522 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:07.501492    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-zbc8w"]
Oct 30 07:01:07.503632 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:07.503614    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zbc8w"
Oct 30 07:01:07.522194 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:07.522174    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-8stfv"
Oct 30 07:01:07.523535 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:07.523488    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Oct 30 07:01:07.523535 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:07.523503    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Oct 30 07:01:07.537584 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:07.537566    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-lfn59\""
Oct 30 07:01:07.538865 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:07.538848    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-zbc8w"]
Oct 30 07:01:07.585263 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:07.585241    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e44c3a31-a231-4a77-903b-ad4afbecaae0-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-zbc8w\" (UID: \"e44c3a31-a231-4a77-903b-ad4afbecaae0\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zbc8w"
Oct 30 07:01:07.585626 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:07.585280    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmxdd\" (UniqueName: \"kubernetes.io/projected/e44c3a31-a231-4a77-903b-ad4afbecaae0-kube-api-access-xmxdd\") pod \"kuadrant-console-plugin-66c4fb575d-zbc8w\" (UID: \"e44c3a31-a231-4a77-903b-ad4afbecaae0\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zbc8w"
Oct 30 07:01:07.585626 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:07.585484    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e44c3a31-a231-4a77-903b-ad4afbecaae0-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-zbc8w\" (UID: \"e44c3a31-a231-4a77-903b-ad4afbecaae0\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zbc8w"
Oct 30 07:01:07.686152 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:07.686121    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e44c3a31-a231-4a77-903b-ad4afbecaae0-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-zbc8w\" (UID: \"e44c3a31-a231-4a77-903b-ad4afbecaae0\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zbc8w"
Oct 30 07:01:07.686312 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:07.686178    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmxdd\" (UniqueName: \"kubernetes.io/projected/e44c3a31-a231-4a77-903b-ad4afbecaae0-kube-api-access-xmxdd\") pod \"kuadrant-console-plugin-66c4fb575d-zbc8w\" (UID: \"e44c3a31-a231-4a77-903b-ad4afbecaae0\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zbc8w"
Oct 30 07:01:07.686312 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:07.686217    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e44c3a31-a231-4a77-903b-ad4afbecaae0-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-zbc8w\" (UID: \"e44c3a31-a231-4a77-903b-ad4afbecaae0\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zbc8w"
Oct 30 07:01:07.686921 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:07.686892    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e44c3a31-a231-4a77-903b-ad4afbecaae0-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-zbc8w\" (UID: \"e44c3a31-a231-4a77-903b-ad4afbecaae0\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zbc8w"
Oct 30 07:01:07.688574 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:07.688552    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e44c3a31-a231-4a77-903b-ad4afbecaae0-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-zbc8w\" (UID: \"e44c3a31-a231-4a77-903b-ad4afbecaae0\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zbc8w"
Oct 30 07:01:07.710637 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:07.710611    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmxdd\" (UniqueName: \"kubernetes.io/projected/e44c3a31-a231-4a77-903b-ad4afbecaae0-kube-api-access-xmxdd\") pod \"kuadrant-console-plugin-66c4fb575d-zbc8w\" (UID: \"e44c3a31-a231-4a77-903b-ad4afbecaae0\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zbc8w"
Oct 30 07:01:07.813088 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:07.813062    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zbc8w"
Oct 30 07:01:08.111793 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:08.111760    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-zbc8w"]
Oct 30 07:01:08.586650 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:08.586609    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zbc8w" event={"ID":"e44c3a31-a231-4a77-903b-ad4afbecaae0","Type":"ContainerStarted","Data":"b3a6227279eb7cce0877a4c32fa87bdc85e808064f29adff3af21e6a325cdae9"}
Oct 30 07:01:09.594051 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:09.594010    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-gwz2f" event={"ID":"c84bb85f-b5d5-4934-b67f-0589d63fc861","Type":"ContainerStarted","Data":"9ceecdd55c4355c117b35ee7b116c9e2459fef2b12cb67b5fcda5d455e4271e4"}
Oct 30 07:01:09.594535 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:09.594164    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-gwz2f"
Oct 30 07:01:09.638701 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:09.638657    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-gwz2f" podStartSLOduration=1.889076384 podStartE2EDuration="6.638641893s" totalImagesPullingTime="4.749565509s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:01:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:01:09.638121702 +0000 UTC m=+228.247528671" watchObservedRunningTime="2026-10-30 07:01:09.638641893 +0000 UTC m=+228.248048860"
Oct 30 07:01:12.583388 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:12.583357    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-krwg6"
Oct 30 07:01:13.616756 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:13.616718    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zbc8w" event={"ID":"e44c3a31-a231-4a77-903b-ad4afbecaae0","Type":"ContainerStarted","Data":"68079def73e71d8310240ac0ce843518c4acdfe42943580a386d3d5a618693f4"}
Oct 30 07:01:13.657695 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:13.657663    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-54599bd898-89xfk"]
Oct 30 07:01:13.662081 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:13.662064    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-54599bd898-89xfk"
Oct 30 07:01:13.669666 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:13.669638    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-54599bd898-89xfk"]
Oct 30 07:01:13.678363 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:13.678314    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zbc8w" podStartSLOduration=1.815072486 podStartE2EDuration="6.678297438s" totalImagesPullingTime="4.863224952s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:01:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:01:13.677931406 +0000 UTC m=+232.287338377" watchObservedRunningTime="2026-10-30 07:01:13.678297438 +0000 UTC m=+232.287704409"
Oct 30 07:01:13.690568 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:13.690463    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 07:01:13.690568 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:13.690475    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 07:01:13.707585 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:13.707553    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-vg4t5\""
Oct 30 07:01:13.707915 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:13.707820    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 30 07:01:13.707915 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:13.707878    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 30 07:01:13.740664 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:13.740634    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/504b6d22-335a-4ba3-b724-da2c4880543d-tls-certs\") pod \"kube-auth-proxy-54599bd898-89xfk\" (UID: \"504b6d22-335a-4ba3-b724-da2c4880543d\") " pod="openshift-ingress/kube-auth-proxy-54599bd898-89xfk"
Oct 30 07:01:13.740840 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:13.740673    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/504b6d22-335a-4ba3-b724-da2c4880543d-tmp\") pod \"kube-auth-proxy-54599bd898-89xfk\" (UID: \"504b6d22-335a-4ba3-b724-da2c4880543d\") " pod="openshift-ingress/kube-auth-proxy-54599bd898-89xfk"
Oct 30 07:01:13.740840 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:13.740777    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsv5t\" (UniqueName: \"kubernetes.io/projected/504b6d22-335a-4ba3-b724-da2c4880543d-kube-api-access-zsv5t\") pod \"kube-auth-proxy-54599bd898-89xfk\" (UID: \"504b6d22-335a-4ba3-b724-da2c4880543d\") " pod="openshift-ingress/kube-auth-proxy-54599bd898-89xfk"
Oct 30 07:01:13.842320 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:13.842289    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/504b6d22-335a-4ba3-b724-da2c4880543d-tls-certs\") pod \"kube-auth-proxy-54599bd898-89xfk\" (UID: \"504b6d22-335a-4ba3-b724-da2c4880543d\") " pod="openshift-ingress/kube-auth-proxy-54599bd898-89xfk"
Oct 30 07:01:13.842475 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:13.842330    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/504b6d22-335a-4ba3-b724-da2c4880543d-tmp\") pod \"kube-auth-proxy-54599bd898-89xfk\" (UID: \"504b6d22-335a-4ba3-b724-da2c4880543d\") " pod="openshift-ingress/kube-auth-proxy-54599bd898-89xfk"
Oct 30 07:01:13.842475 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:13.842378    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsv5t\" (UniqueName: \"kubernetes.io/projected/504b6d22-335a-4ba3-b724-da2c4880543d-kube-api-access-zsv5t\") pod \"kube-auth-proxy-54599bd898-89xfk\" (UID: \"504b6d22-335a-4ba3-b724-da2c4880543d\") " pod="openshift-ingress/kube-auth-proxy-54599bd898-89xfk"
Oct 30 07:01:13.844778 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:13.844747    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/504b6d22-335a-4ba3-b724-da2c4880543d-tmp\") pod \"kube-auth-proxy-54599bd898-89xfk\" (UID: \"504b6d22-335a-4ba3-b724-da2c4880543d\") " pod="openshift-ingress/kube-auth-proxy-54599bd898-89xfk"
Oct 30 07:01:13.844923 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:13.844880    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/504b6d22-335a-4ba3-b724-da2c4880543d-tls-certs\") pod \"kube-auth-proxy-54599bd898-89xfk\" (UID: \"504b6d22-335a-4ba3-b724-da2c4880543d\") " pod="openshift-ingress/kube-auth-proxy-54599bd898-89xfk"
Oct 30 07:01:13.879784 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:13.879717    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsv5t\" (UniqueName: \"kubernetes.io/projected/504b6d22-335a-4ba3-b724-da2c4880543d-kube-api-access-zsv5t\") pod \"kube-auth-proxy-54599bd898-89xfk\" (UID: \"504b6d22-335a-4ba3-b724-da2c4880543d\") " pod="openshift-ingress/kube-auth-proxy-54599bd898-89xfk"
Oct 30 07:01:13.975282 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:13.975248    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-54599bd898-89xfk"
Oct 30 07:01:14.284258 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:14.284230    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-54599bd898-89xfk"]
Oct 30 07:01:14.286298 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:01:14.286270    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod504b6d22_335a_4ba3_b724_da2c4880543d.slice/crio-4cb0c51921775191d9ec00aa828ea14a33df127a9dfabf2b0f927b20f4abc80c WatchSource:0}: Error finding container 4cb0c51921775191d9ec00aa828ea14a33df127a9dfabf2b0f927b20f4abc80c: Status 404 returned error can't find the container with id 4cb0c51921775191d9ec00aa828ea14a33df127a9dfabf2b0f927b20f4abc80c
Oct 30 07:01:14.627847 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:14.627742    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-54599bd898-89xfk" event={"ID":"504b6d22-335a-4ba3-b724-da2c4880543d","Type":"ContainerStarted","Data":"4cb0c51921775191d9ec00aa828ea14a33df127a9dfabf2b0f927b20f4abc80c"}
Oct 30 07:01:14.986672 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:14.986590    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:01:14.990385 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:14.990362    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-9m92k"
Oct 30 07:01:15.027120 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:15.027089    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 30 07:01:15.028838 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:15.028793    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:01:15.053749 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:15.053712    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkmhv\" (UniqueName: \"kubernetes.io/projected/e1113d78-e48d-48b2-aab4-c3f859f69eae-kube-api-access-dkmhv\") pod \"limitador-limitador-7c9b87d574-9m92k\" (UID: \"e1113d78-e48d-48b2-aab4-c3f859f69eae\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-9m92k"
Oct 30 07:01:15.053924 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:15.053847    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e1113d78-e48d-48b2-aab4-c3f859f69eae-config-file\") pod \"limitador-limitador-7c9b87d574-9m92k\" (UID: \"e1113d78-e48d-48b2-aab4-c3f859f69eae\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-9m92k"
Oct 30 07:01:15.081544 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:15.081419    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:01:15.155185 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:15.155118    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkmhv\" (UniqueName: \"kubernetes.io/projected/e1113d78-e48d-48b2-aab4-c3f859f69eae-kube-api-access-dkmhv\") pod \"limitador-limitador-7c9b87d574-9m92k\" (UID: \"e1113d78-e48d-48b2-aab4-c3f859f69eae\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-9m92k"
Oct 30 07:01:15.155185 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:15.155184    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e1113d78-e48d-48b2-aab4-c3f859f69eae-config-file\") pod \"limitador-limitador-7c9b87d574-9m92k\" (UID: \"e1113d78-e48d-48b2-aab4-c3f859f69eae\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-9m92k"
Oct 30 07:01:15.156004 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:15.155975    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e1113d78-e48d-48b2-aab4-c3f859f69eae-config-file\") pod \"limitador-limitador-7c9b87d574-9m92k\" (UID: \"e1113d78-e48d-48b2-aab4-c3f859f69eae\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-9m92k"
Oct 30 07:01:15.183087 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:15.183054    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkmhv\" (UniqueName: \"kubernetes.io/projected/e1113d78-e48d-48b2-aab4-c3f859f69eae-kube-api-access-dkmhv\") pod \"limitador-limitador-7c9b87d574-9m92k\" (UID: \"e1113d78-e48d-48b2-aab4-c3f859f69eae\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-9m92k"
Oct 30 07:01:15.307577 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:15.307539    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-9m92k"
Oct 30 07:01:15.613996 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:15.613967    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:01:15.614462 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:01:15.614436    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1113d78_e48d_48b2_aab4_c3f859f69eae.slice/crio-bf0f15e279d78a253b2fb1cb54a816c1cc4130c9cd7e4b47cd329e538c5b2cc2 WatchSource:0}: Error finding container bf0f15e279d78a253b2fb1cb54a816c1cc4130c9cd7e4b47cd329e538c5b2cc2: Status 404 returned error can't find the container with id bf0f15e279d78a253b2fb1cb54a816c1cc4130c9cd7e4b47cd329e538c5b2cc2
Oct 30 07:01:15.633773 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:15.633738    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-9m92k" event={"ID":"e1113d78-e48d-48b2-aab4-c3f859f69eae","Type":"ContainerStarted","Data":"bf0f15e279d78a253b2fb1cb54a816c1cc4130c9cd7e4b47cd329e538c5b2cc2"}
Oct 30 07:01:17.644488 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:17.644447    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-54599bd898-89xfk" event={"ID":"504b6d22-335a-4ba3-b724-da2c4880543d","Type":"ContainerStarted","Data":"47bab168f86f373059ef47019c3886bf382c5713bdf84d78953298e27770e293"}
Oct 30 07:01:17.693935 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:17.693884    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-54599bd898-89xfk" podStartSLOduration=2.276991679 podStartE2EDuration="4.693869081s" totalImagesPullingTime="2.416877402s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:01:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:01:17.692714339 +0000 UTC m=+236.302121320" watchObservedRunningTime="2026-10-30 07:01:17.693869081 +0000 UTC m=+236.303276048"
Oct 30 07:01:20.601556 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:20.601516    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-gwz2f"
Oct 30 07:01:20.661993 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:20.661960    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-9m92k" event={"ID":"e1113d78-e48d-48b2-aab4-c3f859f69eae","Type":"ContainerStarted","Data":"d9b6cb402bd698d3b8978362e3db6dcd317f824b4f089da4423a699705e176df"}
Oct 30 07:01:20.662169 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:20.662017    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-9m92k"
Oct 30 07:01:20.709401 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:20.709348    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-9m92k" podStartSLOduration=2.626529868 podStartE2EDuration="6.709332752s" totalImagesPullingTime="4.082802884s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:01:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:01:20.707974638 +0000 UTC m=+239.317381608" watchObservedRunningTime="2026-10-30 07:01:20.709332752 +0000 UTC m=+239.318739716"
Oct 30 07:01:31.667375 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:01:31.667346    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-9m92k"
Oct 30 07:01:31.855522 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:01:31.855484    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:02:01.856669 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:02:01.856633    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:02:18.758196 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:02:18.758111    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-l7ztv"]
Oct 30 07:02:18.761852 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:02:18.761827    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l7ztv"
Oct 30 07:02:18.781970 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:02:18.781943    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-l7ztv"]
Oct 30 07:02:18.794966 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:02:18.794948    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-7lt4s\""
Oct 30 07:02:18.797425 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:02:18.797406    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Oct 30 07:02:18.831040 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:02:18.831014    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 07:02:18.831161 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:02:18.831088    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 07:02:18.855605 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:02:18.855577    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4d3c43fe-39f4-49b3-98e9-19ac060b7eb9-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-l7ztv\" (UID: \"4d3c43fe-39f4-49b3-98e9-19ac060b7eb9\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l7ztv"
Oct 30 07:02:18.855724 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:02:18.855618    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp2qh\" (UniqueName: \"kubernetes.io/projected/4d3c43fe-39f4-49b3-98e9-19ac060b7eb9-kube-api-access-gp2qh\") pod \"llmisvc-controller-manager-869d5dcc5d-l7ztv\" (UID: \"4d3c43fe-39f4-49b3-98e9-19ac060b7eb9\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l7ztv"
Oct 30 07:02:18.956286 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:02:18.956255    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4d3c43fe-39f4-49b3-98e9-19ac060b7eb9-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-l7ztv\" (UID: \"4d3c43fe-39f4-49b3-98e9-19ac060b7eb9\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l7ztv"
Oct 30 07:02:18.956468 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:02:18.956308    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp2qh\" (UniqueName: \"kubernetes.io/projected/4d3c43fe-39f4-49b3-98e9-19ac060b7eb9-kube-api-access-gp2qh\") pod \"llmisvc-controller-manager-869d5dcc5d-l7ztv\" (UID: \"4d3c43fe-39f4-49b3-98e9-19ac060b7eb9\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l7ztv"
Oct 30 07:02:18.958707 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:02:18.958686    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4d3c43fe-39f4-49b3-98e9-19ac060b7eb9-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-l7ztv\" (UID: \"4d3c43fe-39f4-49b3-98e9-19ac060b7eb9\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l7ztv"
Oct 30 07:02:18.983711 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:02:18.983679    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp2qh\" (UniqueName: \"kubernetes.io/projected/4d3c43fe-39f4-49b3-98e9-19ac060b7eb9-kube-api-access-gp2qh\") pod \"llmisvc-controller-manager-869d5dcc5d-l7ztv\" (UID: \"4d3c43fe-39f4-49b3-98e9-19ac060b7eb9\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l7ztv"
Oct 30 07:02:19.073679 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:02:19.073651    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l7ztv"
Oct 30 07:02:19.367452 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:02:19.367422    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4d3c43fe_39f4_49b3_98e9_19ac060b7eb9.slice/crio-a674ab06eea29a5fb239ad066e120de5ff1d8dadd718f99def9ffba7a85c1c34 WatchSource:0}: Error finding container a674ab06eea29a5fb239ad066e120de5ff1d8dadd718f99def9ffba7a85c1c34: Status 404 returned error can't find the container with id a674ab06eea29a5fb239ad066e120de5ff1d8dadd718f99def9ffba7a85c1c34
Oct 30 07:02:19.367786 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:02:19.367760    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-l7ztv"]
Oct 30 07:02:19.930232 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:02:19.930196    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l7ztv" event={"ID":"4d3c43fe-39f4-49b3-98e9-19ac060b7eb9","Type":"ContainerStarted","Data":"a674ab06eea29a5fb239ad066e120de5ff1d8dadd718f99def9ffba7a85c1c34"}
Oct 30 07:02:21.943348 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:02:21.943307    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l7ztv" event={"ID":"4d3c43fe-39f4-49b3-98e9-19ac060b7eb9","Type":"ContainerStarted","Data":"490d013dc26c5fd410d8d98fee4c0a6168de4c3d5eaf7288a96bf8df04ecc924"}
Oct 30 07:02:21.943762 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:02:21.943566    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l7ztv"
Oct 30 07:02:22.000493 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:02:22.000443    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l7ztv" podStartSLOduration=2.045095758 podStartE2EDuration="4.000430519s" totalImagesPullingTime="1.955334761s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:02:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:02:21.989348853 +0000 UTC m=+300.598755842" watchObservedRunningTime="2026-10-30 07:02:22.000430519 +0000 UTC m=+300.609837489"
Oct 30 07:02:31.858337 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:02:31.858294    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:02:52.950111 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:02:52.950077    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l7ztv"
Oct 30 07:03:01.521433 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:03:01.521393    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:03:01.855435 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:03:01.855394    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:03:10.382753 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:03:10.382719    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5bddd4c9fd-sjdgf"]
Oct 30 07:03:10.386099 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:03:10.386080    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5bddd4c9fd-sjdgf"
Oct 30 07:03:10.424548 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:03:10.424523    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 30 07:03:10.424756 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:03:10.424735    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-lnjcd\""
Oct 30 07:03:10.424903 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:03:10.424768    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 07:03:10.424903 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:03:10.424790    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 07:03:10.424903 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:03:10.424878    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 07:03:10.425995 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:03:10.425972    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5bddd4c9fd-sjdgf"]
Oct 30 07:03:10.482713 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:03:10.482688    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/567cf906-1440-49f5-876b-692dab3f4fed-maas-api-tls\") pod \"maas-api-5bddd4c9fd-sjdgf\" (UID: \"567cf906-1440-49f5-876b-692dab3f4fed\") " pod="odh-ai-gateway-infra/maas-api-5bddd4c9fd-sjdgf"
Oct 30 07:03:10.482843 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:03:10.482717    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/567cf906-1440-49f5-876b-692dab3f4fed-metrics-certs\") pod \"maas-api-5bddd4c9fd-sjdgf\" (UID: \"567cf906-1440-49f5-876b-692dab3f4fed\") " pod="odh-ai-gateway-infra/maas-api-5bddd4c9fd-sjdgf"
Oct 30 07:03:10.482843 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:03:10.482755    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhz84\" (UniqueName: \"kubernetes.io/projected/567cf906-1440-49f5-876b-692dab3f4fed-kube-api-access-rhz84\") pod \"maas-api-5bddd4c9fd-sjdgf\" (UID: \"567cf906-1440-49f5-876b-692dab3f4fed\") " pod="odh-ai-gateway-infra/maas-api-5bddd4c9fd-sjdgf"
Oct 30 07:03:10.482921 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:03:10.482890    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/567cf906-1440-49f5-876b-692dab3f4fed-openshift-service-ca\") pod \"maas-api-5bddd4c9fd-sjdgf\" (UID: \"567cf906-1440-49f5-876b-692dab3f4fed\") " pod="odh-ai-gateway-infra/maas-api-5bddd4c9fd-sjdgf"
Oct 30 07:03:10.584216 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:03:10.584187    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/567cf906-1440-49f5-876b-692dab3f4fed-maas-api-tls\") pod \"maas-api-5bddd4c9fd-sjdgf\" (UID: \"567cf906-1440-49f5-876b-692dab3f4fed\") " pod="odh-ai-gateway-infra/maas-api-5bddd4c9fd-sjdgf"
Oct 30 07:03:10.584357 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:03:10.584221    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/567cf906-1440-49f5-876b-692dab3f4fed-metrics-certs\") pod \"maas-api-5bddd4c9fd-sjdgf\" (UID: \"567cf906-1440-49f5-876b-692dab3f4fed\") " pod="odh-ai-gateway-infra/maas-api-5bddd4c9fd-sjdgf"
Oct 30 07:03:10.584357 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:03:10.584264    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhz84\" (UniqueName: \"kubernetes.io/projected/567cf906-1440-49f5-876b-692dab3f4fed-kube-api-access-rhz84\") pod \"maas-api-5bddd4c9fd-sjdgf\" (UID: \"567cf906-1440-49f5-876b-692dab3f4fed\") " pod="odh-ai-gateway-infra/maas-api-5bddd4c9fd-sjdgf"
Oct 30 07:03:10.584357 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:03:10.584308    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/567cf906-1440-49f5-876b-692dab3f4fed-openshift-service-ca\") pod \"maas-api-5bddd4c9fd-sjdgf\" (UID: \"567cf906-1440-49f5-876b-692dab3f4fed\") " pod="odh-ai-gateway-infra/maas-api-5bddd4c9fd-sjdgf"
Oct 30 07:03:10.585026 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:03:10.585008    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/567cf906-1440-49f5-876b-692dab3f4fed-openshift-service-ca\") pod \"maas-api-5bddd4c9fd-sjdgf\" (UID: \"567cf906-1440-49f5-876b-692dab3f4fed\") " pod="odh-ai-gateway-infra/maas-api-5bddd4c9fd-sjdgf"
Oct 30 07:03:10.586695 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:03:10.586678    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/567cf906-1440-49f5-876b-692dab3f4fed-maas-api-tls\") pod \"maas-api-5bddd4c9fd-sjdgf\" (UID: \"567cf906-1440-49f5-876b-692dab3f4fed\") " pod="odh-ai-gateway-infra/maas-api-5bddd4c9fd-sjdgf"
Oct 30 07:03:10.586749 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:03:10.586733    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/567cf906-1440-49f5-876b-692dab3f4fed-metrics-certs\") pod \"maas-api-5bddd4c9fd-sjdgf\" (UID: \"567cf906-1440-49f5-876b-692dab3f4fed\") " pod="odh-ai-gateway-infra/maas-api-5bddd4c9fd-sjdgf"
Oct 30 07:03:10.612784 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:03:10.612752    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhz84\" (UniqueName: \"kubernetes.io/projected/567cf906-1440-49f5-876b-692dab3f4fed-kube-api-access-rhz84\") pod \"maas-api-5bddd4c9fd-sjdgf\" (UID: \"567cf906-1440-49f5-876b-692dab3f4fed\") " pod="odh-ai-gateway-infra/maas-api-5bddd4c9fd-sjdgf"
Oct 30 07:03:10.698788 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:03:10.698707    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5bddd4c9fd-sjdgf"
Oct 30 07:03:10.993686 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:03:10.993657    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5bddd4c9fd-sjdgf"]
Oct 30 07:03:10.994427 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:03:10.994400    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod567cf906_1440_49f5_876b_692dab3f4fed.slice/crio-51a562f07e68b90fae6452d80d594bf4c3eac241997874e95f1a8470a6655918 WatchSource:0}: Error finding container 51a562f07e68b90fae6452d80d594bf4c3eac241997874e95f1a8470a6655918: Status 404 returned error can't find the container with id 51a562f07e68b90fae6452d80d594bf4c3eac241997874e95f1a8470a6655918
Oct 30 07:03:11.182133 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:03:11.182098    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5bddd4c9fd-sjdgf" event={"ID":"567cf906-1440-49f5-876b-692dab3f4fed","Type":"ContainerStarted","Data":"51a562f07e68b90fae6452d80d594bf4c3eac241997874e95f1a8470a6655918"}
Oct 30 07:03:14.198501 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:03:14.198466    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5bddd4c9fd-sjdgf" event={"ID":"567cf906-1440-49f5-876b-692dab3f4fed","Type":"ContainerStarted","Data":"cd7b28eb7d439f354bc9fb7cebeb5b9027e632521e79c20f6d97721e86f0a96f"}
Oct 30 07:03:14.199107 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:03:14.198586    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5bddd4c9fd-sjdgf"
Oct 30 07:03:14.244921 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:03:14.244865    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5bddd4c9fd-sjdgf" podStartSLOduration=1.883055508 podStartE2EDuration="4.244851179s" totalImagesPullingTime="2.361795671s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:03:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:03:14.244044628 +0000 UTC m=+352.853451598" watchObservedRunningTime="2026-10-30 07:03:14.244851179 +0000 UTC m=+352.854258150"
Oct 30 07:03:20.209152 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:03:20.209125    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5bddd4c9fd-sjdgf"
Oct 30 07:03:31.855667 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:03:31.855629    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:03:33.726402 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:03:33.726328    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-44xkr"]
Oct 30 07:03:33.730082 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:03:33.730066    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-44xkr"
Oct 30 07:03:33.750509 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:03:33.750484    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-44xkr"]
Oct 30 07:03:33.872465 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:03:33.872436    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0f8fa62e-5595-4c17-b186-f604f06b17ed-cert\") pod \"llmisvc-controller-manager-6466c554d8-44xkr\" (UID: \"0f8fa62e-5595-4c17-b186-f604f06b17ed\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-44xkr"
Oct 30 07:03:33.872598 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:03:33.872482    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv9g4\" (UniqueName: \"kubernetes.io/projected/0f8fa62e-5595-4c17-b186-f604f06b17ed-kube-api-access-pv9g4\") pod \"llmisvc-controller-manager-6466c554d8-44xkr\" (UID: \"0f8fa62e-5595-4c17-b186-f604f06b17ed\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-44xkr"
Oct 30 07:03:33.973608 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:03:33.973579    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0f8fa62e-5595-4c17-b186-f604f06b17ed-cert\") pod \"llmisvc-controller-manager-6466c554d8-44xkr\" (UID: \"0f8fa62e-5595-4c17-b186-f604f06b17ed\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-44xkr"
Oct 30 07:03:33.973715 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:03:33.973629    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv9g4\" (UniqueName: \"kubernetes.io/projected/0f8fa62e-5595-4c17-b186-f604f06b17ed-kube-api-access-pv9g4\") pod \"llmisvc-controller-manager-6466c554d8-44xkr\" (UID: \"0f8fa62e-5595-4c17-b186-f604f06b17ed\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-44xkr"
Oct 30 07:03:33.976106 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:03:33.976082    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0f8fa62e-5595-4c17-b186-f604f06b17ed-cert\") pod \"llmisvc-controller-manager-6466c554d8-44xkr\" (UID: \"0f8fa62e-5595-4c17-b186-f604f06b17ed\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-44xkr"
Oct 30 07:03:33.998822 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:03:33.998749    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv9g4\" (UniqueName: \"kubernetes.io/projected/0f8fa62e-5595-4c17-b186-f604f06b17ed-kube-api-access-pv9g4\") pod \"llmisvc-controller-manager-6466c554d8-44xkr\" (UID: \"0f8fa62e-5595-4c17-b186-f604f06b17ed\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-44xkr"
Oct 30 07:03:34.042658 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:03:34.042632    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-44xkr"
Oct 30 07:03:34.335481 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:03:34.335455    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f8fa62e_5595_4c17_b186_f604f06b17ed.slice/crio-a1de8cce79b49db6a691aae5dd47c97cfc14b711c2db687b46a6d8deda7478a9 WatchSource:0}: Error finding container a1de8cce79b49db6a691aae5dd47c97cfc14b711c2db687b46a6d8deda7478a9: Status 404 returned error can't find the container with id a1de8cce79b49db6a691aae5dd47c97cfc14b711c2db687b46a6d8deda7478a9
Oct 30 07:03:34.335587 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:03:34.335545    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-44xkr"]
Oct 30 07:03:35.297614 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:03:35.297578    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-44xkr" event={"ID":"0f8fa62e-5595-4c17-b186-f604f06b17ed","Type":"ContainerStarted","Data":"8f074e648e766bea115df9b8a5837159e3c796dc14cf04ae57e4c859f0f9c7b8"}
Oct 30 07:03:35.298038 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:03:35.297652    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-44xkr" event={"ID":"0f8fa62e-5595-4c17-b186-f604f06b17ed","Type":"ContainerStarted","Data":"a1de8cce79b49db6a691aae5dd47c97cfc14b711c2db687b46a6d8deda7478a9"}
Oct 30 07:03:35.298038 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:03:35.297692    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-44xkr"
Oct 30 07:03:35.342677 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:03:35.342618    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-44xkr" podStartSLOduration=2.024196833 podStartE2EDuration="2.342602552s" totalImagesPullingTime="318.405719ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:03:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:03:35.341788705 +0000 UTC m=+373.951195675" watchObservedRunningTime="2026-10-30 07:03:35.342602552 +0000 UTC m=+373.952009523"
Oct 30 07:04:01.857224 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:04:01.857178    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:04:06.302796 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:04:06.302764    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-44xkr"
Oct 30 07:04:06.377480 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:04:06.377440    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-l7ztv"]
Oct 30 07:04:06.377664 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:04:06.377615    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l7ztv" podUID="4d3c43fe-39f4-49b3-98e9-19ac060b7eb9" containerName="manager" containerID="cri-o://490d013dc26c5fd410d8d98fee4c0a6168de4c3d5eaf7288a96bf8df04ecc924" gracePeriod=30
Oct 30 07:04:06.609831 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:04:06.609791    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l7ztv"
Oct 30 07:04:06.648644 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:04:06.648618    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d3c43fe-39f4-49b3-98e9-19ac060b7eb9-cert\" (UniqueName: \"kubernetes.io/secret/4d3c43fe-39f4-49b3-98e9-19ac060b7eb9-cert\") pod \"4d3c43fe-39f4-49b3-98e9-19ac060b7eb9\" (UID: \"4d3c43fe-39f4-49b3-98e9-19ac060b7eb9\") "
Oct 30 07:04:06.648765 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:04:06.648671    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d3c43fe-39f4-49b3-98e9-19ac060b7eb9-kube-api-access-gp2qh\" (UniqueName: \"kubernetes.io/projected/4d3c43fe-39f4-49b3-98e9-19ac060b7eb9-kube-api-access-gp2qh\") pod \"4d3c43fe-39f4-49b3-98e9-19ac060b7eb9\" (UID: \"4d3c43fe-39f4-49b3-98e9-19ac060b7eb9\") "
Oct 30 07:04:06.650788 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:04:06.650764    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d3c43fe-39f4-49b3-98e9-19ac060b7eb9-kube-api-access-gp2qh" pod "4d3c43fe-39f4-49b3-98e9-19ac060b7eb9" (UID: "4d3c43fe-39f4-49b3-98e9-19ac060b7eb9"). InnerVolumeSpecName "kube-api-access-gp2qh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:04:06.650869 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:04:06.650828    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d3c43fe-39f4-49b3-98e9-19ac060b7eb9-cert" pod "4d3c43fe-39f4-49b3-98e9-19ac060b7eb9" (UID: "4d3c43fe-39f4-49b3-98e9-19ac060b7eb9"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:04:06.749286 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:04:06.749261    6224 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4d3c43fe-39f4-49b3-98e9-19ac060b7eb9-cert\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:04:06.749286 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:04:06.749284    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gp2qh\" (UniqueName: \"kubernetes.io/projected/4d3c43fe-39f4-49b3-98e9-19ac060b7eb9-kube-api-access-gp2qh\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:04:07.448068 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:04:07.448032    6224 generic.go:356] "Generic (PLEG): container finished" podID="4d3c43fe-39f4-49b3-98e9-19ac060b7eb9" containerID="490d013dc26c5fd410d8d98fee4c0a6168de4c3d5eaf7288a96bf8df04ecc924" exitCode=0
Oct 30 07:04:07.448452 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:04:07.448084    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l7ztv"
Oct 30 07:04:07.448452 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:04:07.448124    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l7ztv" event={"ID":"4d3c43fe-39f4-49b3-98e9-19ac060b7eb9","Type":"ContainerDied","Data":"490d013dc26c5fd410d8d98fee4c0a6168de4c3d5eaf7288a96bf8df04ecc924"}
Oct 30 07:04:07.448452 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:04:07.448183    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l7ztv" event={"ID":"4d3c43fe-39f4-49b3-98e9-19ac060b7eb9","Type":"ContainerDied","Data":"a674ab06eea29a5fb239ad066e120de5ff1d8dadd718f99def9ffba7a85c1c34"}
Oct 30 07:04:07.448452 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:04:07.448213    6224 scope.go:122] "RemoveContainer" containerID="490d013dc26c5fd410d8d98fee4c0a6168de4c3d5eaf7288a96bf8df04ecc924"
Oct 30 07:04:07.457092 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:04:07.457074    6224 scope.go:122] "RemoveContainer" containerID="490d013dc26c5fd410d8d98fee4c0a6168de4c3d5eaf7288a96bf8df04ecc924"
Oct 30 07:04:07.457348 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:04:07.457330    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"490d013dc26c5fd410d8d98fee4c0a6168de4c3d5eaf7288a96bf8df04ecc924\": container with ID starting with 490d013dc26c5fd410d8d98fee4c0a6168de4c3d5eaf7288a96bf8df04ecc924 not found: ID does not exist" containerID="490d013dc26c5fd410d8d98fee4c0a6168de4c3d5eaf7288a96bf8df04ecc924"
Oct 30 07:04:07.457487 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:04:07.457467    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"490d013dc26c5fd410d8d98fee4c0a6168de4c3d5eaf7288a96bf8df04ecc924"} err="failed to get container status \"490d013dc26c5fd410d8d98fee4c0a6168de4c3d5eaf7288a96bf8df04ecc924\": rpc error: code = NotFound desc = could not find container \"490d013dc26c5fd410d8d98fee4c0a6168de4c3d5eaf7288a96bf8df04ecc924\": container with ID starting with 490d013dc26c5fd410d8d98fee4c0a6168de4c3d5eaf7288a96bf8df04ecc924 not found: ID does not exist"
Oct 30 07:04:07.506272 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:04:07.506245    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-l7ztv"]
Oct 30 07:04:07.512665 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:04:07.512630    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-l7ztv"]
Oct 30 07:04:09.500836 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:04:09.500787    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d3c43fe-39f4-49b3-98e9-19ac060b7eb9" path="/var/lib/kubelet/pods/4d3c43fe-39f4-49b3-98e9-19ac060b7eb9/volumes"
Oct 30 07:04:31.855745 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:04:31.855708    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:04:45.085090 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:04:45.085053    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:04:49.400997 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:04:49.400940    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:04:52.582743 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:04:52.582704    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:04:56.349941 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:04:56.349905    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:05:01.855559 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:05:01.855515    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:05:04.466832 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:04.466717    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:05:22.380716 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:22.380679    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qbltv"]
Oct 30 07:05:22.381232 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:22.381199    6224 cpu_manager.go:414] "containerMap: removing container" podUID="4d3c43fe-39f4-49b3-98e9-19ac060b7eb9" containerName="manager"
Oct 30 07:05:22.381232 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:22.381218    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d3c43fe-39f4-49b3-98e9-19ac060b7eb9" containerName="manager"
Oct 30 07:05:22.381353 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:22.381335    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d3c43fe-39f4-49b3-98e9-19ac060b7eb9" containerName="manager"
Oct 30 07:05:22.384431 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:22.384411    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qbltv"
Oct 30 07:05:22.401031 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:22.401008    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 07:05:22.416608 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:22.416588    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qbltv"]
Oct 30 07:05:22.484375 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:22.484351    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7177df4a-a897-4dc3-b71b-c70431bb12fd-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-qbltv\" (UID: \"7177df4a-a897-4dc3-b71b-c70431bb12fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qbltv"
Oct 30 07:05:22.484492 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:22.484390    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t98c\" (UniqueName: \"kubernetes.io/projected/7177df4a-a897-4dc3-b71b-c70431bb12fd-kube-api-access-6t98c\") pod \"maas-default-gateway-openshift-default-6dcf757899-qbltv\" (UID: \"7177df4a-a897-4dc3-b71b-c70431bb12fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qbltv"
Oct 30 07:05:22.484492 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:22.484411    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7177df4a-a897-4dc3-b71b-c70431bb12fd-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-qbltv\" (UID: \"7177df4a-a897-4dc3-b71b-c70431bb12fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qbltv"
Oct 30 07:05:22.484492 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:22.484467    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7177df4a-a897-4dc3-b71b-c70431bb12fd-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-qbltv\" (UID: \"7177df4a-a897-4dc3-b71b-c70431bb12fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qbltv"
Oct 30 07:05:22.484597 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:22.484512    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7177df4a-a897-4dc3-b71b-c70431bb12fd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-qbltv\" (UID: \"7177df4a-a897-4dc3-b71b-c70431bb12fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qbltv"
Oct 30 07:05:22.484597 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:22.484549    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7177df4a-a897-4dc3-b71b-c70431bb12fd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-qbltv\" (UID: \"7177df4a-a897-4dc3-b71b-c70431bb12fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qbltv"
Oct 30 07:05:22.484657 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:22.484600    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7177df4a-a897-4dc3-b71b-c70431bb12fd-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-qbltv\" (UID: \"7177df4a-a897-4dc3-b71b-c70431bb12fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qbltv"
Oct 30 07:05:22.484691 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:22.484657    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7177df4a-a897-4dc3-b71b-c70431bb12fd-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-qbltv\" (UID: \"7177df4a-a897-4dc3-b71b-c70431bb12fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qbltv"
Oct 30 07:05:22.484726 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:22.484707    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7177df4a-a897-4dc3-b71b-c70431bb12fd-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-qbltv\" (UID: \"7177df4a-a897-4dc3-b71b-c70431bb12fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qbltv"
Oct 30 07:05:22.492059 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:22.492042    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-sp9gr\""
Oct 30 07:05:22.585839 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:22.585797    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7177df4a-a897-4dc3-b71b-c70431bb12fd-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-qbltv\" (UID: \"7177df4a-a897-4dc3-b71b-c70431bb12fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qbltv"
Oct 30 07:05:22.585974 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:22.585855    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7177df4a-a897-4dc3-b71b-c70431bb12fd-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-qbltv\" (UID: \"7177df4a-a897-4dc3-b71b-c70431bb12fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qbltv"
Oct 30 07:05:22.585974 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:22.585879    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7177df4a-a897-4dc3-b71b-c70431bb12fd-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-qbltv\" (UID: \"7177df4a-a897-4dc3-b71b-c70431bb12fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qbltv"
Oct 30 07:05:22.585974 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:22.585916    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7177df4a-a897-4dc3-b71b-c70431bb12fd-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-qbltv\" (UID: \"7177df4a-a897-4dc3-b71b-c70431bb12fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qbltv"
Oct 30 07:05:22.585974 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:22.585941    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t98c\" (UniqueName: \"kubernetes.io/projected/7177df4a-a897-4dc3-b71b-c70431bb12fd-kube-api-access-6t98c\") pod \"maas-default-gateway-openshift-default-6dcf757899-qbltv\" (UID: \"7177df4a-a897-4dc3-b71b-c70431bb12fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qbltv"
Oct 30 07:05:22.586200 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:22.585972    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7177df4a-a897-4dc3-b71b-c70431bb12fd-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-qbltv\" (UID: \"7177df4a-a897-4dc3-b71b-c70431bb12fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qbltv"
Oct 30 07:05:22.586200 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:22.586022    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7177df4a-a897-4dc3-b71b-c70431bb12fd-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-qbltv\" (UID: \"7177df4a-a897-4dc3-b71b-c70431bb12fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qbltv"
Oct 30 07:05:22.586200 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:22.586130    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7177df4a-a897-4dc3-b71b-c70431bb12fd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-qbltv\" (UID: \"7177df4a-a897-4dc3-b71b-c70431bb12fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qbltv"
Oct 30 07:05:22.586200 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:22.586166    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7177df4a-a897-4dc3-b71b-c70431bb12fd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-qbltv\" (UID: \"7177df4a-a897-4dc3-b71b-c70431bb12fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qbltv"
Oct 30 07:05:22.586412 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:22.586276    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7177df4a-a897-4dc3-b71b-c70431bb12fd-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-qbltv\" (UID: \"7177df4a-a897-4dc3-b71b-c70431bb12fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qbltv"
Oct 30 07:05:22.586412 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:22.586309    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7177df4a-a897-4dc3-b71b-c70431bb12fd-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-qbltv\" (UID: \"7177df4a-a897-4dc3-b71b-c70431bb12fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qbltv"
Oct 30 07:05:22.586693 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:22.586671    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7177df4a-a897-4dc3-b71b-c70431bb12fd-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-qbltv\" (UID: \"7177df4a-a897-4dc3-b71b-c70431bb12fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qbltv"
Oct 30 07:05:22.586777 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:22.586701    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7177df4a-a897-4dc3-b71b-c70431bb12fd-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-qbltv\" (UID: \"7177df4a-a897-4dc3-b71b-c70431bb12fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qbltv"
Oct 30 07:05:22.587074 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:22.587052    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7177df4a-a897-4dc3-b71b-c70431bb12fd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-qbltv\" (UID: \"7177df4a-a897-4dc3-b71b-c70431bb12fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qbltv"
Oct 30 07:05:22.588339 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:22.588318    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7177df4a-a897-4dc3-b71b-c70431bb12fd-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-qbltv\" (UID: \"7177df4a-a897-4dc3-b71b-c70431bb12fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qbltv"
Oct 30 07:05:22.588915 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:22.588898    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7177df4a-a897-4dc3-b71b-c70431bb12fd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-qbltv\" (UID: \"7177df4a-a897-4dc3-b71b-c70431bb12fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qbltv"
Oct 30 07:05:22.610651 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:22.610625    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7177df4a-a897-4dc3-b71b-c70431bb12fd-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-qbltv\" (UID: \"7177df4a-a897-4dc3-b71b-c70431bb12fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qbltv"
Oct 30 07:05:22.610737 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:22.610658    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t98c\" (UniqueName: \"kubernetes.io/projected/7177df4a-a897-4dc3-b71b-c70431bb12fd-kube-api-access-6t98c\") pod \"maas-default-gateway-openshift-default-6dcf757899-qbltv\" (UID: \"7177df4a-a897-4dc3-b71b-c70431bb12fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qbltv"
Oct 30 07:05:22.698472 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:22.698412    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qbltv"
Oct 30 07:05:23.011117 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:23.011085    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qbltv"]
Oct 30 07:05:23.011469 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:05:23.011445    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7177df4a_a897_4dc3_b71b_c70431bb12fd.slice/crio-351d714d6a34fced82e646937a8f3a6c43a43c1fe5aef5e283928168e90fb8c5 WatchSource:0}: Error finding container 351d714d6a34fced82e646937a8f3a6c43a43c1fe5aef5e283928168e90fb8c5: Status 404 returned error can't find the container with id 351d714d6a34fced82e646937a8f3a6c43a43c1fe5aef5e283928168e90fb8c5
Oct 30 07:05:23.013316 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:23.013300    6224 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 07:05:23.805376 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:23.805339    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qbltv" event={"ID":"7177df4a-a897-4dc3-b71b-c70431bb12fd","Type":"ContainerStarted","Data":"351d714d6a34fced82e646937a8f3a6c43a43c1fe5aef5e283928168e90fb8c5"}
Oct 30 07:05:26.052085 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:26.052054    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:05:26.052382 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:26.052121    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:05:26.052382 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:26.052151    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:05:26.823616 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:26.823577    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qbltv" event={"ID":"7177df4a-a897-4dc3-b71b-c70431bb12fd","Type":"ContainerStarted","Data":"5e1f4df81215c3b4bad30111114830b75524755b468a49e32f71ff036375e93e"}
Oct 30 07:05:26.889847 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:26.889773    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qbltv" podStartSLOduration=1.851994444 podStartE2EDuration="4.889757082s" totalImagesPullingTime="3.037762638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:05:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:05:26.888370181 +0000 UTC m=+485.497777163" watchObservedRunningTime="2026-10-30 07:05:26.889757082 +0000 UTC m=+485.499164052"
Oct 30 07:05:27.699141 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:27.699103    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qbltv"
Oct 30 07:05:27.700670 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:27.700644    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-qbltv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Oct 30 07:05:27.700796 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:27.700697    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qbltv" podUID="7177df4a-a897-4dc3-b71b-c70431bb12fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Oct 30 07:05:28.699038 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:28.699005    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-qbltv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Oct 30 07:05:28.699228 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:28.699056    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qbltv" podUID="7177df4a-a897-4dc3-b71b-c70431bb12fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Oct 30 07:05:29.699189 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:29.699152    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-qbltv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Oct 30 07:05:29.699383 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:29.699216    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qbltv" podUID="7177df4a-a897-4dc3-b71b-c70431bb12fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Oct 30 07:05:30.699342 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:30.699301    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-qbltv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Oct 30 07:05:30.699527 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:30.699356    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qbltv" podUID="7177df4a-a897-4dc3-b71b-c70431bb12fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Oct 30 07:05:31.699387 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:31.699349    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-qbltv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Oct 30 07:05:31.699571 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:31.699403    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qbltv" podUID="7177df4a-a897-4dc3-b71b-c70431bb12fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Oct 30 07:05:31.856331 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:05:31.856296    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:05:32.699140 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:32.699100    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-qbltv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Oct 30 07:05:32.699353 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:32.699151    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qbltv" podUID="7177df4a-a897-4dc3-b71b-c70431bb12fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Oct 30 07:05:34.180401 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:34.180356    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qbltv"
Oct 30 07:05:34.180877 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:34.180684    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qbltv"
Oct 30 07:05:34.181603 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:34.181577    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qbltv"
Oct 30 07:05:37.379123 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:37.379087    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bgnvh"]
Oct 30 07:05:37.382786 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:37.382762    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bgnvh"
Oct 30 07:05:37.405894 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:37.405482    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bgnvh"]
Oct 30 07:05:37.539038 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:37.539003    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14ba5fea-7709-42cb-8fe3-b63093fdcda7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-bgnvh\" (UID: \"14ba5fea-7709-42cb-8fe3-b63093fdcda7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bgnvh"
Oct 30 07:05:37.539211 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:37.539067    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8l6t\" (UniqueName: \"kubernetes.io/projected/14ba5fea-7709-42cb-8fe3-b63093fdcda7-kube-api-access-p8l6t\") pod \"maas-default-gateway-openshift-default-6dcf757899-bgnvh\" (UID: \"14ba5fea-7709-42cb-8fe3-b63093fdcda7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bgnvh"
Oct 30 07:05:37.539211 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:37.539151    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14ba5fea-7709-42cb-8fe3-b63093fdcda7-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-bgnvh\" (UID: \"14ba5fea-7709-42cb-8fe3-b63093fdcda7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bgnvh"
Oct 30 07:05:37.539211 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:37.539177    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14ba5fea-7709-42cb-8fe3-b63093fdcda7-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-bgnvh\" (UID: \"14ba5fea-7709-42cb-8fe3-b63093fdcda7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bgnvh"
Oct 30 07:05:37.539211 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:37.539208    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14ba5fea-7709-42cb-8fe3-b63093fdcda7-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-bgnvh\" (UID: \"14ba5fea-7709-42cb-8fe3-b63093fdcda7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bgnvh"
Oct 30 07:05:37.539443 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:37.539264    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14ba5fea-7709-42cb-8fe3-b63093fdcda7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-bgnvh\" (UID: \"14ba5fea-7709-42cb-8fe3-b63093fdcda7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bgnvh"
Oct 30 07:05:37.539443 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:37.539333    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14ba5fea-7709-42cb-8fe3-b63093fdcda7-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-bgnvh\" (UID: \"14ba5fea-7709-42cb-8fe3-b63093fdcda7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bgnvh"
Oct 30 07:05:37.539443 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:37.539369    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14ba5fea-7709-42cb-8fe3-b63093fdcda7-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-bgnvh\" (UID: \"14ba5fea-7709-42cb-8fe3-b63093fdcda7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bgnvh"
Oct 30 07:05:37.539552 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:37.539486    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14ba5fea-7709-42cb-8fe3-b63093fdcda7-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-bgnvh\" (UID: \"14ba5fea-7709-42cb-8fe3-b63093fdcda7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bgnvh"
Oct 30 07:05:37.640457 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:37.640386    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14ba5fea-7709-42cb-8fe3-b63093fdcda7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-bgnvh\" (UID: \"14ba5fea-7709-42cb-8fe3-b63093fdcda7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bgnvh"
Oct 30 07:05:37.640457 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:37.640439    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8l6t\" (UniqueName: \"kubernetes.io/projected/14ba5fea-7709-42cb-8fe3-b63093fdcda7-kube-api-access-p8l6t\") pod \"maas-default-gateway-openshift-default-6dcf757899-bgnvh\" (UID: \"14ba5fea-7709-42cb-8fe3-b63093fdcda7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bgnvh"
Oct 30 07:05:37.640666 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:37.640474    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14ba5fea-7709-42cb-8fe3-b63093fdcda7-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-bgnvh\" (UID: \"14ba5fea-7709-42cb-8fe3-b63093fdcda7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bgnvh"
Oct 30 07:05:37.640666 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:37.640491    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14ba5fea-7709-42cb-8fe3-b63093fdcda7-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-bgnvh\" (UID: \"14ba5fea-7709-42cb-8fe3-b63093fdcda7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bgnvh"
Oct 30 07:05:37.640666 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:37.640518    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14ba5fea-7709-42cb-8fe3-b63093fdcda7-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-bgnvh\" (UID: \"14ba5fea-7709-42cb-8fe3-b63093fdcda7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bgnvh"
Oct 30 07:05:37.640666 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:37.640547    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14ba5fea-7709-42cb-8fe3-b63093fdcda7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-bgnvh\" (UID: \"14ba5fea-7709-42cb-8fe3-b63093fdcda7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bgnvh"
Oct 30 07:05:37.640666 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:37.640599    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14ba5fea-7709-42cb-8fe3-b63093fdcda7-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-bgnvh\" (UID: \"14ba5fea-7709-42cb-8fe3-b63093fdcda7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bgnvh"
Oct 30 07:05:37.640666 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:37.640640    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14ba5fea-7709-42cb-8fe3-b63093fdcda7-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-bgnvh\" (UID: \"14ba5fea-7709-42cb-8fe3-b63093fdcda7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bgnvh"
Oct 30 07:05:37.640988 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:37.640682    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14ba5fea-7709-42cb-8fe3-b63093fdcda7-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-bgnvh\" (UID: \"14ba5fea-7709-42cb-8fe3-b63093fdcda7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bgnvh"
Oct 30 07:05:37.640988 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:37.640979    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14ba5fea-7709-42cb-8fe3-b63093fdcda7-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-bgnvh\" (UID: \"14ba5fea-7709-42cb-8fe3-b63093fdcda7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bgnvh"
Oct 30 07:05:37.641120 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:37.641096    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14ba5fea-7709-42cb-8fe3-b63093fdcda7-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-bgnvh\" (UID: \"14ba5fea-7709-42cb-8fe3-b63093fdcda7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bgnvh"
Oct 30 07:05:37.641178 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:37.641126    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14ba5fea-7709-42cb-8fe3-b63093fdcda7-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-bgnvh\" (UID: \"14ba5fea-7709-42cb-8fe3-b63093fdcda7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bgnvh"
Oct 30 07:05:37.641231 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:37.641196    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14ba5fea-7709-42cb-8fe3-b63093fdcda7-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-bgnvh\" (UID: \"14ba5fea-7709-42cb-8fe3-b63093fdcda7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bgnvh"
Oct 30 07:05:37.641402 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:37.641377    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14ba5fea-7709-42cb-8fe3-b63093fdcda7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-bgnvh\" (UID: \"14ba5fea-7709-42cb-8fe3-b63093fdcda7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bgnvh"
Oct 30 07:05:37.642902 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:37.642877    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14ba5fea-7709-42cb-8fe3-b63093fdcda7-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-bgnvh\" (UID: \"14ba5fea-7709-42cb-8fe3-b63093fdcda7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bgnvh"
Oct 30 07:05:37.643006 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:37.642928    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14ba5fea-7709-42cb-8fe3-b63093fdcda7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-bgnvh\" (UID: \"14ba5fea-7709-42cb-8fe3-b63093fdcda7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bgnvh"
Oct 30 07:05:37.665943 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:37.665919    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14ba5fea-7709-42cb-8fe3-b63093fdcda7-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-bgnvh\" (UID: \"14ba5fea-7709-42cb-8fe3-b63093fdcda7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bgnvh"
Oct 30 07:05:37.666230 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:37.666213    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8l6t\" (UniqueName: \"kubernetes.io/projected/14ba5fea-7709-42cb-8fe3-b63093fdcda7-kube-api-access-p8l6t\") pod \"maas-default-gateway-openshift-default-6dcf757899-bgnvh\" (UID: \"14ba5fea-7709-42cb-8fe3-b63093fdcda7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bgnvh"
Oct 30 07:05:37.696940 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:37.696917    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bgnvh"
Oct 30 07:05:38.005239 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:38.005210    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bgnvh"]
Oct 30 07:05:38.006129 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:05:38.006105    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14ba5fea_7709_42cb_8fe3_b63093fdcda7.slice/crio-17f9db5ae43882b3d346317d79989841e9b4f07b9e1acbba7997553f6f5aaea2 WatchSource:0}: Error finding container 17f9db5ae43882b3d346317d79989841e9b4f07b9e1acbba7997553f6f5aaea2: Status 404 returned error can't find the container with id 17f9db5ae43882b3d346317d79989841e9b4f07b9e1acbba7997553f6f5aaea2
Oct 30 07:05:38.009142 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:38.009104    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:05:38.009260 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:38.009168    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:05:38.009260 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:38.009194    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:05:38.896556 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:38.896517    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bgnvh" event={"ID":"14ba5fea-7709-42cb-8fe3-b63093fdcda7","Type":"ContainerStarted","Data":"5e8f88e9e3c524f1b361251c3f0b6e8ac09d68ed2d5bf237a767580b1076d8fe"}
Oct 30 07:05:38.897000 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:38.896585    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bgnvh" event={"ID":"14ba5fea-7709-42cb-8fe3-b63093fdcda7","Type":"ContainerStarted","Data":"17f9db5ae43882b3d346317d79989841e9b4f07b9e1acbba7997553f6f5aaea2"}
Oct 30 07:05:38.959886 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:38.959826    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bgnvh" podStartSLOduration=1.959792757 podStartE2EDuration="1.959792757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:05:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:05:38.959098738 +0000 UTC m=+497.568505708" watchObservedRunningTime="2026-10-30 07:05:38.959792757 +0000 UTC m=+497.569199728"
Oct 30 07:05:39.697825 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:39.697782    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bgnvh"
Oct 30 07:05:39.699370 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:39.699343    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-bgnvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Oct 30 07:05:39.699507 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:39.699393    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bgnvh" podUID="14ba5fea-7709-42cb-8fe3-b63093fdcda7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Oct 30 07:05:40.697706 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:40.697669    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-bgnvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Oct 30 07:05:40.698117 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:40.697738    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bgnvh" podUID="14ba5fea-7709-42cb-8fe3-b63093fdcda7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Oct 30 07:05:41.697513 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:41.697476    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-bgnvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Oct 30 07:05:41.697686 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:41.697531    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bgnvh" podUID="14ba5fea-7709-42cb-8fe3-b63093fdcda7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Oct 30 07:05:42.697695 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:42.697657    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-bgnvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Oct 30 07:05:42.698213 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:42.697719    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bgnvh" podUID="14ba5fea-7709-42cb-8fe3-b63093fdcda7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Oct 30 07:05:43.698163 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:43.698121    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-bgnvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Oct 30 07:05:43.698637 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:43.698178    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bgnvh" podUID="14ba5fea-7709-42cb-8fe3-b63093fdcda7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Oct 30 07:05:44.697617 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:44.697582    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-bgnvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Oct 30 07:05:44.697854 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:44.697637    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bgnvh" podUID="14ba5fea-7709-42cb-8fe3-b63093fdcda7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Oct 30 07:05:46.130960 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:46.130927    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bgnvh"
Oct 30 07:05:46.131444 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:46.131327    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bgnvh"
Oct 30 07:05:46.132050 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:46.132027    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bgnvh"
Oct 30 07:05:52.364281 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:52.364249    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wkjb2"]
Oct 30 07:05:52.368021 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:52.367998    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wkjb2"
Oct 30 07:05:52.402215 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:52.402183    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac450c4b-8d00-4efb-9023-1681c843a993-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-wkjb2\" (UID: \"ac450c4b-8d00-4efb-9023-1681c843a993\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wkjb2"
Oct 30 07:05:52.402414 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:52.402223    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac450c4b-8d00-4efb-9023-1681c843a993-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-wkjb2\" (UID: \"ac450c4b-8d00-4efb-9023-1681c843a993\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wkjb2"
Oct 30 07:05:52.402414 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:52.402302    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac450c4b-8d00-4efb-9023-1681c843a993-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-wkjb2\" (UID: \"ac450c4b-8d00-4efb-9023-1681c843a993\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wkjb2"
Oct 30 07:05:52.402414 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:52.402368    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac450c4b-8d00-4efb-9023-1681c843a993-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-wkjb2\" (UID: \"ac450c4b-8d00-4efb-9023-1681c843a993\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wkjb2"
Oct 30 07:05:52.402414 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:52.402393    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac450c4b-8d00-4efb-9023-1681c843a993-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-wkjb2\" (UID: \"ac450c4b-8d00-4efb-9023-1681c843a993\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wkjb2"
Oct 30 07:05:52.402643 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:52.402453    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8kdz\" (UniqueName: \"kubernetes.io/projected/ac450c4b-8d00-4efb-9023-1681c843a993-kube-api-access-h8kdz\") pod \"maas-default-gateway-openshift-default-6dcf757899-wkjb2\" (UID: \"ac450c4b-8d00-4efb-9023-1681c843a993\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wkjb2"
Oct 30 07:05:52.402643 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:52.402481    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac450c4b-8d00-4efb-9023-1681c843a993-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-wkjb2\" (UID: \"ac450c4b-8d00-4efb-9023-1681c843a993\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wkjb2"
Oct 30 07:05:52.402643 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:52.402500    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac450c4b-8d00-4efb-9023-1681c843a993-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-wkjb2\" (UID: \"ac450c4b-8d00-4efb-9023-1681c843a993\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wkjb2"
Oct 30 07:05:52.402643 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:52.402533    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac450c4b-8d00-4efb-9023-1681c843a993-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-wkjb2\" (UID: \"ac450c4b-8d00-4efb-9023-1681c843a993\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wkjb2"
Oct 30 07:05:52.402877 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:52.402692    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wkjb2"]
Oct 30 07:05:52.503691 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:52.503654    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac450c4b-8d00-4efb-9023-1681c843a993-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-wkjb2\" (UID: \"ac450c4b-8d00-4efb-9023-1681c843a993\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wkjb2"
Oct 30 07:05:52.503891 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:52.503699    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac450c4b-8d00-4efb-9023-1681c843a993-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-wkjb2\" (UID: \"ac450c4b-8d00-4efb-9023-1681c843a993\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wkjb2"
Oct 30 07:05:52.503891 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:52.503744    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac450c4b-8d00-4efb-9023-1681c843a993-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-wkjb2\" (UID: \"ac450c4b-8d00-4efb-9023-1681c843a993\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wkjb2"
Oct 30 07:05:52.503891 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:52.503767    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac450c4b-8d00-4efb-9023-1681c843a993-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-wkjb2\" (UID: \"ac450c4b-8d00-4efb-9023-1681c843a993\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wkjb2"
Oct 30 07:05:52.503891 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:52.503794    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac450c4b-8d00-4efb-9023-1681c843a993-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-wkjb2\" (UID: \"ac450c4b-8d00-4efb-9023-1681c843a993\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wkjb2"
Oct 30 07:05:52.503891 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:52.503861    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8kdz\" (UniqueName: \"kubernetes.io/projected/ac450c4b-8d00-4efb-9023-1681c843a993-kube-api-access-h8kdz\") pod \"maas-default-gateway-openshift-default-6dcf757899-wkjb2\" (UID: \"ac450c4b-8d00-4efb-9023-1681c843a993\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wkjb2"
Oct 30 07:05:52.503891 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:52.503886    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac450c4b-8d00-4efb-9023-1681c843a993-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-wkjb2\" (UID: \"ac450c4b-8d00-4efb-9023-1681c843a993\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wkjb2"
Oct 30 07:05:52.504236 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:52.503914    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac450c4b-8d00-4efb-9023-1681c843a993-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-wkjb2\" (UID: \"ac450c4b-8d00-4efb-9023-1681c843a993\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wkjb2"
Oct 30 07:05:52.504236 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:52.504108    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac450c4b-8d00-4efb-9023-1681c843a993-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-wkjb2\" (UID: \"ac450c4b-8d00-4efb-9023-1681c843a993\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wkjb2"
Oct 30 07:05:52.504236 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:52.504163    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac450c4b-8d00-4efb-9023-1681c843a993-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-wkjb2\" (UID: \"ac450c4b-8d00-4efb-9023-1681c843a993\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wkjb2"
Oct 30 07:05:52.504386 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:52.504232    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac450c4b-8d00-4efb-9023-1681c843a993-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-wkjb2\" (UID: \"ac450c4b-8d00-4efb-9023-1681c843a993\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wkjb2"
Oct 30 07:05:52.504386 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:52.504265    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac450c4b-8d00-4efb-9023-1681c843a993-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-wkjb2\" (UID: \"ac450c4b-8d00-4efb-9023-1681c843a993\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wkjb2"
Oct 30 07:05:52.504386 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:52.504332    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac450c4b-8d00-4efb-9023-1681c843a993-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-wkjb2\" (UID: \"ac450c4b-8d00-4efb-9023-1681c843a993\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wkjb2"
Oct 30 07:05:52.504689 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:52.504667    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac450c4b-8d00-4efb-9023-1681c843a993-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-wkjb2\" (UID: \"ac450c4b-8d00-4efb-9023-1681c843a993\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wkjb2"
Oct 30 07:05:52.506298 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:52.506275    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac450c4b-8d00-4efb-9023-1681c843a993-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-wkjb2\" (UID: \"ac450c4b-8d00-4efb-9023-1681c843a993\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wkjb2"
Oct 30 07:05:52.506474 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:52.506456    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac450c4b-8d00-4efb-9023-1681c843a993-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-wkjb2\" (UID: \"ac450c4b-8d00-4efb-9023-1681c843a993\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wkjb2"
Oct 30 07:05:52.530879 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:52.530857    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac450c4b-8d00-4efb-9023-1681c843a993-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-wkjb2\" (UID: \"ac450c4b-8d00-4efb-9023-1681c843a993\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wkjb2"
Oct 30 07:05:52.531012 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:52.530996    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8kdz\" (UniqueName: \"kubernetes.io/projected/ac450c4b-8d00-4efb-9023-1681c843a993-kube-api-access-h8kdz\") pod \"maas-default-gateway-openshift-default-6dcf757899-wkjb2\" (UID: \"ac450c4b-8d00-4efb-9023-1681c843a993\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wkjb2"
Oct 30 07:05:52.680619 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:52.680536    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wkjb2"
Oct 30 07:05:52.989526 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:52.989457    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wkjb2"]
Oct 30 07:05:52.996792 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:05:52.996762    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac450c4b_8d00_4efb_9023_1681c843a993.slice/crio-98448383271d45d9f7afdbec2aad39b3b2f61457747dfda2ce31116ccb409e2c WatchSource:0}: Error finding container 98448383271d45d9f7afdbec2aad39b3b2f61457747dfda2ce31116ccb409e2c: Status 404 returned error can't find the container with id 98448383271d45d9f7afdbec2aad39b3b2f61457747dfda2ce31116ccb409e2c
Oct 30 07:05:52.998975 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:52.998945    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:05:52.999057 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:52.999007    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:05:52.999057 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:52.999036    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:05:53.993123 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:53.993072    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wkjb2" event={"ID":"ac450c4b-8d00-4efb-9023-1681c843a993","Type":"ContainerStarted","Data":"35a9ccadaaf89a0a11fd2aef81e111c7f460e4df5c346f97faa4edd7567dce01"}
Oct 30 07:05:53.993522 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:53.993133    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wkjb2" event={"ID":"ac450c4b-8d00-4efb-9023-1681c843a993","Type":"ContainerStarted","Data":"98448383271d45d9f7afdbec2aad39b3b2f61457747dfda2ce31116ccb409e2c"}
Oct 30 07:05:54.082832 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:54.076105    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wkjb2" podStartSLOduration=2.076083873 podStartE2EDuration="2.076083873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:05:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:05:54.059693689 +0000 UTC m=+512.669100658" watchObservedRunningTime="2026-10-30 07:05:54.076083873 +0000 UTC m=+512.685490845"
Oct 30 07:05:54.682948 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:54.682865    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wkjb2"
Oct 30 07:05:54.683402 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:54.683372    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-wkjb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 30 07:05:54.683609 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:54.683582    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wkjb2" podUID="ac450c4b-8d00-4efb-9023-1681c843a993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 30 07:05:55.684830 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:55.682221    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-wkjb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 30 07:05:55.684830 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:55.682283    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wkjb2" podUID="ac450c4b-8d00-4efb-9023-1681c843a993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 30 07:05:56.681255 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:56.681218    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-wkjb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 30 07:05:56.681521 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:56.681491    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wkjb2" podUID="ac450c4b-8d00-4efb-9023-1681c843a993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 30 07:05:57.684368 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:57.684330    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-wkjb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 30 07:05:57.685032 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:57.684995    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wkjb2" podUID="ac450c4b-8d00-4efb-9023-1681c843a993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 30 07:05:58.687831 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:58.687179    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-wkjb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 30 07:05:58.687831 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:58.687239    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wkjb2" podUID="ac450c4b-8d00-4efb-9023-1681c843a993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 30 07:05:59.681847 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:59.681788    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-wkjb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 30 07:05:59.682031 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:05:59.681859    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wkjb2" podUID="ac450c4b-8d00-4efb-9023-1681c843a993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 30 07:06:01.150231 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:01.147790    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:06:01.251524 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:01.251490    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wkjb2"
Oct 30 07:06:01.251948 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:01.251908    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wkjb2"
Oct 30 07:06:01.252913 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:01.252888    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wkjb2"
Oct 30 07:06:01.856489 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:06:01.856454    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:06:02.857227 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:02.857192    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-415d1917-79bf654d4-qrzxl"]
Oct 30 07:06:02.866946 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:02.866918    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-415d1917-79bf654d4-qrzxl"
Oct 30 07:06:02.881045 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:02.880980    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-415d1917-79bf654d4-qrzxl"]
Oct 30 07:06:02.908015 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:02.907430    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-415d1917\""
Oct 30 07:06:02.908015 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:02.907519    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-415d1917-dockercfg-lqsjk\""
Oct 30 07:06:02.998914 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:02.998881    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csfwh\" (UniqueName: \"kubernetes.io/projected/c6d2a268-b633-4fb4-ae09-e08fff300c79-kube-api-access-csfwh\") pod \"payload-processing-e2e-derive-415d1917-79bf654d4-qrzxl\" (UID: \"c6d2a268-b633-4fb4-ae09-e08fff300c79\") " pod="openshift-ingress/payload-processing-e2e-derive-415d1917-79bf654d4-qrzxl"
Oct 30 07:06:02.999091 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:02.998944    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6d2a268-b633-4fb4-ae09-e08fff300c79-plugins-config-volume\") pod \"payload-processing-e2e-derive-415d1917-79bf654d4-qrzxl\" (UID: \"c6d2a268-b633-4fb4-ae09-e08fff300c79\") " pod="openshift-ingress/payload-processing-e2e-derive-415d1917-79bf654d4-qrzxl"
Oct 30 07:06:03.099678 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:03.099641    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6d2a268-b633-4fb4-ae09-e08fff300c79-plugins-config-volume\") pod \"payload-processing-e2e-derive-415d1917-79bf654d4-qrzxl\" (UID: \"c6d2a268-b633-4fb4-ae09-e08fff300c79\") " pod="openshift-ingress/payload-processing-e2e-derive-415d1917-79bf654d4-qrzxl"
Oct 30 07:06:03.099866 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:03.099797    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csfwh\" (UniqueName: \"kubernetes.io/projected/c6d2a268-b633-4fb4-ae09-e08fff300c79-kube-api-access-csfwh\") pod \"payload-processing-e2e-derive-415d1917-79bf654d4-qrzxl\" (UID: \"c6d2a268-b633-4fb4-ae09-e08fff300c79\") " pod="openshift-ingress/payload-processing-e2e-derive-415d1917-79bf654d4-qrzxl"
Oct 30 07:06:03.100643 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:03.100623    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6d2a268-b633-4fb4-ae09-e08fff300c79-plugins-config-volume\") pod \"payload-processing-e2e-derive-415d1917-79bf654d4-qrzxl\" (UID: \"c6d2a268-b633-4fb4-ae09-e08fff300c79\") " pod="openshift-ingress/payload-processing-e2e-derive-415d1917-79bf654d4-qrzxl"
Oct 30 07:06:03.124771 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:03.124713    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csfwh\" (UniqueName: \"kubernetes.io/projected/c6d2a268-b633-4fb4-ae09-e08fff300c79-kube-api-access-csfwh\") pod \"payload-processing-e2e-derive-415d1917-79bf654d4-qrzxl\" (UID: \"c6d2a268-b633-4fb4-ae09-e08fff300c79\") " pod="openshift-ingress/payload-processing-e2e-derive-415d1917-79bf654d4-qrzxl"
Oct 30 07:06:03.180665 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:03.180632    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-415d1917-79bf654d4-qrzxl"
Oct 30 07:06:03.476450 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:03.476410    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-415d1917-79bf654d4-qrzxl"]
Oct 30 07:06:03.477132 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:06:03.477105    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6d2a268_b633_4fb4_ae09_e08fff300c79.slice/crio-d8c04ab562045c482e6b824206343cc875caccb4f7909d8e4faa5900d66e6bad WatchSource:0}: Error finding container d8c04ab562045c482e6b824206343cc875caccb4f7909d8e4faa5900d66e6bad: Status 404 returned error can't find the container with id d8c04ab562045c482e6b824206343cc875caccb4f7909d8e4faa5900d66e6bad
Oct 30 07:06:04.043397 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:04.043355    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-415d1917-79bf654d4-qrzxl" event={"ID":"c6d2a268-b633-4fb4-ae09-e08fff300c79","Type":"ContainerStarted","Data":"d8c04ab562045c482e6b824206343cc875caccb4f7909d8e4faa5900d66e6bad"}
Oct 30 07:06:06.055017 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:06.054979    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-415d1917-79bf654d4-qrzxl" event={"ID":"c6d2a268-b633-4fb4-ae09-e08fff300c79","Type":"ContainerStarted","Data":"6c3b55c62179a1f7cd776791f68ec4b774ce41dd49c352d82d74064476ebd275"}
Oct 30 07:06:06.055404 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:06.055068    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-415d1917-79bf654d4-qrzxl"
Oct 30 07:06:06.099740 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:06.099690    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-415d1917-79bf654d4-qrzxl" podStartSLOduration=2.144132264 podStartE2EDuration="4.099676367s" totalImagesPullingTime="1.955544103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:06:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:06:06.098703787 +0000 UTC m=+524.708110758" watchObservedRunningTime="2026-10-30 07:06:06.099676367 +0000 UTC m=+524.709083337"
Oct 30 07:06:06.578164 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:06.578127    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5b764790-549c4b49b7-8vdf7"]
Oct 30 07:06:06.581644 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:06.581624    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b764790-549c4b49b7-8vdf7"
Oct 30 07:06:06.600751 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:06.600714    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-5b764790-serving-cert\""
Oct 30 07:06:06.602828 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:06.602785    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5b764790-549c4b49b7-8vdf7"]
Oct 30 07:06:06.733531 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:06.733500    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b49162b-a69f-408e-ad97-d9882ffde8a6-metrics-certs\") pod \"maas-api-e2e-ait-5b764790-549c4b49b7-8vdf7\" (UID: \"4b49162b-a69f-408e-ad97-d9882ffde8a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b764790-549c4b49b7-8vdf7"
Oct 30 07:06:06.733703 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:06.733542    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b49162b-a69f-408e-ad97-d9882ffde8a6-maas-api-tls\") pod \"maas-api-e2e-ait-5b764790-549c4b49b7-8vdf7\" (UID: \"4b49162b-a69f-408e-ad97-d9882ffde8a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b764790-549c4b49b7-8vdf7"
Oct 30 07:06:06.733703 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:06.733655    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9bnk\" (UniqueName: \"kubernetes.io/projected/4b49162b-a69f-408e-ad97-d9882ffde8a6-kube-api-access-c9bnk\") pod \"maas-api-e2e-ait-5b764790-549c4b49b7-8vdf7\" (UID: \"4b49162b-a69f-408e-ad97-d9882ffde8a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b764790-549c4b49b7-8vdf7"
Oct 30 07:06:06.733774 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:06.733745    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b49162b-a69f-408e-ad97-d9882ffde8a6-openshift-service-ca\") pod \"maas-api-e2e-ait-5b764790-549c4b49b7-8vdf7\" (UID: \"4b49162b-a69f-408e-ad97-d9882ffde8a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b764790-549c4b49b7-8vdf7"
Oct 30 07:06:06.835070 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:06.834975    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b49162b-a69f-408e-ad97-d9882ffde8a6-maas-api-tls\") pod \"maas-api-e2e-ait-5b764790-549c4b49b7-8vdf7\" (UID: \"4b49162b-a69f-408e-ad97-d9882ffde8a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b764790-549c4b49b7-8vdf7"
Oct 30 07:06:06.835070 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:06.835026    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9bnk\" (UniqueName: \"kubernetes.io/projected/4b49162b-a69f-408e-ad97-d9882ffde8a6-kube-api-access-c9bnk\") pod \"maas-api-e2e-ait-5b764790-549c4b49b7-8vdf7\" (UID: \"4b49162b-a69f-408e-ad97-d9882ffde8a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b764790-549c4b49b7-8vdf7"
Oct 30 07:06:06.835309 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:06.835114    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b49162b-a69f-408e-ad97-d9882ffde8a6-openshift-service-ca\") pod \"maas-api-e2e-ait-5b764790-549c4b49b7-8vdf7\" (UID: \"4b49162b-a69f-408e-ad97-d9882ffde8a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b764790-549c4b49b7-8vdf7"
Oct 30 07:06:06.835309 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:06:06.835136    6224 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-5b764790-serving-cert: secret "maas-api-e2e-ait-5b764790-serving-cert" not found
Oct 30 07:06:06.835309 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:06.835160    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b49162b-a69f-408e-ad97-d9882ffde8a6-metrics-certs\") pod \"maas-api-e2e-ait-5b764790-549c4b49b7-8vdf7\" (UID: \"4b49162b-a69f-408e-ad97-d9882ffde8a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b764790-549c4b49b7-8vdf7"
Oct 30 07:06:06.835309 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:06:06.835233    6224 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b49162b-a69f-408e-ad97-d9882ffde8a6-maas-api-tls podName:4b49162b-a69f-408e-ad97-d9882ffde8a6 nodeName:}" failed. No retries permitted until 2026-10-30 07:06:07.335218125 +0000 UTC m=+525.944625098 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4b49162b-a69f-408e-ad97-d9882ffde8a6-maas-api-tls") pod "maas-api-e2e-ait-5b764790-549c4b49b7-8vdf7" (UID: "4b49162b-a69f-408e-ad97-d9882ffde8a6") : secret "maas-api-e2e-ait-5b764790-serving-cert" not found
Oct 30 07:06:06.835775 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:06.835752    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b49162b-a69f-408e-ad97-d9882ffde8a6-openshift-service-ca\") pod \"maas-api-e2e-ait-5b764790-549c4b49b7-8vdf7\" (UID: \"4b49162b-a69f-408e-ad97-d9882ffde8a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b764790-549c4b49b7-8vdf7"
Oct 30 07:06:06.837606 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:06.837584    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b49162b-a69f-408e-ad97-d9882ffde8a6-metrics-certs\") pod \"maas-api-e2e-ait-5b764790-549c4b49b7-8vdf7\" (UID: \"4b49162b-a69f-408e-ad97-d9882ffde8a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b764790-549c4b49b7-8vdf7"
Oct 30 07:06:06.860829 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:06.860777    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9bnk\" (UniqueName: \"kubernetes.io/projected/4b49162b-a69f-408e-ad97-d9882ffde8a6-kube-api-access-c9bnk\") pod \"maas-api-e2e-ait-5b764790-549c4b49b7-8vdf7\" (UID: \"4b49162b-a69f-408e-ad97-d9882ffde8a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b764790-549c4b49b7-8vdf7"
Oct 30 07:06:07.339766 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:07.339720    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b49162b-a69f-408e-ad97-d9882ffde8a6-maas-api-tls\") pod \"maas-api-e2e-ait-5b764790-549c4b49b7-8vdf7\" (UID: \"4b49162b-a69f-408e-ad97-d9882ffde8a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b764790-549c4b49b7-8vdf7"
Oct 30 07:06:07.341977 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:07.341951    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b49162b-a69f-408e-ad97-d9882ffde8a6-maas-api-tls\") pod \"maas-api-e2e-ait-5b764790-549c4b49b7-8vdf7\" (UID: \"4b49162b-a69f-408e-ad97-d9882ffde8a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b764790-549c4b49b7-8vdf7"
Oct 30 07:06:07.494506 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:07.494465    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b764790-549c4b49b7-8vdf7"
Oct 30 07:06:07.819874 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:07.819834    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5b764790-549c4b49b7-8vdf7"]
Oct 30 07:06:08.065700 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:08.065664    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b764790-549c4b49b7-8vdf7" event={"ID":"4b49162b-a69f-408e-ad97-d9882ffde8a6","Type":"ContainerStarted","Data":"506ac4415401e72d8866ca613cf361f79969d9931a4b6172c3a06d42e474d4ce"}
Oct 30 07:06:09.072293 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:09.072258    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b764790-549c4b49b7-8vdf7" event={"ID":"4b49162b-a69f-408e-ad97-d9882ffde8a6","Type":"ContainerStarted","Data":"85a53eb94b4ff29e0a63fe34939cce061e13f5e050ac155e854ab562ae9f2ee6"}
Oct 30 07:06:09.072693 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:09.072324    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b764790-549c4b49b7-8vdf7"
Oct 30 07:06:09.121917 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:09.121871    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b764790-549c4b49b7-8vdf7" podStartSLOduration=2.801684236 podStartE2EDuration="3.121858301s" totalImagesPullingTime="320.174065ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:06:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:06:09.121516081 +0000 UTC m=+527.730923051" watchObservedRunningTime="2026-10-30 07:06:09.121858301 +0000 UTC m=+527.731265270"
Oct 30 07:06:10.352092 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:10.352058    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:06:10.620674 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:10.620597    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-415d1917-5768bccb44-ldsfg"]
Oct 30 07:06:10.624198 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:10.624183    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-415d1917-5768bccb44-ldsfg"
Oct 30 07:06:10.657194 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:10.657165    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-415d1917-serving-cert\""
Oct 30 07:06:10.659650 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:10.659623    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-415d1917-5768bccb44-ldsfg"]
Oct 30 07:06:10.770275 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:10.770242    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmdfm\" (UniqueName: \"kubernetes.io/projected/cfa8014b-1cac-4a52-b1bf-287a060328cd-kube-api-access-lmdfm\") pod \"maas-api-e2e-derive-415d1917-5768bccb44-ldsfg\" (UID: \"cfa8014b-1cac-4a52-b1bf-287a060328cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-415d1917-5768bccb44-ldsfg"
Oct 30 07:06:10.770461 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:10.770313    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfa8014b-1cac-4a52-b1bf-287a060328cd-maas-api-tls\") pod \"maas-api-e2e-derive-415d1917-5768bccb44-ldsfg\" (UID: \"cfa8014b-1cac-4a52-b1bf-287a060328cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-415d1917-5768bccb44-ldsfg"
Oct 30 07:06:10.770461 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:10.770340    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfa8014b-1cac-4a52-b1bf-287a060328cd-openshift-service-ca\") pod \"maas-api-e2e-derive-415d1917-5768bccb44-ldsfg\" (UID: \"cfa8014b-1cac-4a52-b1bf-287a060328cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-415d1917-5768bccb44-ldsfg"
Oct 30 07:06:10.770461 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:10.770369    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfa8014b-1cac-4a52-b1bf-287a060328cd-metrics-certs\") pod \"maas-api-e2e-derive-415d1917-5768bccb44-ldsfg\" (UID: \"cfa8014b-1cac-4a52-b1bf-287a060328cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-415d1917-5768bccb44-ldsfg"
Oct 30 07:06:10.871055 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:10.870960    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfa8014b-1cac-4a52-b1bf-287a060328cd-maas-api-tls\") pod \"maas-api-e2e-derive-415d1917-5768bccb44-ldsfg\" (UID: \"cfa8014b-1cac-4a52-b1bf-287a060328cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-415d1917-5768bccb44-ldsfg"
Oct 30 07:06:10.871055 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:10.871002    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfa8014b-1cac-4a52-b1bf-287a060328cd-openshift-service-ca\") pod \"maas-api-e2e-derive-415d1917-5768bccb44-ldsfg\" (UID: \"cfa8014b-1cac-4a52-b1bf-287a060328cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-415d1917-5768bccb44-ldsfg"
Oct 30 07:06:10.871055 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:10.871019    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfa8014b-1cac-4a52-b1bf-287a060328cd-metrics-certs\") pod \"maas-api-e2e-derive-415d1917-5768bccb44-ldsfg\" (UID: \"cfa8014b-1cac-4a52-b1bf-287a060328cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-415d1917-5768bccb44-ldsfg"
Oct 30 07:06:10.871325 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:06:10.871143    6224 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-415d1917-serving-cert: secret "maas-api-e2e-derive-415d1917-serving-cert" not found
Oct 30 07:06:10.871325 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:10.871145    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmdfm\" (UniqueName: \"kubernetes.io/projected/cfa8014b-1cac-4a52-b1bf-287a060328cd-kube-api-access-lmdfm\") pod \"maas-api-e2e-derive-415d1917-5768bccb44-ldsfg\" (UID: \"cfa8014b-1cac-4a52-b1bf-287a060328cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-415d1917-5768bccb44-ldsfg"
Oct 30 07:06:10.871325 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:06:10.871221    6224 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfa8014b-1cac-4a52-b1bf-287a060328cd-maas-api-tls podName:cfa8014b-1cac-4a52-b1bf-287a060328cd nodeName:}" failed. No retries permitted until 2026-10-30 07:06:11.371199829 +0000 UTC m=+529.980606780 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cfa8014b-1cac-4a52-b1bf-287a060328cd-maas-api-tls") pod "maas-api-e2e-derive-415d1917-5768bccb44-ldsfg" (UID: "cfa8014b-1cac-4a52-b1bf-287a060328cd") : secret "maas-api-e2e-derive-415d1917-serving-cert" not found
Oct 30 07:06:10.871668 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:10.871647    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfa8014b-1cac-4a52-b1bf-287a060328cd-openshift-service-ca\") pod \"maas-api-e2e-derive-415d1917-5768bccb44-ldsfg\" (UID: \"cfa8014b-1cac-4a52-b1bf-287a060328cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-415d1917-5768bccb44-ldsfg"
Oct 30 07:06:10.873441 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:10.873418    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfa8014b-1cac-4a52-b1bf-287a060328cd-metrics-certs\") pod \"maas-api-e2e-derive-415d1917-5768bccb44-ldsfg\" (UID: \"cfa8014b-1cac-4a52-b1bf-287a060328cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-415d1917-5768bccb44-ldsfg"
Oct 30 07:06:10.898574 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:10.898553    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmdfm\" (UniqueName: \"kubernetes.io/projected/cfa8014b-1cac-4a52-b1bf-287a060328cd-kube-api-access-lmdfm\") pod \"maas-api-e2e-derive-415d1917-5768bccb44-ldsfg\" (UID: \"cfa8014b-1cac-4a52-b1bf-287a060328cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-415d1917-5768bccb44-ldsfg"
Oct 30 07:06:11.376325 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:11.376285    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfa8014b-1cac-4a52-b1bf-287a060328cd-maas-api-tls\") pod \"maas-api-e2e-derive-415d1917-5768bccb44-ldsfg\" (UID: \"cfa8014b-1cac-4a52-b1bf-287a060328cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-415d1917-5768bccb44-ldsfg"
Oct 30 07:06:11.378569 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:11.378548    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfa8014b-1cac-4a52-b1bf-287a060328cd-maas-api-tls\") pod \"maas-api-e2e-derive-415d1917-5768bccb44-ldsfg\" (UID: \"cfa8014b-1cac-4a52-b1bf-287a060328cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-415d1917-5768bccb44-ldsfg"
Oct 30 07:06:11.536937 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:11.536908    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-415d1917-5768bccb44-ldsfg"
Oct 30 07:06:11.837373 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:11.837345    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-415d1917-5768bccb44-ldsfg"]
Oct 30 07:06:11.837914 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:06:11.837890    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfa8014b_1cac_4a52_b1bf_287a060328cd.slice/crio-a938a98fbc8930761f95f04e0d3b9f2b362a9761d4a4749c4e42debc7d3dcde6 WatchSource:0}: Error finding container a938a98fbc8930761f95f04e0d3b9f2b362a9761d4a4749c4e42debc7d3dcde6: Status 404 returned error can't find the container with id a938a98fbc8930761f95f04e0d3b9f2b362a9761d4a4749c4e42debc7d3dcde6
Oct 30 07:06:12.088437 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:12.088359    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-415d1917-5768bccb44-ldsfg" event={"ID":"cfa8014b-1cac-4a52-b1bf-287a060328cd","Type":"ContainerStarted","Data":"a938a98fbc8930761f95f04e0d3b9f2b362a9761d4a4749c4e42debc7d3dcde6"}
Oct 30 07:06:13.094622 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:13.094582    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-415d1917-5768bccb44-ldsfg" event={"ID":"cfa8014b-1cac-4a52-b1bf-287a060328cd","Type":"ContainerStarted","Data":"64548328fdf0e0781a3a985fbe96babc57e738a0b16fcc141b9413b9bb1486ba"}
Oct 30 07:06:13.097068 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:13.094644    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-415d1917-5768bccb44-ldsfg"
Oct 30 07:06:13.142419 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:13.142367    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-415d1917-5768bccb44-ldsfg" podStartSLOduration=2.807029019 podStartE2EDuration="3.142353739s" totalImagesPullingTime="335.32472ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:06:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:06:13.141300324 +0000 UTC m=+531.750707293" watchObservedRunningTime="2026-10-30 07:06:13.142353739 +0000 UTC m=+531.751760708"
Oct 30 07:06:15.082207 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:15.082178    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b764790-549c4b49b7-8vdf7"
Oct 30 07:06:17.063162 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:17.063128    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-415d1917-79bf654d4-qrzxl"
Oct 30 07:06:17.288846 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:17.288816    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:06:19.104467 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:19.104437    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-415d1917-5768bccb44-ldsfg"
Oct 30 07:06:29.605332 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:29.605297    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:06:31.856349 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:06:31.856310    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:06:36.441673 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:36.441590    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-415d1917-79bf654d4-qrzxl"]
Oct 30 07:06:39.389710 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:39.389681    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5b764790-549c4b49b7-8vdf7"]
Oct 30 07:06:39.390197 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:39.389902    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b764790-549c4b49b7-8vdf7" podUID="4b49162b-a69f-408e-ad97-d9882ffde8a6" containerName="maas-api" containerID="cri-o://85a53eb94b4ff29e0a63fe34939cce061e13f5e050ac155e854ab562ae9f2ee6" gracePeriod=30
Oct 30 07:06:39.637227 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:39.637205    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b764790-549c4b49b7-8vdf7"
Oct 30 07:06:39.720943 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:39.720860    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b49162b-a69f-408e-ad97-d9882ffde8a6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b49162b-a69f-408e-ad97-d9882ffde8a6-maas-api-tls\") pod \"4b49162b-a69f-408e-ad97-d9882ffde8a6\" (UID: \"4b49162b-a69f-408e-ad97-d9882ffde8a6\") "
Oct 30 07:06:39.721103 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:39.720951    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b49162b-a69f-408e-ad97-d9882ffde8a6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b49162b-a69f-408e-ad97-d9882ffde8a6-metrics-certs\") pod \"4b49162b-a69f-408e-ad97-d9882ffde8a6\" (UID: \"4b49162b-a69f-408e-ad97-d9882ffde8a6\") "
Oct 30 07:06:39.721103 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:39.721042    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b49162b-a69f-408e-ad97-d9882ffde8a6-kube-api-access-c9bnk\" (UniqueName: \"kubernetes.io/projected/4b49162b-a69f-408e-ad97-d9882ffde8a6-kube-api-access-c9bnk\") pod \"4b49162b-a69f-408e-ad97-d9882ffde8a6\" (UID: \"4b49162b-a69f-408e-ad97-d9882ffde8a6\") "
Oct 30 07:06:39.721103 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:39.721097    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b49162b-a69f-408e-ad97-d9882ffde8a6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b49162b-a69f-408e-ad97-d9882ffde8a6-openshift-service-ca\") pod \"4b49162b-a69f-408e-ad97-d9882ffde8a6\" (UID: \"4b49162b-a69f-408e-ad97-d9882ffde8a6\") "
Oct 30 07:06:39.721571 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:39.721538    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b49162b-a69f-408e-ad97-d9882ffde8a6-openshift-service-ca" pod "4b49162b-a69f-408e-ad97-d9882ffde8a6" (UID: "4b49162b-a69f-408e-ad97-d9882ffde8a6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:06:39.723354 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:39.723264    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b49162b-a69f-408e-ad97-d9882ffde8a6-maas-api-tls" pod "4b49162b-a69f-408e-ad97-d9882ffde8a6" (UID: "4b49162b-a69f-408e-ad97-d9882ffde8a6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:06:39.723354 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:39.723307    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b49162b-a69f-408e-ad97-d9882ffde8a6-kube-api-access-c9bnk" pod "4b49162b-a69f-408e-ad97-d9882ffde8a6" (UID: "4b49162b-a69f-408e-ad97-d9882ffde8a6"). InnerVolumeSpecName "kube-api-access-c9bnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:06:39.723535 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:39.723444    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b49162b-a69f-408e-ad97-d9882ffde8a6-metrics-certs" pod "4b49162b-a69f-408e-ad97-d9882ffde8a6" (UID: "4b49162b-a69f-408e-ad97-d9882ffde8a6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:06:39.822704 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:39.822681    6224 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b49162b-a69f-408e-ad97-d9882ffde8a6-maas-api-tls\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:06:39.822704 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:39.822704    6224 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b49162b-a69f-408e-ad97-d9882ffde8a6-metrics-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:06:39.822882 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:39.822714    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9bnk\" (UniqueName: \"kubernetes.io/projected/4b49162b-a69f-408e-ad97-d9882ffde8a6-kube-api-access-c9bnk\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:06:39.822882 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:39.822727    6224 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b49162b-a69f-408e-ad97-d9882ffde8a6-openshift-service-ca\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:06:39.860839 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:39.860788    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-415d1917-5768bccb44-ldsfg"]
Oct 30 07:06:39.861057 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:39.861035    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-415d1917-5768bccb44-ldsfg" podUID="cfa8014b-1cac-4a52-b1bf-287a060328cd" containerName="maas-api" containerID="cri-o://64548328fdf0e0781a3a985fbe96babc57e738a0b16fcc141b9413b9bb1486ba" gracePeriod=30
Oct 30 07:06:40.093351 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:40.093326    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-415d1917-5768bccb44-ldsfg"
Oct 30 07:06:40.124908 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:40.124883    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfa8014b-1cac-4a52-b1bf-287a060328cd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfa8014b-1cac-4a52-b1bf-287a060328cd-maas-api-tls\") pod \"cfa8014b-1cac-4a52-b1bf-287a060328cd\" (UID: \"cfa8014b-1cac-4a52-b1bf-287a060328cd\") "
Oct 30 07:06:40.125091 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:40.125073    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfa8014b-1cac-4a52-b1bf-287a060328cd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfa8014b-1cac-4a52-b1bf-287a060328cd-openshift-service-ca\") pod \"cfa8014b-1cac-4a52-b1bf-287a060328cd\" (UID: \"cfa8014b-1cac-4a52-b1bf-287a060328cd\") "
Oct 30 07:06:40.125147 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:40.125119    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfa8014b-1cac-4a52-b1bf-287a060328cd-kube-api-access-lmdfm\" (UniqueName: \"kubernetes.io/projected/cfa8014b-1cac-4a52-b1bf-287a060328cd-kube-api-access-lmdfm\") pod \"cfa8014b-1cac-4a52-b1bf-287a060328cd\" (UID: \"cfa8014b-1cac-4a52-b1bf-287a060328cd\") "
Oct 30 07:06:40.125209 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:40.125163    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfa8014b-1cac-4a52-b1bf-287a060328cd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfa8014b-1cac-4a52-b1bf-287a060328cd-metrics-certs\") pod \"cfa8014b-1cac-4a52-b1bf-287a060328cd\" (UID: \"cfa8014b-1cac-4a52-b1bf-287a060328cd\") "
Oct 30 07:06:40.125469 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:40.125446    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfa8014b-1cac-4a52-b1bf-287a060328cd-openshift-service-ca" pod "cfa8014b-1cac-4a52-b1bf-287a060328cd" (UID: "cfa8014b-1cac-4a52-b1bf-287a060328cd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:06:40.126973 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:40.126941    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfa8014b-1cac-4a52-b1bf-287a060328cd-maas-api-tls" pod "cfa8014b-1cac-4a52-b1bf-287a060328cd" (UID: "cfa8014b-1cac-4a52-b1bf-287a060328cd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:06:40.127059 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:40.127044    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfa8014b-1cac-4a52-b1bf-287a060328cd-kube-api-access-lmdfm" pod "cfa8014b-1cac-4a52-b1bf-287a060328cd" (UID: "cfa8014b-1cac-4a52-b1bf-287a060328cd"). InnerVolumeSpecName "kube-api-access-lmdfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:06:40.127102 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:40.127079    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfa8014b-1cac-4a52-b1bf-287a060328cd-metrics-certs" pod "cfa8014b-1cac-4a52-b1bf-287a060328cd" (UID: "cfa8014b-1cac-4a52-b1bf-287a060328cd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:06:40.225853 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:40.225761    6224 generic.go:356] "Generic (PLEG): container finished" podID="4b49162b-a69f-408e-ad97-d9882ffde8a6" containerID="85a53eb94b4ff29e0a63fe34939cce061e13f5e050ac155e854ab562ae9f2ee6" exitCode=0
Oct 30 07:06:40.225853 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:40.225827    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b764790-549c4b49b7-8vdf7"
Oct 30 07:06:40.225853 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:40.225836    6224 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfa8014b-1cac-4a52-b1bf-287a060328cd-openshift-service-ca\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:06:40.225853 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:40.225854    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmdfm\" (UniqueName: \"kubernetes.io/projected/cfa8014b-1cac-4a52-b1bf-287a060328cd-kube-api-access-lmdfm\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:06:40.226148 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:40.225864    6224 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfa8014b-1cac-4a52-b1bf-287a060328cd-metrics-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:06:40.226148 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:40.225875    6224 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfa8014b-1cac-4a52-b1bf-287a060328cd-maas-api-tls\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:06:40.226148 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:40.225845    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b764790-549c4b49b7-8vdf7" event={"ID":"4b49162b-a69f-408e-ad97-d9882ffde8a6","Type":"ContainerDied","Data":"85a53eb94b4ff29e0a63fe34939cce061e13f5e050ac155e854ab562ae9f2ee6"}
Oct 30 07:06:40.226148 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:40.225924    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b764790-549c4b49b7-8vdf7" event={"ID":"4b49162b-a69f-408e-ad97-d9882ffde8a6","Type":"ContainerDied","Data":"506ac4415401e72d8866ca613cf361f79969d9931a4b6172c3a06d42e474d4ce"}
Oct 30 07:06:40.226148 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:40.225953    6224 scope.go:122] "RemoveContainer" containerID="85a53eb94b4ff29e0a63fe34939cce061e13f5e050ac155e854ab562ae9f2ee6"
Oct 30 07:06:40.227300 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:40.227272    6224 generic.go:356] "Generic (PLEG): container finished" podID="cfa8014b-1cac-4a52-b1bf-287a060328cd" containerID="64548328fdf0e0781a3a985fbe96babc57e738a0b16fcc141b9413b9bb1486ba" exitCode=0
Oct 30 07:06:40.227433 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:40.227319    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-415d1917-5768bccb44-ldsfg"
Oct 30 07:06:40.227433 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:40.227322    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-415d1917-5768bccb44-ldsfg" event={"ID":"cfa8014b-1cac-4a52-b1bf-287a060328cd","Type":"ContainerDied","Data":"64548328fdf0e0781a3a985fbe96babc57e738a0b16fcc141b9413b9bb1486ba"}
Oct 30 07:06:40.227433 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:40.227383    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-415d1917-5768bccb44-ldsfg" event={"ID":"cfa8014b-1cac-4a52-b1bf-287a060328cd","Type":"ContainerDied","Data":"a938a98fbc8930761f95f04e0d3b9f2b362a9761d4a4749c4e42debc7d3dcde6"}
Oct 30 07:06:40.240706 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:40.240688    6224 scope.go:122] "RemoveContainer" containerID="85a53eb94b4ff29e0a63fe34939cce061e13f5e050ac155e854ab562ae9f2ee6"
Oct 30 07:06:40.241014 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:06:40.240996    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85a53eb94b4ff29e0a63fe34939cce061e13f5e050ac155e854ab562ae9f2ee6\": container with ID starting with 85a53eb94b4ff29e0a63fe34939cce061e13f5e050ac155e854ab562ae9f2ee6 not found: ID does not exist" containerID="85a53eb94b4ff29e0a63fe34939cce061e13f5e050ac155e854ab562ae9f2ee6"
Oct 30 07:06:40.241076 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:40.241021    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85a53eb94b4ff29e0a63fe34939cce061e13f5e050ac155e854ab562ae9f2ee6"} err="failed to get container status \"85a53eb94b4ff29e0a63fe34939cce061e13f5e050ac155e854ab562ae9f2ee6\": rpc error: code = NotFound desc = could not find container \"85a53eb94b4ff29e0a63fe34939cce061e13f5e050ac155e854ab562ae9f2ee6\": container with ID starting with 85a53eb94b4ff29e0a63fe34939cce061e13f5e050ac155e854ab562ae9f2ee6 not found: ID does not exist"
Oct 30 07:06:40.241076 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:40.241038    6224 scope.go:122] "RemoveContainer" containerID="64548328fdf0e0781a3a985fbe96babc57e738a0b16fcc141b9413b9bb1486ba"
Oct 30 07:06:40.249250 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:40.249234    6224 scope.go:122] "RemoveContainer" containerID="64548328fdf0e0781a3a985fbe96babc57e738a0b16fcc141b9413b9bb1486ba"
Oct 30 07:06:40.249513 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:06:40.249498    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64548328fdf0e0781a3a985fbe96babc57e738a0b16fcc141b9413b9bb1486ba\": container with ID starting with 64548328fdf0e0781a3a985fbe96babc57e738a0b16fcc141b9413b9bb1486ba not found: ID does not exist" containerID="64548328fdf0e0781a3a985fbe96babc57e738a0b16fcc141b9413b9bb1486ba"
Oct 30 07:06:40.249554 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:40.249520    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64548328fdf0e0781a3a985fbe96babc57e738a0b16fcc141b9413b9bb1486ba"} err="failed to get container status \"64548328fdf0e0781a3a985fbe96babc57e738a0b16fcc141b9413b9bb1486ba\": rpc error: code = NotFound desc = could not find container \"64548328fdf0e0781a3a985fbe96babc57e738a0b16fcc141b9413b9bb1486ba\": container with ID starting with 64548328fdf0e0781a3a985fbe96babc57e738a0b16fcc141b9413b9bb1486ba not found: ID does not exist"
Oct 30 07:06:40.290046 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:40.290022    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-415d1917-5768bccb44-ldsfg"]
Oct 30 07:06:40.297794 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:40.297763    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-415d1917-5768bccb44-ldsfg"]
Oct 30 07:06:40.337684 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:40.337647    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5b764790-549c4b49b7-8vdf7"]
Oct 30 07:06:40.341541 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:40.341517    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5b764790-549c4b49b7-8vdf7"]
Oct 30 07:06:40.989674 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:40.989633    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:06:41.442203 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:41.442145    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-415d1917-79bf654d4-qrzxl" podUID="c6d2a268-b633-4fb4-ae09-e08fff300c79" containerName="payload-processing" containerID="cri-o://6c3b55c62179a1f7cd776791f68ec4b774ce41dd49c352d82d74064476ebd275" gracePeriod=25
Oct 30 07:06:41.502902 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:41.502870    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b49162b-a69f-408e-ad97-d9882ffde8a6" path="/var/lib/kubelet/pods/4b49162b-a69f-408e-ad97-d9882ffde8a6/volumes"
Oct 30 07:06:41.503281 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:41.503268    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfa8014b-1cac-4a52-b1bf-287a060328cd" path="/var/lib/kubelet/pods/cfa8014b-1cac-4a52-b1bf-287a060328cd/volumes"
Oct 30 07:06:41.689939 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:41.689912    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-415d1917-79bf654d4-qrzxl"
Oct 30 07:06:41.737301 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:41.737227    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6d2a268-b633-4fb4-ae09-e08fff300c79-kube-api-access-csfwh\" (UniqueName: \"kubernetes.io/projected/c6d2a268-b633-4fb4-ae09-e08fff300c79-kube-api-access-csfwh\") pod \"c6d2a268-b633-4fb4-ae09-e08fff300c79\" (UID: \"c6d2a268-b633-4fb4-ae09-e08fff300c79\") "
Oct 30 07:06:41.737301 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:41.737293    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6d2a268-b633-4fb4-ae09-e08fff300c79-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6d2a268-b633-4fb4-ae09-e08fff300c79-plugins-config-volume\") pod \"c6d2a268-b633-4fb4-ae09-e08fff300c79\" (UID: \"c6d2a268-b633-4fb4-ae09-e08fff300c79\") "
Oct 30 07:06:41.737687 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:41.737661    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6d2a268-b633-4fb4-ae09-e08fff300c79-plugins-config-volume" pod "c6d2a268-b633-4fb4-ae09-e08fff300c79" (UID: "c6d2a268-b633-4fb4-ae09-e08fff300c79"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:06:41.739520 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:41.739489    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6d2a268-b633-4fb4-ae09-e08fff300c79-kube-api-access-csfwh" pod "c6d2a268-b633-4fb4-ae09-e08fff300c79" (UID: "c6d2a268-b633-4fb4-ae09-e08fff300c79"). InnerVolumeSpecName "kube-api-access-csfwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:06:41.838676 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:41.838641    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csfwh\" (UniqueName: \"kubernetes.io/projected/c6d2a268-b633-4fb4-ae09-e08fff300c79-kube-api-access-csfwh\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:06:41.838676 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:41.838669    6224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6d2a268-b633-4fb4-ae09-e08fff300c79-plugins-config-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:06:42.240534 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:42.240502    6224 generic.go:356] "Generic (PLEG): container finished" podID="c6d2a268-b633-4fb4-ae09-e08fff300c79" containerID="6c3b55c62179a1f7cd776791f68ec4b774ce41dd49c352d82d74064476ebd275" exitCode=0
Oct 30 07:06:42.241020 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:42.240549    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-415d1917-79bf654d4-qrzxl"
Oct 30 07:06:42.241020 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:42.240573    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-415d1917-79bf654d4-qrzxl" event={"ID":"c6d2a268-b633-4fb4-ae09-e08fff300c79","Type":"ContainerDied","Data":"6c3b55c62179a1f7cd776791f68ec4b774ce41dd49c352d82d74064476ebd275"}
Oct 30 07:06:42.241020 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:42.240614    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-415d1917-79bf654d4-qrzxl" event={"ID":"c6d2a268-b633-4fb4-ae09-e08fff300c79","Type":"ContainerDied","Data":"d8c04ab562045c482e6b824206343cc875caccb4f7909d8e4faa5900d66e6bad"}
Oct 30 07:06:42.241020 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:42.240635    6224 scope.go:122] "RemoveContainer" containerID="6c3b55c62179a1f7cd776791f68ec4b774ce41dd49c352d82d74064476ebd275"
Oct 30 07:06:42.250022 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:42.250001    6224 scope.go:122] "RemoveContainer" containerID="6c3b55c62179a1f7cd776791f68ec4b774ce41dd49c352d82d74064476ebd275"
Oct 30 07:06:42.250320 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:06:42.250300    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c3b55c62179a1f7cd776791f68ec4b774ce41dd49c352d82d74064476ebd275\": container with ID starting with 6c3b55c62179a1f7cd776791f68ec4b774ce41dd49c352d82d74064476ebd275 not found: ID does not exist" containerID="6c3b55c62179a1f7cd776791f68ec4b774ce41dd49c352d82d74064476ebd275"
Oct 30 07:06:42.250408 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:42.250367    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c3b55c62179a1f7cd776791f68ec4b774ce41dd49c352d82d74064476ebd275"} err="failed to get container status \"6c3b55c62179a1f7cd776791f68ec4b774ce41dd49c352d82d74064476ebd275\": rpc error: code = NotFound desc = could not find container \"6c3b55c62179a1f7cd776791f68ec4b774ce41dd49c352d82d74064476ebd275\": container with ID starting with 6c3b55c62179a1f7cd776791f68ec4b774ce41dd49c352d82d74064476ebd275 not found: ID does not exist"
Oct 30 07:06:42.296782 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:42.296747    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-415d1917-79bf654d4-qrzxl"]
Oct 30 07:06:42.310832 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:42.310779    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-415d1917-79bf654d4-qrzxl"]
Oct 30 07:06:43.501771 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:43.501740    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6d2a268-b633-4fb4-ae09-e08fff300c79" path="/var/lib/kubelet/pods/c6d2a268-b633-4fb4-ae09-e08fff300c79/volumes"
Oct 30 07:06:44.758257 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:44.758223    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-2bab4c-56fcccb486-2wtg7"]
Oct 30 07:06:44.758766 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:44.758746    6224 cpu_manager.go:414] "containerMap: removing container" podUID="c6d2a268-b633-4fb4-ae09-e08fff300c79" containerName="payload-processing"
Oct 30 07:06:44.758880 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:44.758769    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6d2a268-b633-4fb4-ae09-e08fff300c79" containerName="payload-processing"
Oct 30 07:06:44.758880 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:44.758782    6224 cpu_manager.go:414] "containerMap: removing container" podUID="4b49162b-a69f-408e-ad97-d9882ffde8a6" containerName="maas-api"
Oct 30 07:06:44.758880 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:44.758788    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b49162b-a69f-408e-ad97-d9882ffde8a6" containerName="maas-api"
Oct 30 07:06:44.758880 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:44.758827    6224 cpu_manager.go:414] "containerMap: removing container" podUID="cfa8014b-1cac-4a52-b1bf-287a060328cd" containerName="maas-api"
Oct 30 07:06:44.758880 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:44.758836    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfa8014b-1cac-4a52-b1bf-287a060328cd" containerName="maas-api"
Oct 30 07:06:44.759127 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:44.758921    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfa8014b-1cac-4a52-b1bf-287a060328cd" containerName="maas-api"
Oct 30 07:06:44.759127 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:44.758940    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6d2a268-b633-4fb4-ae09-e08fff300c79" containerName="payload-processing"
Oct 30 07:06:44.759127 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:44.758950    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b49162b-a69f-408e-ad97-d9882ffde8a6" containerName="maas-api"
Oct 30 07:06:44.763238 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:44.763216    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-2bab4c-56fcccb486-2wtg7"
Oct 30 07:06:44.781297 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:44.781278    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-2bab4c\""
Oct 30 07:06:44.799373 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:44.799345    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-2bab4c-56fcccb486-2wtg7"]
Oct 30 07:06:44.865618 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:44.865584    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8dcq\" (UniqueName: \"kubernetes.io/projected/1f058cca-8314-4ff2-aa1f-3169f7d770fa-kube-api-access-r8dcq\") pod \"payload-processing-e2e-worker-w4-2bab4c-56fcccb486-2wtg7\" (UID: \"1f058cca-8314-4ff2-aa1f-3169f7d770fa\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-2bab4c-56fcccb486-2wtg7"
Oct 30 07:06:44.865788 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:44.865630    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f058cca-8314-4ff2-aa1f-3169f7d770fa-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-2bab4c-56fcccb486-2wtg7\" (UID: \"1f058cca-8314-4ff2-aa1f-3169f7d770fa\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-2bab4c-56fcccb486-2wtg7"
Oct 30 07:06:44.967086 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:44.967048    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8dcq\" (UniqueName: \"kubernetes.io/projected/1f058cca-8314-4ff2-aa1f-3169f7d770fa-kube-api-access-r8dcq\") pod \"payload-processing-e2e-worker-w4-2bab4c-56fcccb486-2wtg7\" (UID: \"1f058cca-8314-4ff2-aa1f-3169f7d770fa\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-2bab4c-56fcccb486-2wtg7"
Oct 30 07:06:44.967265 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:44.967097    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f058cca-8314-4ff2-aa1f-3169f7d770fa-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-2bab4c-56fcccb486-2wtg7\" (UID: \"1f058cca-8314-4ff2-aa1f-3169f7d770fa\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-2bab4c-56fcccb486-2wtg7"
Oct 30 07:06:44.967773 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:44.967754    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f058cca-8314-4ff2-aa1f-3169f7d770fa-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-2bab4c-56fcccb486-2wtg7\" (UID: \"1f058cca-8314-4ff2-aa1f-3169f7d770fa\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-2bab4c-56fcccb486-2wtg7"
Oct 30 07:06:44.992654 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:44.992624    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8dcq\" (UniqueName: \"kubernetes.io/projected/1f058cca-8314-4ff2-aa1f-3169f7d770fa-kube-api-access-r8dcq\") pod \"payload-processing-e2e-worker-w4-2bab4c-56fcccb486-2wtg7\" (UID: \"1f058cca-8314-4ff2-aa1f-3169f7d770fa\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-2bab4c-56fcccb486-2wtg7"
Oct 30 07:06:45.076362 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:45.076333    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-2bab4c-56fcccb486-2wtg7"
Oct 30 07:06:45.372836 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:45.372795    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-2bab4c-56fcccb486-2wtg7"]
Oct 30 07:06:45.373842 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:06:45.373816    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f058cca_8314_4ff2_aa1f_3169f7d770fa.slice/crio-4d5ef980043d360149ac5018defd22a061352eb92e72377da09e7a7b6da23855 WatchSource:0}: Error finding container 4d5ef980043d360149ac5018defd22a061352eb92e72377da09e7a7b6da23855: Status 404 returned error can't find the container with id 4d5ef980043d360149ac5018defd22a061352eb92e72377da09e7a7b6da23855
Oct 30 07:06:46.266217 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:46.266176    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-2bab4c-56fcccb486-2wtg7" event={"ID":"1f058cca-8314-4ff2-aa1f-3169f7d770fa","Type":"ContainerStarted","Data":"e04889e3495adfb22152b033b96fa006a7ba5ad401b5bafdfa44d7730f33f9ee"}
Oct 30 07:06:46.266659 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:46.266240    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-2bab4c-56fcccb486-2wtg7" event={"ID":"1f058cca-8314-4ff2-aa1f-3169f7d770fa","Type":"ContainerStarted","Data":"4d5ef980043d360149ac5018defd22a061352eb92e72377da09e7a7b6da23855"}
Oct 30 07:06:46.266659 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:46.266267    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-2bab4c-56fcccb486-2wtg7"
Oct 30 07:06:46.315270 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:46.315221    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-2bab4c-56fcccb486-2wtg7" podStartSLOduration=2.315205065 podStartE2EDuration="2.315205065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:06:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:06:46.314992874 +0000 UTC m=+564.924399844" watchObservedRunningTime="2026-10-30 07:06:46.315205065 +0000 UTC m=+564.924612036"
Oct 30 07:06:47.341143 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:47.341098    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:06:51.097423 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:51.097384    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:06:54.395915 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:54.395883    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-4fe2ed-676bcd8cf6-pw5b5"]
Oct 30 07:06:54.399581 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:54.399563    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4fe2ed-676bcd8cf6-pw5b5"
Oct 30 07:06:54.431505 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:54.431472    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-4fe2ed-serving-cert\""
Oct 30 07:06:54.434198 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:54.434170    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-4fe2ed-676bcd8cf6-pw5b5"]
Oct 30 07:06:54.451968 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:54.451936    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4d66eef-bccf-4b9f-8c10-d4ac990c531f-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-4fe2ed-676bcd8cf6-pw5b5\" (UID: \"a4d66eef-bccf-4b9f-8c10-d4ac990c531f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4fe2ed-676bcd8cf6-pw5b5"
Oct 30 07:06:54.452102 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:54.452039    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4d66eef-bccf-4b9f-8c10-d4ac990c531f-metrics-certs\") pod \"maas-api-e2e-worker-w1-4fe2ed-676bcd8cf6-pw5b5\" (UID: \"a4d66eef-bccf-4b9f-8c10-d4ac990c531f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4fe2ed-676bcd8cf6-pw5b5"
Oct 30 07:06:54.452102 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:54.452077    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc2nc\" (UniqueName: \"kubernetes.io/projected/a4d66eef-bccf-4b9f-8c10-d4ac990c531f-kube-api-access-zc2nc\") pod \"maas-api-e2e-worker-w1-4fe2ed-676bcd8cf6-pw5b5\" (UID: \"a4d66eef-bccf-4b9f-8c10-d4ac990c531f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4fe2ed-676bcd8cf6-pw5b5"
Oct 30 07:06:54.452214 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:54.452193    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4d66eef-bccf-4b9f-8c10-d4ac990c531f-maas-api-tls\") pod \"maas-api-e2e-worker-w1-4fe2ed-676bcd8cf6-pw5b5\" (UID: \"a4d66eef-bccf-4b9f-8c10-d4ac990c531f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4fe2ed-676bcd8cf6-pw5b5"
Oct 30 07:06:54.553730 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:54.553696    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4d66eef-bccf-4b9f-8c10-d4ac990c531f-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-4fe2ed-676bcd8cf6-pw5b5\" (UID: \"a4d66eef-bccf-4b9f-8c10-d4ac990c531f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4fe2ed-676bcd8cf6-pw5b5"
Oct 30 07:06:54.553910 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:54.553746    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4d66eef-bccf-4b9f-8c10-d4ac990c531f-metrics-certs\") pod \"maas-api-e2e-worker-w1-4fe2ed-676bcd8cf6-pw5b5\" (UID: \"a4d66eef-bccf-4b9f-8c10-d4ac990c531f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4fe2ed-676bcd8cf6-pw5b5"
Oct 30 07:06:54.553910 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:54.553773    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc2nc\" (UniqueName: \"kubernetes.io/projected/a4d66eef-bccf-4b9f-8c10-d4ac990c531f-kube-api-access-zc2nc\") pod \"maas-api-e2e-worker-w1-4fe2ed-676bcd8cf6-pw5b5\" (UID: \"a4d66eef-bccf-4b9f-8c10-d4ac990c531f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4fe2ed-676bcd8cf6-pw5b5"
Oct 30 07:06:54.554036 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:54.553991    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4d66eef-bccf-4b9f-8c10-d4ac990c531f-maas-api-tls\") pod \"maas-api-e2e-worker-w1-4fe2ed-676bcd8cf6-pw5b5\" (UID: \"a4d66eef-bccf-4b9f-8c10-d4ac990c531f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4fe2ed-676bcd8cf6-pw5b5"
Oct 30 07:06:54.554625 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:54.554602    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4d66eef-bccf-4b9f-8c10-d4ac990c531f-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-4fe2ed-676bcd8cf6-pw5b5\" (UID: \"a4d66eef-bccf-4b9f-8c10-d4ac990c531f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4fe2ed-676bcd8cf6-pw5b5"
Oct 30 07:06:54.556307 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:54.556279    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4d66eef-bccf-4b9f-8c10-d4ac990c531f-maas-api-tls\") pod \"maas-api-e2e-worker-w1-4fe2ed-676bcd8cf6-pw5b5\" (UID: \"a4d66eef-bccf-4b9f-8c10-d4ac990c531f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4fe2ed-676bcd8cf6-pw5b5"
Oct 30 07:06:54.556395 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:54.556354    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4d66eef-bccf-4b9f-8c10-d4ac990c531f-metrics-certs\") pod \"maas-api-e2e-worker-w1-4fe2ed-676bcd8cf6-pw5b5\" (UID: \"a4d66eef-bccf-4b9f-8c10-d4ac990c531f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4fe2ed-676bcd8cf6-pw5b5"
Oct 30 07:06:54.579847 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:54.579820    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc2nc\" (UniqueName: \"kubernetes.io/projected/a4d66eef-bccf-4b9f-8c10-d4ac990c531f-kube-api-access-zc2nc\") pod \"maas-api-e2e-worker-w1-4fe2ed-676bcd8cf6-pw5b5\" (UID: \"a4d66eef-bccf-4b9f-8c10-d4ac990c531f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4fe2ed-676bcd8cf6-pw5b5"
Oct 30 07:06:54.712019 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:54.711944    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4fe2ed-676bcd8cf6-pw5b5"
Oct 30 07:06:55.012194 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:55.012164    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-4fe2ed-676bcd8cf6-pw5b5"]
Oct 30 07:06:55.013286 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:06:55.013249    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4d66eef_bccf_4b9f_8c10_d4ac990c531f.slice/crio-c48491b4e3d26d5125336e56fa46997117daf44570ebebc45f4cc68f8181e960 WatchSource:0}: Error finding container c48491b4e3d26d5125336e56fa46997117daf44570ebebc45f4cc68f8181e960: Status 404 returned error can't find the container with id c48491b4e3d26d5125336e56fa46997117daf44570ebebc45f4cc68f8181e960
Oct 30 07:06:55.313055 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:55.313017    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4fe2ed-676bcd8cf6-pw5b5" event={"ID":"a4d66eef-bccf-4b9f-8c10-d4ac990c531f","Type":"ContainerStarted","Data":"c48491b4e3d26d5125336e56fa46997117daf44570ebebc45f4cc68f8181e960"}
Oct 30 07:06:55.443988 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:55.443951    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-2eda2c/simulator-w3-2eda2c-kserve-56776b9849-9bqzm"]
Oct 30 07:06:55.448612 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:55.448592    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2eda2c/simulator-w3-2eda2c-kserve-56776b9849-9bqzm"
Oct 30 07:06:55.467189 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:55.467162    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2eda2c/simulator-w3-2eda2c-kserve-56776b9849-9bqzm"]
Oct 30 07:06:55.478831 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:55.478647    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-2eda2c\"/\"kube-root-ca.crt\""
Oct 30 07:06:55.478831 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:55.478817    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-2eda2c\"/\"openshift-service-ca.crt\""
Oct 30 07:06:55.481677 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:55.481656    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-2eda2c\"/\"default-dockercfg-x9pcs\""
Oct 30 07:06:55.481903 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:55.481880    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-2eda2c\"/\"simulator-w3-2eda2c-kserve-self-signed-certs\""
Oct 30 07:06:55.563032 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:55.562991    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/482e6b7c-2231-423c-b54f-a54e56f16ab7-tmp-dir\") pod \"simulator-w3-2eda2c-kserve-56776b9849-9bqzm\" (UID: \"482e6b7c-2231-423c-b54f-a54e56f16ab7\") " pod="e2e-models-e2e-worker-w3-2eda2c/simulator-w3-2eda2c-kserve-56776b9849-9bqzm"
Oct 30 07:06:55.563204 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:55.563044    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99qq6\" (UniqueName: \"kubernetes.io/projected/482e6b7c-2231-423c-b54f-a54e56f16ab7-kube-api-access-99qq6\") pod \"simulator-w3-2eda2c-kserve-56776b9849-9bqzm\" (UID: \"482e6b7c-2231-423c-b54f-a54e56f16ab7\") " pod="e2e-models-e2e-worker-w3-2eda2c/simulator-w3-2eda2c-kserve-56776b9849-9bqzm"
Oct 30 07:06:55.563204 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:55.563159    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/482e6b7c-2231-423c-b54f-a54e56f16ab7-dshm\") pod \"simulator-w3-2eda2c-kserve-56776b9849-9bqzm\" (UID: \"482e6b7c-2231-423c-b54f-a54e56f16ab7\") " pod="e2e-models-e2e-worker-w3-2eda2c/simulator-w3-2eda2c-kserve-56776b9849-9bqzm"
Oct 30 07:06:55.563327 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:55.563210    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/482e6b7c-2231-423c-b54f-a54e56f16ab7-model-cache\") pod \"simulator-w3-2eda2c-kserve-56776b9849-9bqzm\" (UID: \"482e6b7c-2231-423c-b54f-a54e56f16ab7\") " pod="e2e-models-e2e-worker-w3-2eda2c/simulator-w3-2eda2c-kserve-56776b9849-9bqzm"
Oct 30 07:06:55.563327 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:55.563234    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/482e6b7c-2231-423c-b54f-a54e56f16ab7-tls-certs\") pod \"simulator-w3-2eda2c-kserve-56776b9849-9bqzm\" (UID: \"482e6b7c-2231-423c-b54f-a54e56f16ab7\") " pod="e2e-models-e2e-worker-w3-2eda2c/simulator-w3-2eda2c-kserve-56776b9849-9bqzm"
Oct 30 07:06:55.563487 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:55.563452    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/482e6b7c-2231-423c-b54f-a54e56f16ab7-home\") pod \"simulator-w3-2eda2c-kserve-56776b9849-9bqzm\" (UID: \"482e6b7c-2231-423c-b54f-a54e56f16ab7\") " pod="e2e-models-e2e-worker-w3-2eda2c/simulator-w3-2eda2c-kserve-56776b9849-9bqzm"
Oct 30 07:06:55.663955 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:55.663922    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/482e6b7c-2231-423c-b54f-a54e56f16ab7-home\") pod \"simulator-w3-2eda2c-kserve-56776b9849-9bqzm\" (UID: \"482e6b7c-2231-423c-b54f-a54e56f16ab7\") " pod="e2e-models-e2e-worker-w3-2eda2c/simulator-w3-2eda2c-kserve-56776b9849-9bqzm"
Oct 30 07:06:55.664142 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:55.663969    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/482e6b7c-2231-423c-b54f-a54e56f16ab7-tmp-dir\") pod \"simulator-w3-2eda2c-kserve-56776b9849-9bqzm\" (UID: \"482e6b7c-2231-423c-b54f-a54e56f16ab7\") " pod="e2e-models-e2e-worker-w3-2eda2c/simulator-w3-2eda2c-kserve-56776b9849-9bqzm"
Oct 30 07:06:55.664142 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:55.664100    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99qq6\" (UniqueName: \"kubernetes.io/projected/482e6b7c-2231-423c-b54f-a54e56f16ab7-kube-api-access-99qq6\") pod \"simulator-w3-2eda2c-kserve-56776b9849-9bqzm\" (UID: \"482e6b7c-2231-423c-b54f-a54e56f16ab7\") " pod="e2e-models-e2e-worker-w3-2eda2c/simulator-w3-2eda2c-kserve-56776b9849-9bqzm"
Oct 30 07:06:55.664260 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:55.664205    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/482e6b7c-2231-423c-b54f-a54e56f16ab7-dshm\") pod \"simulator-w3-2eda2c-kserve-56776b9849-9bqzm\" (UID: \"482e6b7c-2231-423c-b54f-a54e56f16ab7\") " pod="e2e-models-e2e-worker-w3-2eda2c/simulator-w3-2eda2c-kserve-56776b9849-9bqzm"
Oct 30 07:06:55.664320 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:55.664265    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/482e6b7c-2231-423c-b54f-a54e56f16ab7-model-cache\") pod \"simulator-w3-2eda2c-kserve-56776b9849-9bqzm\" (UID: \"482e6b7c-2231-423c-b54f-a54e56f16ab7\") " pod="e2e-models-e2e-worker-w3-2eda2c/simulator-w3-2eda2c-kserve-56776b9849-9bqzm"
Oct 30 07:06:55.664320 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:55.664292    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/482e6b7c-2231-423c-b54f-a54e56f16ab7-tls-certs\") pod \"simulator-w3-2eda2c-kserve-56776b9849-9bqzm\" (UID: \"482e6b7c-2231-423c-b54f-a54e56f16ab7\") " pod="e2e-models-e2e-worker-w3-2eda2c/simulator-w3-2eda2c-kserve-56776b9849-9bqzm"
Oct 30 07:06:55.664412 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:55.664341    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/482e6b7c-2231-423c-b54f-a54e56f16ab7-home\") pod \"simulator-w3-2eda2c-kserve-56776b9849-9bqzm\" (UID: \"482e6b7c-2231-423c-b54f-a54e56f16ab7\") " pod="e2e-models-e2e-worker-w3-2eda2c/simulator-w3-2eda2c-kserve-56776b9849-9bqzm"
Oct 30 07:06:55.664412 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:55.664358    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/482e6b7c-2231-423c-b54f-a54e56f16ab7-tmp-dir\") pod \"simulator-w3-2eda2c-kserve-56776b9849-9bqzm\" (UID: \"482e6b7c-2231-423c-b54f-a54e56f16ab7\") " pod="e2e-models-e2e-worker-w3-2eda2c/simulator-w3-2eda2c-kserve-56776b9849-9bqzm"
Oct 30 07:06:55.664604 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:55.664583    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/482e6b7c-2231-423c-b54f-a54e56f16ab7-model-cache\") pod \"simulator-w3-2eda2c-kserve-56776b9849-9bqzm\" (UID: \"482e6b7c-2231-423c-b54f-a54e56f16ab7\") " pod="e2e-models-e2e-worker-w3-2eda2c/simulator-w3-2eda2c-kserve-56776b9849-9bqzm"
Oct 30 07:06:55.666274 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:55.666255    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/482e6b7c-2231-423c-b54f-a54e56f16ab7-dshm\") pod \"simulator-w3-2eda2c-kserve-56776b9849-9bqzm\" (UID: \"482e6b7c-2231-423c-b54f-a54e56f16ab7\") " pod="e2e-models-e2e-worker-w3-2eda2c/simulator-w3-2eda2c-kserve-56776b9849-9bqzm"
Oct 30 07:06:55.666696 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:55.666676    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/482e6b7c-2231-423c-b54f-a54e56f16ab7-tls-certs\") pod \"simulator-w3-2eda2c-kserve-56776b9849-9bqzm\" (UID: \"482e6b7c-2231-423c-b54f-a54e56f16ab7\") " pod="e2e-models-e2e-worker-w3-2eda2c/simulator-w3-2eda2c-kserve-56776b9849-9bqzm"
Oct 30 07:06:55.689270 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:55.689250    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99qq6\" (UniqueName: \"kubernetes.io/projected/482e6b7c-2231-423c-b54f-a54e56f16ab7-kube-api-access-99qq6\") pod \"simulator-w3-2eda2c-kserve-56776b9849-9bqzm\" (UID: \"482e6b7c-2231-423c-b54f-a54e56f16ab7\") " pod="e2e-models-e2e-worker-w3-2eda2c/simulator-w3-2eda2c-kserve-56776b9849-9bqzm"
Oct 30 07:06:55.762651 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:55.762617    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2eda2c/simulator-w3-2eda2c-kserve-56776b9849-9bqzm"
Oct 30 07:06:56.057762 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:56.057738    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2eda2c/simulator-w3-2eda2c-kserve-56776b9849-9bqzm"]
Oct 30 07:06:56.058577 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:06:56.058542    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod482e6b7c_2231_423c_b54f_a54e56f16ab7.slice/crio-c4da7976c80abf5b67e3a42e73c90b4fa01584653045c71d34c7fb91df65b426 WatchSource:0}: Error finding container c4da7976c80abf5b67e3a42e73c90b4fa01584653045c71d34c7fb91df65b426: Status 404 returned error can't find the container with id c4da7976c80abf5b67e3a42e73c90b4fa01584653045c71d34c7fb91df65b426
Oct 30 07:06:56.321709 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:56.321671    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4fe2ed-676bcd8cf6-pw5b5" event={"ID":"a4d66eef-bccf-4b9f-8c10-d4ac990c531f","Type":"ContainerStarted","Data":"7841af5f024f4c00960cd337b56179b015fffb0ea49405a0eba36bb07fa23523"}
Oct 30 07:06:56.321953 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:56.321903    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4fe2ed-676bcd8cf6-pw5b5"
Oct 30 07:06:56.323739 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:56.323709    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2eda2c/simulator-w3-2eda2c-kserve-56776b9849-9bqzm" event={"ID":"482e6b7c-2231-423c-b54f-a54e56f16ab7","Type":"ContainerStarted","Data":"c4da7976c80abf5b67e3a42e73c90b4fa01584653045c71d34c7fb91df65b426"}
Oct 30 07:06:56.369372 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:56.369324    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4fe2ed-676bcd8cf6-pw5b5" podStartSLOduration=1.916566716 podStartE2EDuration="2.369309891s" totalImagesPullingTime="452.743175ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:06:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:06:56.368385729 +0000 UTC m=+574.977792717" watchObservedRunningTime="2026-10-30 07:06:56.369309891 +0000 UTC m=+574.978716862"
Oct 30 07:06:57.274762 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:57.274731    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-2bab4c-56fcccb486-2wtg7"
Oct 30 07:06:57.332474 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:57.332367    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2eda2c/simulator-w3-2eda2c-kserve-56776b9849-9bqzm" event={"ID":"482e6b7c-2231-423c-b54f-a54e56f16ab7","Type":"ContainerStarted","Data":"32924d950abe45c0e04c671daea038e121ef91ddef69154d07678df7f821c1ca"}
Oct 30 07:06:57.379173 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:57.379123    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-2eda2c/simulator-w3-2eda2c-kserve-56776b9849-9bqzm" podStartSLOduration=1.3747357 podStartE2EDuration="2.379109718s" totalImagesPullingTime="1.004374018s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:06:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:06:57.377341712 +0000 UTC m=+575.986748681" watchObservedRunningTime="2026-10-30 07:06:57.379109718 +0000 UTC m=+575.988516688"
Oct 30 07:06:57.763406 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:57.763326    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2eda2c/simulator-w3-2eda2c-kserve-56776b9849-9bqzm"
Oct 30 07:06:59.196830 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:06:59.195858    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:07:01.864118 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:07:01.864073    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:07:02.338296 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:02.338269    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4fe2ed-676bcd8cf6-pw5b5"
Oct 30 07:07:05.763850 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:05.763796    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-2eda2c/simulator-w3-2eda2c-kserve-56776b9849-9bqzm"
Oct 30 07:07:05.775948 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:05.775923    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-2eda2c/simulator-w3-2eda2c-kserve-56776b9849-9bqzm"
Oct 30 07:07:06.030934 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:06.030908    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-bb6be4/simulator-w2-bb6be4-kserve-8485bbf954-dnbbz"]
Oct 30 07:07:06.034459 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:06.034439    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-bb6be4/simulator-w2-bb6be4-kserve-8485bbf954-dnbbz"
Oct 30 07:07:06.052675 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:06.052654    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-bb6be4\"/\"default-dockercfg-77htj\""
Oct 30 07:07:06.055022 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:06.054997    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-bb6be4/simulator-w2-bb6be4-kserve-8485bbf954-dnbbz"]
Oct 30 07:07:06.055608 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:06.055586    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-bb6be4\"/\"simulator-w2-bb6be4-kserve-self-signed-certs\""
Oct 30 07:07:06.055734 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:06.055706    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-bb6be4\"/\"openshift-service-ca.crt\""
Oct 30 07:07:06.055794 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:06.055735    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-bb6be4\"/\"kube-root-ca.crt\""
Oct 30 07:07:06.162022 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:06.161985    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6370f7e2-d6c8-4fb5-8198-32b38837ca31-tmp-dir\") pod \"simulator-w2-bb6be4-kserve-8485bbf954-dnbbz\" (UID: \"6370f7e2-d6c8-4fb5-8198-32b38837ca31\") " pod="e2e-models-e2e-worker-w2-bb6be4/simulator-w2-bb6be4-kserve-8485bbf954-dnbbz"
Oct 30 07:07:06.162204 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:06.162049    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6370f7e2-d6c8-4fb5-8198-32b38837ca31-model-cache\") pod \"simulator-w2-bb6be4-kserve-8485bbf954-dnbbz\" (UID: \"6370f7e2-d6c8-4fb5-8198-32b38837ca31\") " pod="e2e-models-e2e-worker-w2-bb6be4/simulator-w2-bb6be4-kserve-8485bbf954-dnbbz"
Oct 30 07:07:06.162204 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:06.162097    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntsfp\" (UniqueName: \"kubernetes.io/projected/6370f7e2-d6c8-4fb5-8198-32b38837ca31-kube-api-access-ntsfp\") pod \"simulator-w2-bb6be4-kserve-8485bbf954-dnbbz\" (UID: \"6370f7e2-d6c8-4fb5-8198-32b38837ca31\") " pod="e2e-models-e2e-worker-w2-bb6be4/simulator-w2-bb6be4-kserve-8485bbf954-dnbbz"
Oct 30 07:07:06.162204 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:06.162149    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6370f7e2-d6c8-4fb5-8198-32b38837ca31-dshm\") pod \"simulator-w2-bb6be4-kserve-8485bbf954-dnbbz\" (UID: \"6370f7e2-d6c8-4fb5-8198-32b38837ca31\") " pod="e2e-models-e2e-worker-w2-bb6be4/simulator-w2-bb6be4-kserve-8485bbf954-dnbbz"
Oct 30 07:07:06.162375 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:06.162262    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6370f7e2-d6c8-4fb5-8198-32b38837ca31-home\") pod \"simulator-w2-bb6be4-kserve-8485bbf954-dnbbz\" (UID: \"6370f7e2-d6c8-4fb5-8198-32b38837ca31\") " pod="e2e-models-e2e-worker-w2-bb6be4/simulator-w2-bb6be4-kserve-8485bbf954-dnbbz"
Oct 30 07:07:06.162375 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:06.162292    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6370f7e2-d6c8-4fb5-8198-32b38837ca31-tls-certs\") pod \"simulator-w2-bb6be4-kserve-8485bbf954-dnbbz\" (UID: \"6370f7e2-d6c8-4fb5-8198-32b38837ca31\") " pod="e2e-models-e2e-worker-w2-bb6be4/simulator-w2-bb6be4-kserve-8485bbf954-dnbbz"
Oct 30 07:07:06.263741 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:06.263707    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6370f7e2-d6c8-4fb5-8198-32b38837ca31-dshm\") pod \"simulator-w2-bb6be4-kserve-8485bbf954-dnbbz\" (UID: \"6370f7e2-d6c8-4fb5-8198-32b38837ca31\") " pod="e2e-models-e2e-worker-w2-bb6be4/simulator-w2-bb6be4-kserve-8485bbf954-dnbbz"
Oct 30 07:07:06.263955 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:06.263778    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6370f7e2-d6c8-4fb5-8198-32b38837ca31-home\") pod \"simulator-w2-bb6be4-kserve-8485bbf954-dnbbz\" (UID: \"6370f7e2-d6c8-4fb5-8198-32b38837ca31\") " pod="e2e-models-e2e-worker-w2-bb6be4/simulator-w2-bb6be4-kserve-8485bbf954-dnbbz"
Oct 30 07:07:06.263955 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:06.263800    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6370f7e2-d6c8-4fb5-8198-32b38837ca31-tls-certs\") pod \"simulator-w2-bb6be4-kserve-8485bbf954-dnbbz\" (UID: \"6370f7e2-d6c8-4fb5-8198-32b38837ca31\") " pod="e2e-models-e2e-worker-w2-bb6be4/simulator-w2-bb6be4-kserve-8485bbf954-dnbbz"
Oct 30 07:07:06.263955 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:06.263896    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6370f7e2-d6c8-4fb5-8198-32b38837ca31-tmp-dir\") pod \"simulator-w2-bb6be4-kserve-8485bbf954-dnbbz\" (UID: \"6370f7e2-d6c8-4fb5-8198-32b38837ca31\") " pod="e2e-models-e2e-worker-w2-bb6be4/simulator-w2-bb6be4-kserve-8485bbf954-dnbbz"
Oct 30 07:07:06.263955 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:06.263953    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6370f7e2-d6c8-4fb5-8198-32b38837ca31-model-cache\") pod \"simulator-w2-bb6be4-kserve-8485bbf954-dnbbz\" (UID: \"6370f7e2-d6c8-4fb5-8198-32b38837ca31\") " pod="e2e-models-e2e-worker-w2-bb6be4/simulator-w2-bb6be4-kserve-8485bbf954-dnbbz"
Oct 30 07:07:06.264172 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:06.263977    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntsfp\" (UniqueName: \"kubernetes.io/projected/6370f7e2-d6c8-4fb5-8198-32b38837ca31-kube-api-access-ntsfp\") pod \"simulator-w2-bb6be4-kserve-8485bbf954-dnbbz\" (UID: \"6370f7e2-d6c8-4fb5-8198-32b38837ca31\") " pod="e2e-models-e2e-worker-w2-bb6be4/simulator-w2-bb6be4-kserve-8485bbf954-dnbbz"
Oct 30 07:07:06.264249 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:06.264227    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6370f7e2-d6c8-4fb5-8198-32b38837ca31-home\") pod \"simulator-w2-bb6be4-kserve-8485bbf954-dnbbz\" (UID: \"6370f7e2-d6c8-4fb5-8198-32b38837ca31\") " pod="e2e-models-e2e-worker-w2-bb6be4/simulator-w2-bb6be4-kserve-8485bbf954-dnbbz"
Oct 30 07:07:06.264308 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:06.264279    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6370f7e2-d6c8-4fb5-8198-32b38837ca31-tmp-dir\") pod \"simulator-w2-bb6be4-kserve-8485bbf954-dnbbz\" (UID: \"6370f7e2-d6c8-4fb5-8198-32b38837ca31\") " pod="e2e-models-e2e-worker-w2-bb6be4/simulator-w2-bb6be4-kserve-8485bbf954-dnbbz"
Oct 30 07:07:06.264375 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:06.264351    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6370f7e2-d6c8-4fb5-8198-32b38837ca31-model-cache\") pod \"simulator-w2-bb6be4-kserve-8485bbf954-dnbbz\" (UID: \"6370f7e2-d6c8-4fb5-8198-32b38837ca31\") " pod="e2e-models-e2e-worker-w2-bb6be4/simulator-w2-bb6be4-kserve-8485bbf954-dnbbz"
Oct 30 07:07:06.266047 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:06.266019    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6370f7e2-d6c8-4fb5-8198-32b38837ca31-dshm\") pod \"simulator-w2-bb6be4-kserve-8485bbf954-dnbbz\" (UID: \"6370f7e2-d6c8-4fb5-8198-32b38837ca31\") " pod="e2e-models-e2e-worker-w2-bb6be4/simulator-w2-bb6be4-kserve-8485bbf954-dnbbz"
Oct 30 07:07:06.266421 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:06.266401    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6370f7e2-d6c8-4fb5-8198-32b38837ca31-tls-certs\") pod \"simulator-w2-bb6be4-kserve-8485bbf954-dnbbz\" (UID: \"6370f7e2-d6c8-4fb5-8198-32b38837ca31\") " pod="e2e-models-e2e-worker-w2-bb6be4/simulator-w2-bb6be4-kserve-8485bbf954-dnbbz"
Oct 30 07:07:06.289535 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:06.289478    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntsfp\" (UniqueName: \"kubernetes.io/projected/6370f7e2-d6c8-4fb5-8198-32b38837ca31-kube-api-access-ntsfp\") pod \"simulator-w2-bb6be4-kserve-8485bbf954-dnbbz\" (UID: \"6370f7e2-d6c8-4fb5-8198-32b38837ca31\") " pod="e2e-models-e2e-worker-w2-bb6be4/simulator-w2-bb6be4-kserve-8485bbf954-dnbbz"
Oct 30 07:07:06.349010 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:06.348977    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-bb6be4/simulator-w2-bb6be4-kserve-8485bbf954-dnbbz"
Oct 30 07:07:06.366360 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:06.366332    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:07:06.391016 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:06.390989    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2eda2c/simulator-w3-2eda2c-kserve-56776b9849-9bqzm"
Oct 30 07:07:06.656228 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:06.656197    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-bb6be4/simulator-w2-bb6be4-kserve-8485bbf954-dnbbz"]
Oct 30 07:07:06.656630 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:07:06.656595    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6370f7e2_d6c8_4fb5_8198_32b38837ca31.slice/crio-237cfe39a1e08c3eae36a1a820c12a064674552254f07947b4e3d329b1fafe37 WatchSource:0}: Error finding container 237cfe39a1e08c3eae36a1a820c12a064674552254f07947b4e3d329b1fafe37: Status 404 returned error can't find the container with id 237cfe39a1e08c3eae36a1a820c12a064674552254f07947b4e3d329b1fafe37
Oct 30 07:07:07.385749 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:07.385710    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-bb6be4/simulator-w2-bb6be4-kserve-8485bbf954-dnbbz" event={"ID":"6370f7e2-d6c8-4fb5-8198-32b38837ca31","Type":"ContainerStarted","Data":"811a8b06e535ff298712ff1b196869cd36756bd34d8241bafff1c8c07dc21ed9"}
Oct 30 07:07:07.386151 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:07.385758    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-bb6be4/simulator-w2-bb6be4-kserve-8485bbf954-dnbbz" event={"ID":"6370f7e2-d6c8-4fb5-8198-32b38837ca31","Type":"ContainerStarted","Data":"237cfe39a1e08c3eae36a1a820c12a064674552254f07947b4e3d329b1fafe37"}
Oct 30 07:07:07.435739 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:07.435683    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-bb6be4/simulator-w2-bb6be4-kserve-8485bbf954-dnbbz" podStartSLOduration=1.435669441 podStartE2EDuration="1.435669441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:07:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:07:07.434827521 +0000 UTC m=+586.044234482" watchObservedRunningTime="2026-10-30 07:07:07.435669441 +0000 UTC m=+586.045076411"
Oct 30 07:07:08.349764 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:08.349729    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb6be4/simulator-w2-bb6be4-kserve-8485bbf954-dnbbz"
Oct 30 07:07:12.817726 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:12.817684    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-2bab4c/simulator-w4-2bab4c-kserve-6db9444549-7nrk8"]
Oct 30 07:07:12.821562 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:12.821537    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2bab4c/simulator-w4-2bab4c-kserve-6db9444549-7nrk8"
Oct 30 07:07:12.852218 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:12.852188    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-2bab4c\"/\"default-dockercfg-tccm9\""
Oct 30 07:07:12.852218 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:12.852215    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-2bab4c\"/\"simulator-w4-2bab4c-kserve-self-signed-certs\""
Oct 30 07:07:12.853702 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:12.853684    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-2bab4c\"/\"openshift-service-ca.crt\""
Oct 30 07:07:12.853845 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:12.853744    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-2bab4c\"/\"kube-root-ca.crt\""
Oct 30 07:07:12.854144 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:12.854124    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-2bab4c/simulator-w4-2bab4c-kserve-6db9444549-7nrk8"]
Oct 30 07:07:12.928569 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:12.928532    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7b09178-9ac7-492f-890b-5ff3da130981-tls-certs\") pod \"simulator-w4-2bab4c-kserve-6db9444549-7nrk8\" (UID: \"c7b09178-9ac7-492f-890b-5ff3da130981\") " pod="e2e-models-e2e-worker-w4-2bab4c/simulator-w4-2bab4c-kserve-6db9444549-7nrk8"
Oct 30 07:07:12.928759 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:12.928579    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7b09178-9ac7-492f-890b-5ff3da130981-dshm\") pod \"simulator-w4-2bab4c-kserve-6db9444549-7nrk8\" (UID: \"c7b09178-9ac7-492f-890b-5ff3da130981\") " pod="e2e-models-e2e-worker-w4-2bab4c/simulator-w4-2bab4c-kserve-6db9444549-7nrk8"
Oct 30 07:07:12.928759 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:12.928651    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsxdh\" (UniqueName: \"kubernetes.io/projected/c7b09178-9ac7-492f-890b-5ff3da130981-kube-api-access-gsxdh\") pod \"simulator-w4-2bab4c-kserve-6db9444549-7nrk8\" (UID: \"c7b09178-9ac7-492f-890b-5ff3da130981\") " pod="e2e-models-e2e-worker-w4-2bab4c/simulator-w4-2bab4c-kserve-6db9444549-7nrk8"
Oct 30 07:07:12.928759 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:12.928746    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7b09178-9ac7-492f-890b-5ff3da130981-home\") pod \"simulator-w4-2bab4c-kserve-6db9444549-7nrk8\" (UID: \"c7b09178-9ac7-492f-890b-5ff3da130981\") " pod="e2e-models-e2e-worker-w4-2bab4c/simulator-w4-2bab4c-kserve-6db9444549-7nrk8"
Oct 30 07:07:12.928915 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:12.928816    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7b09178-9ac7-492f-890b-5ff3da130981-model-cache\") pod \"simulator-w4-2bab4c-kserve-6db9444549-7nrk8\" (UID: \"c7b09178-9ac7-492f-890b-5ff3da130981\") " pod="e2e-models-e2e-worker-w4-2bab4c/simulator-w4-2bab4c-kserve-6db9444549-7nrk8"
Oct 30 07:07:12.928956 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:12.928921    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7b09178-9ac7-492f-890b-5ff3da130981-tmp-dir\") pod \"simulator-w4-2bab4c-kserve-6db9444549-7nrk8\" (UID: \"c7b09178-9ac7-492f-890b-5ff3da130981\") " pod="e2e-models-e2e-worker-w4-2bab4c/simulator-w4-2bab4c-kserve-6db9444549-7nrk8"
Oct 30 07:07:13.029358 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:13.029325    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsxdh\" (UniqueName: \"kubernetes.io/projected/c7b09178-9ac7-492f-890b-5ff3da130981-kube-api-access-gsxdh\") pod \"simulator-w4-2bab4c-kserve-6db9444549-7nrk8\" (UID: \"c7b09178-9ac7-492f-890b-5ff3da130981\") " pod="e2e-models-e2e-worker-w4-2bab4c/simulator-w4-2bab4c-kserve-6db9444549-7nrk8"
Oct 30 07:07:13.029619 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:13.029377    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7b09178-9ac7-492f-890b-5ff3da130981-home\") pod \"simulator-w4-2bab4c-kserve-6db9444549-7nrk8\" (UID: \"c7b09178-9ac7-492f-890b-5ff3da130981\") " pod="e2e-models-e2e-worker-w4-2bab4c/simulator-w4-2bab4c-kserve-6db9444549-7nrk8"
Oct 30 07:07:13.029619 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:13.029422    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7b09178-9ac7-492f-890b-5ff3da130981-model-cache\") pod \"simulator-w4-2bab4c-kserve-6db9444549-7nrk8\" (UID: \"c7b09178-9ac7-492f-890b-5ff3da130981\") " pod="e2e-models-e2e-worker-w4-2bab4c/simulator-w4-2bab4c-kserve-6db9444549-7nrk8"
Oct 30 07:07:13.029619 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:13.029459    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7b09178-9ac7-492f-890b-5ff3da130981-tmp-dir\") pod \"simulator-w4-2bab4c-kserve-6db9444549-7nrk8\" (UID: \"c7b09178-9ac7-492f-890b-5ff3da130981\") " pod="e2e-models-e2e-worker-w4-2bab4c/simulator-w4-2bab4c-kserve-6db9444549-7nrk8"
Oct 30 07:07:13.029619 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:13.029487    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7b09178-9ac7-492f-890b-5ff3da130981-tls-certs\") pod \"simulator-w4-2bab4c-kserve-6db9444549-7nrk8\" (UID: \"c7b09178-9ac7-492f-890b-5ff3da130981\") " pod="e2e-models-e2e-worker-w4-2bab4c/simulator-w4-2bab4c-kserve-6db9444549-7nrk8"
Oct 30 07:07:13.029619 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:13.029517    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7b09178-9ac7-492f-890b-5ff3da130981-dshm\") pod \"simulator-w4-2bab4c-kserve-6db9444549-7nrk8\" (UID: \"c7b09178-9ac7-492f-890b-5ff3da130981\") " pod="e2e-models-e2e-worker-w4-2bab4c/simulator-w4-2bab4c-kserve-6db9444549-7nrk8"
Oct 30 07:07:13.029927 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:13.029881    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7b09178-9ac7-492f-890b-5ff3da130981-home\") pod \"simulator-w4-2bab4c-kserve-6db9444549-7nrk8\" (UID: \"c7b09178-9ac7-492f-890b-5ff3da130981\") " pod="e2e-models-e2e-worker-w4-2bab4c/simulator-w4-2bab4c-kserve-6db9444549-7nrk8"
Oct 30 07:07:13.029927 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:13.029917    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7b09178-9ac7-492f-890b-5ff3da130981-model-cache\") pod \"simulator-w4-2bab4c-kserve-6db9444549-7nrk8\" (UID: \"c7b09178-9ac7-492f-890b-5ff3da130981\") " pod="e2e-models-e2e-worker-w4-2bab4c/simulator-w4-2bab4c-kserve-6db9444549-7nrk8"
Oct 30 07:07:13.030005 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:13.029976    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7b09178-9ac7-492f-890b-5ff3da130981-tmp-dir\") pod \"simulator-w4-2bab4c-kserve-6db9444549-7nrk8\" (UID: \"c7b09178-9ac7-492f-890b-5ff3da130981\") " pod="e2e-models-e2e-worker-w4-2bab4c/simulator-w4-2bab4c-kserve-6db9444549-7nrk8"
Oct 30 07:07:13.031962 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:13.031938    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7b09178-9ac7-492f-890b-5ff3da130981-dshm\") pod \"simulator-w4-2bab4c-kserve-6db9444549-7nrk8\" (UID: \"c7b09178-9ac7-492f-890b-5ff3da130981\") " pod="e2e-models-e2e-worker-w4-2bab4c/simulator-w4-2bab4c-kserve-6db9444549-7nrk8"
Oct 30 07:07:13.032170 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:13.032150    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7b09178-9ac7-492f-890b-5ff3da130981-tls-certs\") pod \"simulator-w4-2bab4c-kserve-6db9444549-7nrk8\" (UID: \"c7b09178-9ac7-492f-890b-5ff3da130981\") " pod="e2e-models-e2e-worker-w4-2bab4c/simulator-w4-2bab4c-kserve-6db9444549-7nrk8"
Oct 30 07:07:13.055656 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:13.055616    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsxdh\" (UniqueName: \"kubernetes.io/projected/c7b09178-9ac7-492f-890b-5ff3da130981-kube-api-access-gsxdh\") pod \"simulator-w4-2bab4c-kserve-6db9444549-7nrk8\" (UID: \"c7b09178-9ac7-492f-890b-5ff3da130981\") " pod="e2e-models-e2e-worker-w4-2bab4c/simulator-w4-2bab4c-kserve-6db9444549-7nrk8"
Oct 30 07:07:13.135125 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:13.135029    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2bab4c/simulator-w4-2bab4c-kserve-6db9444549-7nrk8"
Oct 30 07:07:13.442085 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:13.442054    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-2bab4c/simulator-w4-2bab4c-kserve-6db9444549-7nrk8"]
Oct 30 07:07:13.442935 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:07:13.442897    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc7b09178_9ac7_492f_890b_5ff3da130981.slice/crio-f33e7b1af29f8b5999a9768c230a15273afce33e54b33039439ca5ddeecdd3f5 WatchSource:0}: Error finding container f33e7b1af29f8b5999a9768c230a15273afce33e54b33039439ca5ddeecdd3f5: Status 404 returned error can't find the container with id f33e7b1af29f8b5999a9768c230a15273afce33e54b33039439ca5ddeecdd3f5
Oct 30 07:07:14.421676 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:14.421636    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2bab4c/simulator-w4-2bab4c-kserve-6db9444549-7nrk8" event={"ID":"c7b09178-9ac7-492f-890b-5ff3da130981","Type":"ContainerStarted","Data":"cf869217a4c504119fbe41e156c00e8d160a9434870ff7842b165aa5285dff99"}
Oct 30 07:07:14.422166 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:14.421692    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2bab4c/simulator-w4-2bab4c-kserve-6db9444549-7nrk8" event={"ID":"c7b09178-9ac7-492f-890b-5ff3da130981","Type":"ContainerStarted","Data":"f33e7b1af29f8b5999a9768c230a15273afce33e54b33039439ca5ddeecdd3f5"}
Oct 30 07:07:14.467306 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:14.467262    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-2bab4c/simulator-w4-2bab4c-kserve-6db9444549-7nrk8" podStartSLOduration=2.467249339 podStartE2EDuration="2.467249339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:07:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:07:14.466525673 +0000 UTC m=+593.075932644" watchObservedRunningTime="2026-10-30 07:07:14.467249339 +0000 UTC m=+593.076656309"
Oct 30 07:07:15.136174 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:15.136127    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2bab4c/simulator-w4-2bab4c-kserve-6db9444549-7nrk8"
Oct 30 07:07:16.350210 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:16.350171    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-bb6be4/simulator-w2-bb6be4-kserve-8485bbf954-dnbbz"
Oct 30 07:07:16.362697 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:16.362669    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-bb6be4/simulator-w2-bb6be4-kserve-8485bbf954-dnbbz"
Oct 30 07:07:16.429351 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:16.429320    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:07:16.443515 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:16.443490    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb6be4/simulator-w2-bb6be4-kserve-8485bbf954-dnbbz"
Oct 30 07:07:23.135863 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:23.135824    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-2bab4c/simulator-w4-2bab4c-kserve-6db9444549-7nrk8"
Oct 30 07:07:23.148104 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:23.148081    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-2bab4c/simulator-w4-2bab4c-kserve-6db9444549-7nrk8"
Oct 30 07:07:23.483888 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:23.483796    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2bab4c/simulator-w4-2bab4c-kserve-6db9444549-7nrk8"
Oct 30 07:07:27.981528 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:27.981491    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:07:31.855716 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:07:31.855680    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:07:37.024948 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:37.024913    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-bb6be4/premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t"]
Oct 30 07:07:37.028720 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:37.028699    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-bb6be4/premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t"
Oct 30 07:07:37.048722 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:37.048693    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-bb6be4/premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t"]
Oct 30 07:07:37.060706 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:37.060682    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-bb6be4\"/\"premium-w2-bb6be4-kserve-self-signed-certs\""
Oct 30 07:07:37.065037 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:37.065014    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8mtk\" (UniqueName: \"kubernetes.io/projected/7dd5133c-480f-4fa0-9dd1-27662004aae9-kube-api-access-m8mtk\") pod \"premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t\" (UID: \"7dd5133c-480f-4fa0-9dd1-27662004aae9\") " pod="e2e-models-e2e-worker-w2-bb6be4/premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t"
Oct 30 07:07:37.065135 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:37.065048    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7dd5133c-480f-4fa0-9dd1-27662004aae9-home\") pod \"premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t\" (UID: \"7dd5133c-480f-4fa0-9dd1-27662004aae9\") " pod="e2e-models-e2e-worker-w2-bb6be4/premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t"
Oct 30 07:07:37.065135 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:37.065068    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7dd5133c-480f-4fa0-9dd1-27662004aae9-tmp-dir\") pod \"premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t\" (UID: \"7dd5133c-480f-4fa0-9dd1-27662004aae9\") " pod="e2e-models-e2e-worker-w2-bb6be4/premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t"
Oct 30 07:07:37.065135 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:37.065086    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7dd5133c-480f-4fa0-9dd1-27662004aae9-model-cache\") pod \"premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t\" (UID: \"7dd5133c-480f-4fa0-9dd1-27662004aae9\") " pod="e2e-models-e2e-worker-w2-bb6be4/premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t"
Oct 30 07:07:37.065252 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:37.065213    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7dd5133c-480f-4fa0-9dd1-27662004aae9-dshm\") pod \"premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t\" (UID: \"7dd5133c-480f-4fa0-9dd1-27662004aae9\") " pod="e2e-models-e2e-worker-w2-bb6be4/premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t"
Oct 30 07:07:37.065291 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:37.065263    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7dd5133c-480f-4fa0-9dd1-27662004aae9-tls-certs\") pod \"premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t\" (UID: \"7dd5133c-480f-4fa0-9dd1-27662004aae9\") " pod="e2e-models-e2e-worker-w2-bb6be4/premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t"
Oct 30 07:07:37.166337 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:37.166298    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7dd5133c-480f-4fa0-9dd1-27662004aae9-model-cache\") pod \"premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t\" (UID: \"7dd5133c-480f-4fa0-9dd1-27662004aae9\") " pod="e2e-models-e2e-worker-w2-bb6be4/premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t"
Oct 30 07:07:37.166532 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:37.166393    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7dd5133c-480f-4fa0-9dd1-27662004aae9-dshm\") pod \"premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t\" (UID: \"7dd5133c-480f-4fa0-9dd1-27662004aae9\") " pod="e2e-models-e2e-worker-w2-bb6be4/premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t"
Oct 30 07:07:37.166603 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:37.166556    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7dd5133c-480f-4fa0-9dd1-27662004aae9-tls-certs\") pod \"premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t\" (UID: \"7dd5133c-480f-4fa0-9dd1-27662004aae9\") " pod="e2e-models-e2e-worker-w2-bb6be4/premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t"
Oct 30 07:07:37.166664 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:37.166639    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8mtk\" (UniqueName: \"kubernetes.io/projected/7dd5133c-480f-4fa0-9dd1-27662004aae9-kube-api-access-m8mtk\") pod \"premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t\" (UID: \"7dd5133c-480f-4fa0-9dd1-27662004aae9\") " pod="e2e-models-e2e-worker-w2-bb6be4/premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t"
Oct 30 07:07:37.166720 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:37.166678    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7dd5133c-480f-4fa0-9dd1-27662004aae9-home\") pod \"premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t\" (UID: \"7dd5133c-480f-4fa0-9dd1-27662004aae9\") " pod="e2e-models-e2e-worker-w2-bb6be4/premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t"
Oct 30 07:07:37.166720 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:37.166703    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7dd5133c-480f-4fa0-9dd1-27662004aae9-model-cache\") pod \"premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t\" (UID: \"7dd5133c-480f-4fa0-9dd1-27662004aae9\") " pod="e2e-models-e2e-worker-w2-bb6be4/premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t"
Oct 30 07:07:37.166720 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:37.166714    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7dd5133c-480f-4fa0-9dd1-27662004aae9-tmp-dir\") pod \"premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t\" (UID: \"7dd5133c-480f-4fa0-9dd1-27662004aae9\") " pod="e2e-models-e2e-worker-w2-bb6be4/premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t"
Oct 30 07:07:37.167027 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:37.167001    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7dd5133c-480f-4fa0-9dd1-27662004aae9-tmp-dir\") pod \"premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t\" (UID: \"7dd5133c-480f-4fa0-9dd1-27662004aae9\") " pod="e2e-models-e2e-worker-w2-bb6be4/premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t"
Oct 30 07:07:37.167119 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:37.167077    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7dd5133c-480f-4fa0-9dd1-27662004aae9-home\") pod \"premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t\" (UID: \"7dd5133c-480f-4fa0-9dd1-27662004aae9\") " pod="e2e-models-e2e-worker-w2-bb6be4/premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t"
Oct 30 07:07:37.168790 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:37.168770    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7dd5133c-480f-4fa0-9dd1-27662004aae9-dshm\") pod \"premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t\" (UID: \"7dd5133c-480f-4fa0-9dd1-27662004aae9\") " pod="e2e-models-e2e-worker-w2-bb6be4/premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t"
Oct 30 07:07:37.169206 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:37.169183    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7dd5133c-480f-4fa0-9dd1-27662004aae9-tls-certs\") pod \"premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t\" (UID: \"7dd5133c-480f-4fa0-9dd1-27662004aae9\") " pod="e2e-models-e2e-worker-w2-bb6be4/premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t"
Oct 30 07:07:37.192092 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:37.192065    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8mtk\" (UniqueName: \"kubernetes.io/projected/7dd5133c-480f-4fa0-9dd1-27662004aae9-kube-api-access-m8mtk\") pod \"premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t\" (UID: \"7dd5133c-480f-4fa0-9dd1-27662004aae9\") " pod="e2e-models-e2e-worker-w2-bb6be4/premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t"
Oct 30 07:07:37.340964 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:37.340930    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-bb6be4/premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t"
Oct 30 07:07:37.640169 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:37.640138    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-bb6be4/premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t"]
Oct 30 07:07:37.641511 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:07:37.641489    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7dd5133c_480f_4fa0_9dd1_27662004aae9.slice/crio-59a2aad79c02f433e063ec33611d880d51b04d87d06806c24f711e04265bd95c WatchSource:0}: Error finding container 59a2aad79c02f433e063ec33611d880d51b04d87d06806c24f711e04265bd95c: Status 404 returned error can't find the container with id 59a2aad79c02f433e063ec33611d880d51b04d87d06806c24f711e04265bd95c
Oct 30 07:07:38.549172 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:38.549133    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-bb6be4/premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t" event={"ID":"7dd5133c-480f-4fa0-9dd1-27662004aae9","Type":"ContainerStarted","Data":"a3887e8e179a28288c6197f63e0d5a3549dec18c36e3ff7675e25d2c6cc18830"}
Oct 30 07:07:38.549571 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:38.549180    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-bb6be4/premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t" event={"ID":"7dd5133c-480f-4fa0-9dd1-27662004aae9","Type":"ContainerStarted","Data":"59a2aad79c02f433e063ec33611d880d51b04d87d06806c24f711e04265bd95c"}
Oct 30 07:07:38.594895 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:38.594852    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-bb6be4/premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t" podStartSLOduration=2.594838202 podStartE2EDuration="2.594838202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:07:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:07:38.593767837 +0000 UTC m=+617.203174806" watchObservedRunningTime="2026-10-30 07:07:38.594838202 +0000 UTC m=+617.204245169"
Oct 30 07:07:39.341984 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:39.341950    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb6be4/premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t"
Oct 30 07:07:40.357136 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:40.357095    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:07:47.341875 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:47.341842    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-bb6be4/premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t"
Oct 30 07:07:47.354611 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:47.354587    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-bb6be4/premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t"
Oct 30 07:07:47.604852 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:47.604750    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb6be4/premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t"
Oct 30 07:07:50.616525 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:50.616492    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-2bab4c/premium-w4-2bab4c-kserve-665887fc68-c6dgf"]
Oct 30 07:07:50.620261 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:50.620241    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2bab4c/premium-w4-2bab4c-kserve-665887fc68-c6dgf"
Oct 30 07:07:50.638872 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:50.638853    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-2bab4c\"/\"premium-w4-2bab4c-kserve-self-signed-certs\""
Oct 30 07:07:50.656132 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:50.656103    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-2bab4c/premium-w4-2bab4c-kserve-665887fc68-c6dgf"]
Oct 30 07:07:50.694265 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:50.694241    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9463476-fcf5-4d66-bfd2-b85ce4ed651f-tls-certs\") pod \"premium-w4-2bab4c-kserve-665887fc68-c6dgf\" (UID: \"f9463476-fcf5-4d66-bfd2-b85ce4ed651f\") " pod="e2e-models-e2e-worker-w4-2bab4c/premium-w4-2bab4c-kserve-665887fc68-c6dgf"
Oct 30 07:07:50.694415 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:50.694285    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9463476-fcf5-4d66-bfd2-b85ce4ed651f-dshm\") pod \"premium-w4-2bab4c-kserve-665887fc68-c6dgf\" (UID: \"f9463476-fcf5-4d66-bfd2-b85ce4ed651f\") " pod="e2e-models-e2e-worker-w4-2bab4c/premium-w4-2bab4c-kserve-665887fc68-c6dgf"
Oct 30 07:07:50.694415 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:50.694327    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9463476-fcf5-4d66-bfd2-b85ce4ed651f-tmp-dir\") pod \"premium-w4-2bab4c-kserve-665887fc68-c6dgf\" (UID: \"f9463476-fcf5-4d66-bfd2-b85ce4ed651f\") " pod="e2e-models-e2e-worker-w4-2bab4c/premium-w4-2bab4c-kserve-665887fc68-c6dgf"
Oct 30 07:07:50.694508 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:50.694429    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9463476-fcf5-4d66-bfd2-b85ce4ed651f-home\") pod \"premium-w4-2bab4c-kserve-665887fc68-c6dgf\" (UID: \"f9463476-fcf5-4d66-bfd2-b85ce4ed651f\") " pod="e2e-models-e2e-worker-w4-2bab4c/premium-w4-2bab4c-kserve-665887fc68-c6dgf"
Oct 30 07:07:50.694508 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:50.694457    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2bg2\" (UniqueName: \"kubernetes.io/projected/f9463476-fcf5-4d66-bfd2-b85ce4ed651f-kube-api-access-j2bg2\") pod \"premium-w4-2bab4c-kserve-665887fc68-c6dgf\" (UID: \"f9463476-fcf5-4d66-bfd2-b85ce4ed651f\") " pod="e2e-models-e2e-worker-w4-2bab4c/premium-w4-2bab4c-kserve-665887fc68-c6dgf"
Oct 30 07:07:50.694582 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:50.694512    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9463476-fcf5-4d66-bfd2-b85ce4ed651f-model-cache\") pod \"premium-w4-2bab4c-kserve-665887fc68-c6dgf\" (UID: \"f9463476-fcf5-4d66-bfd2-b85ce4ed651f\") " pod="e2e-models-e2e-worker-w4-2bab4c/premium-w4-2bab4c-kserve-665887fc68-c6dgf"
Oct 30 07:07:50.796097 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:50.796062    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9463476-fcf5-4d66-bfd2-b85ce4ed651f-model-cache\") pod \"premium-w4-2bab4c-kserve-665887fc68-c6dgf\" (UID: \"f9463476-fcf5-4d66-bfd2-b85ce4ed651f\") " pod="e2e-models-e2e-worker-w4-2bab4c/premium-w4-2bab4c-kserve-665887fc68-c6dgf"
Oct 30 07:07:50.796306 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:50.796132    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9463476-fcf5-4d66-bfd2-b85ce4ed651f-tls-certs\") pod \"premium-w4-2bab4c-kserve-665887fc68-c6dgf\" (UID: \"f9463476-fcf5-4d66-bfd2-b85ce4ed651f\") " pod="e2e-models-e2e-worker-w4-2bab4c/premium-w4-2bab4c-kserve-665887fc68-c6dgf"
Oct 30 07:07:50.796306 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:50.796178    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9463476-fcf5-4d66-bfd2-b85ce4ed651f-dshm\") pod \"premium-w4-2bab4c-kserve-665887fc68-c6dgf\" (UID: \"f9463476-fcf5-4d66-bfd2-b85ce4ed651f\") " pod="e2e-models-e2e-worker-w4-2bab4c/premium-w4-2bab4c-kserve-665887fc68-c6dgf"
Oct 30 07:07:50.796306 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:50.796221    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9463476-fcf5-4d66-bfd2-b85ce4ed651f-tmp-dir\") pod \"premium-w4-2bab4c-kserve-665887fc68-c6dgf\" (UID: \"f9463476-fcf5-4d66-bfd2-b85ce4ed651f\") " pod="e2e-models-e2e-worker-w4-2bab4c/premium-w4-2bab4c-kserve-665887fc68-c6dgf"
Oct 30 07:07:50.796306 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:50.796267    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9463476-fcf5-4d66-bfd2-b85ce4ed651f-home\") pod \"premium-w4-2bab4c-kserve-665887fc68-c6dgf\" (UID: \"f9463476-fcf5-4d66-bfd2-b85ce4ed651f\") " pod="e2e-models-e2e-worker-w4-2bab4c/premium-w4-2bab4c-kserve-665887fc68-c6dgf"
Oct 30 07:07:50.796306 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:50.796301    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2bg2\" (UniqueName: \"kubernetes.io/projected/f9463476-fcf5-4d66-bfd2-b85ce4ed651f-kube-api-access-j2bg2\") pod \"premium-w4-2bab4c-kserve-665887fc68-c6dgf\" (UID: \"f9463476-fcf5-4d66-bfd2-b85ce4ed651f\") " pod="e2e-models-e2e-worker-w4-2bab4c/premium-w4-2bab4c-kserve-665887fc68-c6dgf"
Oct 30 07:07:50.796575 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:50.796544    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9463476-fcf5-4d66-bfd2-b85ce4ed651f-model-cache\") pod \"premium-w4-2bab4c-kserve-665887fc68-c6dgf\" (UID: \"f9463476-fcf5-4d66-bfd2-b85ce4ed651f\") " pod="e2e-models-e2e-worker-w4-2bab4c/premium-w4-2bab4c-kserve-665887fc68-c6dgf"
Oct 30 07:07:50.796947 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:50.796922    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9463476-fcf5-4d66-bfd2-b85ce4ed651f-tmp-dir\") pod \"premium-w4-2bab4c-kserve-665887fc68-c6dgf\" (UID: \"f9463476-fcf5-4d66-bfd2-b85ce4ed651f\") " pod="e2e-models-e2e-worker-w4-2bab4c/premium-w4-2bab4c-kserve-665887fc68-c6dgf"
Oct 30 07:07:50.797121 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:50.797094    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9463476-fcf5-4d66-bfd2-b85ce4ed651f-home\") pod \"premium-w4-2bab4c-kserve-665887fc68-c6dgf\" (UID: \"f9463476-fcf5-4d66-bfd2-b85ce4ed651f\") " pod="e2e-models-e2e-worker-w4-2bab4c/premium-w4-2bab4c-kserve-665887fc68-c6dgf"
Oct 30 07:07:50.799538 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:50.799506    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9463476-fcf5-4d66-bfd2-b85ce4ed651f-dshm\") pod \"premium-w4-2bab4c-kserve-665887fc68-c6dgf\" (UID: \"f9463476-fcf5-4d66-bfd2-b85ce4ed651f\") " pod="e2e-models-e2e-worker-w4-2bab4c/premium-w4-2bab4c-kserve-665887fc68-c6dgf"
Oct 30 07:07:50.800395 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:50.800374    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9463476-fcf5-4d66-bfd2-b85ce4ed651f-tls-certs\") pod \"premium-w4-2bab4c-kserve-665887fc68-c6dgf\" (UID: \"f9463476-fcf5-4d66-bfd2-b85ce4ed651f\") " pod="e2e-models-e2e-worker-w4-2bab4c/premium-w4-2bab4c-kserve-665887fc68-c6dgf"
Oct 30 07:07:50.822449 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:50.822419    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2bg2\" (UniqueName: \"kubernetes.io/projected/f9463476-fcf5-4d66-bfd2-b85ce4ed651f-kube-api-access-j2bg2\") pod \"premium-w4-2bab4c-kserve-665887fc68-c6dgf\" (UID: \"f9463476-fcf5-4d66-bfd2-b85ce4ed651f\") " pod="e2e-models-e2e-worker-w4-2bab4c/premium-w4-2bab4c-kserve-665887fc68-c6dgf"
Oct 30 07:07:50.932628 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:50.932552    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2bab4c/premium-w4-2bab4c-kserve-665887fc68-c6dgf"
Oct 30 07:07:51.232906 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:51.232873    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-2bab4c/premium-w4-2bab4c-kserve-665887fc68-c6dgf"]
Oct 30 07:07:51.233187 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:07:51.233160    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf9463476_fcf5_4d66_bfd2_b85ce4ed651f.slice/crio-b5655b958bde911e3d0417332d55188aa28181bcdd3eaaa914592156ddd92063 WatchSource:0}: Error finding container b5655b958bde911e3d0417332d55188aa28181bcdd3eaaa914592156ddd92063: Status 404 returned error can't find the container with id b5655b958bde911e3d0417332d55188aa28181bcdd3eaaa914592156ddd92063
Oct 30 07:07:51.617651 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:51.617613    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2bab4c/premium-w4-2bab4c-kserve-665887fc68-c6dgf" event={"ID":"f9463476-fcf5-4d66-bfd2-b85ce4ed651f","Type":"ContainerStarted","Data":"6cb9fbd119431132e53c8a595bdba5447d6cad23fc4ec2e667058fecbec124a5"}
Oct 30 07:07:51.617651 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:51.617656    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2bab4c/premium-w4-2bab4c-kserve-665887fc68-c6dgf" event={"ID":"f9463476-fcf5-4d66-bfd2-b85ce4ed651f","Type":"ContainerStarted","Data":"b5655b958bde911e3d0417332d55188aa28181bcdd3eaaa914592156ddd92063"}
Oct 30 07:07:51.663940 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:51.663883    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-2bab4c/premium-w4-2bab4c-kserve-665887fc68-c6dgf" podStartSLOduration=1.663869689 podStartE2EDuration="1.663869689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:07:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:07:51.662011401 +0000 UTC m=+630.271418373" watchObservedRunningTime="2026-10-30 07:07:51.663869689 +0000 UTC m=+630.273276684"
Oct 30 07:07:51.933034 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:51.932950    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2bab4c/premium-w4-2bab4c-kserve-665887fc68-c6dgf"
Oct 30 07:07:54.340133 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:07:54.340101    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:08:00.933448 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:00.933417    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-2bab4c/premium-w4-2bab4c-kserve-665887fc68-c6dgf"
Oct 30 07:08:00.946198 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:00.946173    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-2bab4c/premium-w4-2bab4c-kserve-665887fc68-c6dgf"
Oct 30 07:08:01.681255 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:01.681221    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2bab4c/premium-w4-2bab4c-kserve-665887fc68-c6dgf"
Oct 30 07:08:01.856920 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:08:01.856884    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:08:04.741824 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:04.741719    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-06b26e/premium-w0-06b26e-kserve-c9dbd9879-b4jlk"]
Oct 30 07:08:04.746667 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:04.746643    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-06b26e/premium-w0-06b26e-kserve-c9dbd9879-b4jlk"
Oct 30 07:08:04.777132 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:04.777108    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-06b26e\"/\"kube-root-ca.crt\""
Oct 30 07:08:04.777261 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:04.777132    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-06b26e\"/\"openshift-service-ca.crt\""
Oct 30 07:08:04.777261 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:04.777220    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-06b26e\"/\"premium-w0-06b26e-kserve-self-signed-certs\""
Oct 30 07:08:04.777650 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:04.777628    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-06b26e\"/\"default-dockercfg-spf2z\""
Oct 30 07:08:04.779374 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:04.779350    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-06b26e/premium-w0-06b26e-kserve-c9dbd9879-b4jlk"]
Oct 30 07:08:04.828613 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:04.828577    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1230dcae-e1cd-460d-86fc-bd40c8427b5f-home\") pod \"premium-w0-06b26e-kserve-c9dbd9879-b4jlk\" (UID: \"1230dcae-e1cd-460d-86fc-bd40c8427b5f\") " pod="e2e-models-e2e-worker-w0-06b26e/premium-w0-06b26e-kserve-c9dbd9879-b4jlk"
Oct 30 07:08:04.828760 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:04.828641    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1230dcae-e1cd-460d-86fc-bd40c8427b5f-tls-certs\") pod \"premium-w0-06b26e-kserve-c9dbd9879-b4jlk\" (UID: \"1230dcae-e1cd-460d-86fc-bd40c8427b5f\") " pod="e2e-models-e2e-worker-w0-06b26e/premium-w0-06b26e-kserve-c9dbd9879-b4jlk"
Oct 30 07:08:04.828760 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:04.828745    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1230dcae-e1cd-460d-86fc-bd40c8427b5f-tmp-dir\") pod \"premium-w0-06b26e-kserve-c9dbd9879-b4jlk\" (UID: \"1230dcae-e1cd-460d-86fc-bd40c8427b5f\") " pod="e2e-models-e2e-worker-w0-06b26e/premium-w0-06b26e-kserve-c9dbd9879-b4jlk"
Oct 30 07:08:04.828876 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:04.828772    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1230dcae-e1cd-460d-86fc-bd40c8427b5f-model-cache\") pod \"premium-w0-06b26e-kserve-c9dbd9879-b4jlk\" (UID: \"1230dcae-e1cd-460d-86fc-bd40c8427b5f\") " pod="e2e-models-e2e-worker-w0-06b26e/premium-w0-06b26e-kserve-c9dbd9879-b4jlk"
Oct 30 07:08:04.828876 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:04.828796    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szbqn\" (UniqueName: \"kubernetes.io/projected/1230dcae-e1cd-460d-86fc-bd40c8427b5f-kube-api-access-szbqn\") pod \"premium-w0-06b26e-kserve-c9dbd9879-b4jlk\" (UID: \"1230dcae-e1cd-460d-86fc-bd40c8427b5f\") " pod="e2e-models-e2e-worker-w0-06b26e/premium-w0-06b26e-kserve-c9dbd9879-b4jlk"
Oct 30 07:08:04.828959 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:04.828940    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1230dcae-e1cd-460d-86fc-bd40c8427b5f-dshm\") pod \"premium-w0-06b26e-kserve-c9dbd9879-b4jlk\" (UID: \"1230dcae-e1cd-460d-86fc-bd40c8427b5f\") " pod="e2e-models-e2e-worker-w0-06b26e/premium-w0-06b26e-kserve-c9dbd9879-b4jlk"
Oct 30 07:08:04.930377 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:04.930328    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1230dcae-e1cd-460d-86fc-bd40c8427b5f-tmp-dir\") pod \"premium-w0-06b26e-kserve-c9dbd9879-b4jlk\" (UID: \"1230dcae-e1cd-460d-86fc-bd40c8427b5f\") " pod="e2e-models-e2e-worker-w0-06b26e/premium-w0-06b26e-kserve-c9dbd9879-b4jlk"
Oct 30 07:08:04.930377 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:04.930380    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1230dcae-e1cd-460d-86fc-bd40c8427b5f-model-cache\") pod \"premium-w0-06b26e-kserve-c9dbd9879-b4jlk\" (UID: \"1230dcae-e1cd-460d-86fc-bd40c8427b5f\") " pod="e2e-models-e2e-worker-w0-06b26e/premium-w0-06b26e-kserve-c9dbd9879-b4jlk"
Oct 30 07:08:04.930691 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:04.930407    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szbqn\" (UniqueName: \"kubernetes.io/projected/1230dcae-e1cd-460d-86fc-bd40c8427b5f-kube-api-access-szbqn\") pod \"premium-w0-06b26e-kserve-c9dbd9879-b4jlk\" (UID: \"1230dcae-e1cd-460d-86fc-bd40c8427b5f\") " pod="e2e-models-e2e-worker-w0-06b26e/premium-w0-06b26e-kserve-c9dbd9879-b4jlk"
Oct 30 07:08:04.930691 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:04.930492    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1230dcae-e1cd-460d-86fc-bd40c8427b5f-dshm\") pod \"premium-w0-06b26e-kserve-c9dbd9879-b4jlk\" (UID: \"1230dcae-e1cd-460d-86fc-bd40c8427b5f\") " pod="e2e-models-e2e-worker-w0-06b26e/premium-w0-06b26e-kserve-c9dbd9879-b4jlk"
Oct 30 07:08:04.930691 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:04.930526    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1230dcae-e1cd-460d-86fc-bd40c8427b5f-home\") pod \"premium-w0-06b26e-kserve-c9dbd9879-b4jlk\" (UID: \"1230dcae-e1cd-460d-86fc-bd40c8427b5f\") " pod="e2e-models-e2e-worker-w0-06b26e/premium-w0-06b26e-kserve-c9dbd9879-b4jlk"
Oct 30 07:08:04.930691 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:04.930577    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1230dcae-e1cd-460d-86fc-bd40c8427b5f-tls-certs\") pod \"premium-w0-06b26e-kserve-c9dbd9879-b4jlk\" (UID: \"1230dcae-e1cd-460d-86fc-bd40c8427b5f\") " pod="e2e-models-e2e-worker-w0-06b26e/premium-w0-06b26e-kserve-c9dbd9879-b4jlk"
Oct 30 07:08:04.930935 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:04.930798    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1230dcae-e1cd-460d-86fc-bd40c8427b5f-tmp-dir\") pod \"premium-w0-06b26e-kserve-c9dbd9879-b4jlk\" (UID: \"1230dcae-e1cd-460d-86fc-bd40c8427b5f\") " pod="e2e-models-e2e-worker-w0-06b26e/premium-w0-06b26e-kserve-c9dbd9879-b4jlk"
Oct 30 07:08:04.931340 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:04.931310    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1230dcae-e1cd-460d-86fc-bd40c8427b5f-model-cache\") pod \"premium-w0-06b26e-kserve-c9dbd9879-b4jlk\" (UID: \"1230dcae-e1cd-460d-86fc-bd40c8427b5f\") " pod="e2e-models-e2e-worker-w0-06b26e/premium-w0-06b26e-kserve-c9dbd9879-b4jlk"
Oct 30 07:08:04.931693 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:04.931670    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1230dcae-e1cd-460d-86fc-bd40c8427b5f-home\") pod \"premium-w0-06b26e-kserve-c9dbd9879-b4jlk\" (UID: \"1230dcae-e1cd-460d-86fc-bd40c8427b5f\") " pod="e2e-models-e2e-worker-w0-06b26e/premium-w0-06b26e-kserve-c9dbd9879-b4jlk"
Oct 30 07:08:04.934061 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:04.934038    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1230dcae-e1cd-460d-86fc-bd40c8427b5f-tls-certs\") pod \"premium-w0-06b26e-kserve-c9dbd9879-b4jlk\" (UID: \"1230dcae-e1cd-460d-86fc-bd40c8427b5f\") " pod="e2e-models-e2e-worker-w0-06b26e/premium-w0-06b26e-kserve-c9dbd9879-b4jlk"
Oct 30 07:08:04.938851 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:04.938795    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1230dcae-e1cd-460d-86fc-bd40c8427b5f-dshm\") pod \"premium-w0-06b26e-kserve-c9dbd9879-b4jlk\" (UID: \"1230dcae-e1cd-460d-86fc-bd40c8427b5f\") " pod="e2e-models-e2e-worker-w0-06b26e/premium-w0-06b26e-kserve-c9dbd9879-b4jlk"
Oct 30 07:08:04.956386 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:04.956348    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szbqn\" (UniqueName: \"kubernetes.io/projected/1230dcae-e1cd-460d-86fc-bd40c8427b5f-kube-api-access-szbqn\") pod \"premium-w0-06b26e-kserve-c9dbd9879-b4jlk\" (UID: \"1230dcae-e1cd-460d-86fc-bd40c8427b5f\") " pod="e2e-models-e2e-worker-w0-06b26e/premium-w0-06b26e-kserve-c9dbd9879-b4jlk"
Oct 30 07:08:05.059197 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:05.059167    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-06b26e/premium-w0-06b26e-kserve-c9dbd9879-b4jlk"
Oct 30 07:08:05.356471 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:05.356446    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-06b26e/premium-w0-06b26e-kserve-c9dbd9879-b4jlk"]
Oct 30 07:08:05.356744 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:08:05.356718    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1230dcae_e1cd_460d_86fc_bd40c8427b5f.slice/crio-fac0494352b179db837f55dfce53f4b9fa52a542c7b88aed208003e081bc6c29 WatchSource:0}: Error finding container fac0494352b179db837f55dfce53f4b9fa52a542c7b88aed208003e081bc6c29: Status 404 returned error can't find the container with id fac0494352b179db837f55dfce53f4b9fa52a542c7b88aed208003e081bc6c29
Oct 30 07:08:05.692594 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:05.692500    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-06b26e/premium-w0-06b26e-kserve-c9dbd9879-b4jlk" event={"ID":"1230dcae-e1cd-460d-86fc-bd40c8427b5f","Type":"ContainerStarted","Data":"8ce4769417f2f3ff210967dee957c14f7b29519f85b0cf88f3df971dbb0aaaeb"}
Oct 30 07:08:05.692594 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:05.692559    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-06b26e/premium-w0-06b26e-kserve-c9dbd9879-b4jlk" event={"ID":"1230dcae-e1cd-460d-86fc-bd40c8427b5f","Type":"ContainerStarted","Data":"fac0494352b179db837f55dfce53f4b9fa52a542c7b88aed208003e081bc6c29"}
Oct 30 07:08:05.737887 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:05.737837    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-06b26e/premium-w0-06b26e-kserve-c9dbd9879-b4jlk" podStartSLOduration=1.737824579 podStartE2EDuration="1.737824579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:08:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:08:05.737558694 +0000 UTC m=+644.346965664" watchObservedRunningTime="2026-10-30 07:08:05.737824579 +0000 UTC m=+644.347231546"
Oct 30 07:08:06.059576 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:06.059535    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-06b26e/premium-w0-06b26e-kserve-c9dbd9879-b4jlk"
Oct 30 07:08:15.059942 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:15.059900    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-06b26e/premium-w0-06b26e-kserve-c9dbd9879-b4jlk"
Oct 30 07:08:15.073066 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:15.073040    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-06b26e/premium-w0-06b26e-kserve-c9dbd9879-b4jlk"
Oct 30 07:08:15.753358 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:15.753320    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-06b26e/premium-w0-06b26e-kserve-c9dbd9879-b4jlk"
Oct 30 07:08:17.210450 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:17.210409    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:08:31.855411 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:08:31.855373    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:08:41.336333 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:41.336297    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:08:57.796967 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:57.796934    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9"]
Oct 30 07:08:57.800888 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:57.800868    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9"
Oct 30 07:08:57.821656 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:57.821630    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-2eda2c-openshift-default-dockercfg-bdk2w\""
Oct 30 07:08:57.829683 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:57.829654    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7-workload-socket\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9\" (UID: \"44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9"
Oct 30 07:08:57.829816 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:57.829708    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7-istio-podinfo\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9\" (UID: \"44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9"
Oct 30 07:08:57.829816 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:57.829742    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7-istio-envoy\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9\" (UID: \"44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9"
Oct 30 07:08:57.829816 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:57.829772    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7-credential-socket\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9\" (UID: \"44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9"
Oct 30 07:08:57.829949 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:57.829883    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7-workload-certs\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9\" (UID: \"44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9"
Oct 30 07:08:57.829999 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:57.829980    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxlxg\" (UniqueName: \"kubernetes.io/projected/44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7-kube-api-access-lxlxg\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9\" (UID: \"44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9"
Oct 30 07:08:57.830041 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:57.830030    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7-istio-data\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9\" (UID: \"44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9"
Oct 30 07:08:57.830161 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:57.830135    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7-istio-token\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9\" (UID: \"44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9"
Oct 30 07:08:57.830246 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:57.830224    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7-istiod-ca-cert\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9\" (UID: \"44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9"
Oct 30 07:08:57.838113 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:57.838087    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9"]
Oct 30 07:08:57.931218 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:57.931188    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7-istiod-ca-cert\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9\" (UID: \"44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9"
Oct 30 07:08:57.931409 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:57.931251    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7-workload-socket\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9\" (UID: \"44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9"
Oct 30 07:08:57.931409 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:57.931280    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7-istio-podinfo\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9\" (UID: \"44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9"
Oct 30 07:08:57.931409 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:57.931304    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7-istio-envoy\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9\" (UID: \"44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9"
Oct 30 07:08:57.931409 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:57.931324    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7-credential-socket\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9\" (UID: \"44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9"
Oct 30 07:08:57.931409 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:57.931374    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7-workload-certs\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9\" (UID: \"44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9"
Oct 30 07:08:57.931691 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:57.931419    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxlxg\" (UniqueName: \"kubernetes.io/projected/44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7-kube-api-access-lxlxg\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9\" (UID: \"44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9"
Oct 30 07:08:57.931691 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:57.931461    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7-istio-data\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9\" (UID: \"44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9"
Oct 30 07:08:57.931691 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:57.931525    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7-istio-token\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9\" (UID: \"44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9"
Oct 30 07:08:57.931691 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:57.931655    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7-workload-socket\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9\" (UID: \"44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9"
Oct 30 07:08:57.931961 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:57.931878    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7-credential-socket\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9\" (UID: \"44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9"
Oct 30 07:08:57.931961 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:57.931932    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7-workload-certs\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9\" (UID: \"44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9"
Oct 30 07:08:57.932069 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:57.931985    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7-istio-data\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9\" (UID: \"44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9"
Oct 30 07:08:57.932128 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:57.932075    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7-istiod-ca-cert\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9\" (UID: \"44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9"
Oct 30 07:08:57.933541 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:57.933517    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7-istio-envoy\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9\" (UID: \"44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9"
Oct 30 07:08:57.934142 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:57.934121    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7-istio-podinfo\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9\" (UID: \"44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9"
Oct 30 07:08:57.959233 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:57.959207    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7-istio-token\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9\" (UID: \"44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9"
Oct 30 07:08:57.959428 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:57.959408    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxlxg\" (UniqueName: \"kubernetes.io/projected/44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7-kube-api-access-lxlxg\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9\" (UID: \"44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9"
Oct 30 07:08:58.113607 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:58.113530    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9"
Oct 30 07:08:58.425296 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:58.425264    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9"]
Oct 30 07:08:58.425620 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:08:58.425598    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44d3fa93_d6ab_40f7_a8f7_ee15e4ee0ca7.slice/crio-c1a03e6201d4cb9b6aaadf70b2cca37c0b78e21f238eeba49e677e39dc29a3ce WatchSource:0}: Error finding container c1a03e6201d4cb9b6aaadf70b2cca37c0b78e21f238eeba49e677e39dc29a3ce: Status 404 returned error can't find the container with id c1a03e6201d4cb9b6aaadf70b2cca37c0b78e21f238eeba49e677e39dc29a3ce
Oct 30 07:08:58.428318 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:58.428283    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:08:58.428411 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:58.428345    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:08:58.428411 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:58.428374    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:08:58.991345 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:58.991305    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9" event={"ID":"44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7","Type":"ContainerStarted","Data":"e8fa1e378c61c7febe12a7df2a4d245e543c7f570a095ae4007565ffe764ed56"}
Oct 30 07:08:58.991887 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:58.991365    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9" event={"ID":"44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7","Type":"ContainerStarted","Data":"c1a03e6201d4cb9b6aaadf70b2cca37c0b78e21f238eeba49e677e39dc29a3ce"}
Oct 30 07:08:59.081093 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:59.081031    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9" podStartSLOduration=2.081009703 podStartE2EDuration="2.081009703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:08:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:08:59.05360448 +0000 UTC m=+697.663011452" watchObservedRunningTime="2026-10-30 07:08:59.081009703 +0000 UTC m=+697.690416676"
Oct 30 07:08:59.114192 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:59.114153    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9"
Oct 30 07:08:59.115502 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:59.115468    6224 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Oct 30 07:08:59.115967 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:08:59.115523    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9" podUID="44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Oct 30 07:09:00.114671 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:00.114635    6224 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Oct 30 07:09:00.115069 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:00.114690    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9" podUID="44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Oct 30 07:09:01.114442 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:01.114405    6224 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Oct 30 07:09:01.114654 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:01.114463    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9" podUID="44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Oct 30 07:09:01.855591 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:09:01.855555    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:09:02.114959 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:02.114871    6224 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Oct 30 07:09:02.114959 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:02.114942    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9" podUID="44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Oct 30 07:09:03.114134 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:03.114095    6224 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Oct 30 07:09:03.114591 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:03.114148    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9" podUID="44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Oct 30 07:09:04.114676 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:04.114633    6224 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Oct 30 07:09:04.115181 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:04.114695    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9" podUID="44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Oct 30 07:09:05.749021 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:05.748978    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9"
Oct 30 07:09:05.750542 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:05.750521    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9"
Oct 30 07:09:05.753271 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:05.753245    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9"
Oct 30 07:09:08.908160 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:08.908130    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff"]
Oct 30 07:09:08.911457 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:08.911441    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff"
Oct 30 07:09:08.928017 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:08.927997    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-2bab4c-openshift-default-dockercfg-gqrfc\""
Oct 30 07:09:08.944213 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:08.944190    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff"]
Oct 30 07:09:09.069319 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:09.069284    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ada682d6-e6ed-4d78-a3cf-07072b35f4f0-credential-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff\" (UID: \"ada682d6-e6ed-4d78-a3cf-07072b35f4f0\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff"
Oct 30 07:09:09.069483 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:09.069329    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ada682d6-e6ed-4d78-a3cf-07072b35f4f0-workload-certs\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff\" (UID: \"ada682d6-e6ed-4d78-a3cf-07072b35f4f0\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff"
Oct 30 07:09:09.069483 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:09.069354    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ada682d6-e6ed-4d78-a3cf-07072b35f4f0-istio-envoy\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff\" (UID: \"ada682d6-e6ed-4d78-a3cf-07072b35f4f0\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff"
Oct 30 07:09:09.069583 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:09.069481    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwxnx\" (UniqueName: \"kubernetes.io/projected/ada682d6-e6ed-4d78-a3cf-07072b35f4f0-kube-api-access-jwxnx\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff\" (UID: \"ada682d6-e6ed-4d78-a3cf-07072b35f4f0\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff"
Oct 30 07:09:09.069583 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:09.069516    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ada682d6-e6ed-4d78-a3cf-07072b35f4f0-istio-podinfo\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff\" (UID: \"ada682d6-e6ed-4d78-a3cf-07072b35f4f0\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff"
Oct 30 07:09:09.069583 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:09.069541    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ada682d6-e6ed-4d78-a3cf-07072b35f4f0-workload-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff\" (UID: \"ada682d6-e6ed-4d78-a3cf-07072b35f4f0\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff"
Oct 30 07:09:09.069691 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:09.069651    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ada682d6-e6ed-4d78-a3cf-07072b35f4f0-istiod-ca-cert\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff\" (UID: \"ada682d6-e6ed-4d78-a3cf-07072b35f4f0\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff"
Oct 30 07:09:09.069732 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:09.069709    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ada682d6-e6ed-4d78-a3cf-07072b35f4f0-istio-token\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff\" (UID: \"ada682d6-e6ed-4d78-a3cf-07072b35f4f0\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff"
Oct 30 07:09:09.069769 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:09.069757    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ada682d6-e6ed-4d78-a3cf-07072b35f4f0-istio-data\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff\" (UID: \"ada682d6-e6ed-4d78-a3cf-07072b35f4f0\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff"
Oct 30 07:09:09.170822 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:09.170750    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ada682d6-e6ed-4d78-a3cf-07072b35f4f0-istio-token\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff\" (UID: \"ada682d6-e6ed-4d78-a3cf-07072b35f4f0\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff"
Oct 30 07:09:09.170822 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:09.170780    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ada682d6-e6ed-4d78-a3cf-07072b35f4f0-istio-data\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff\" (UID: \"ada682d6-e6ed-4d78-a3cf-07072b35f4f0\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff"
Oct 30 07:09:09.170822 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:09.170800    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ada682d6-e6ed-4d78-a3cf-07072b35f4f0-credential-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff\" (UID: \"ada682d6-e6ed-4d78-a3cf-07072b35f4f0\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff"
Oct 30 07:09:09.171052 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:09.170849    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ada682d6-e6ed-4d78-a3cf-07072b35f4f0-workload-certs\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff\" (UID: \"ada682d6-e6ed-4d78-a3cf-07072b35f4f0\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff"
Oct 30 07:09:09.171052 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:09.170882    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ada682d6-e6ed-4d78-a3cf-07072b35f4f0-istio-envoy\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff\" (UID: \"ada682d6-e6ed-4d78-a3cf-07072b35f4f0\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff"
Oct 30 07:09:09.171052 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:09.170938    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwxnx\" (UniqueName: \"kubernetes.io/projected/ada682d6-e6ed-4d78-a3cf-07072b35f4f0-kube-api-access-jwxnx\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff\" (UID: \"ada682d6-e6ed-4d78-a3cf-07072b35f4f0\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff"
Oct 30 07:09:09.171052 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:09.170968    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ada682d6-e6ed-4d78-a3cf-07072b35f4f0-istio-podinfo\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff\" (UID: \"ada682d6-e6ed-4d78-a3cf-07072b35f4f0\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff"
Oct 30 07:09:09.171052 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:09.170999    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ada682d6-e6ed-4d78-a3cf-07072b35f4f0-workload-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff\" (UID: \"ada682d6-e6ed-4d78-a3cf-07072b35f4f0\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff"
Oct 30 07:09:09.171052 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:09.171036    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ada682d6-e6ed-4d78-a3cf-07072b35f4f0-istiod-ca-cert\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff\" (UID: \"ada682d6-e6ed-4d78-a3cf-07072b35f4f0\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff"
Oct 30 07:09:09.171356 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:09.171191    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ada682d6-e6ed-4d78-a3cf-07072b35f4f0-credential-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff\" (UID: \"ada682d6-e6ed-4d78-a3cf-07072b35f4f0\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff"
Oct 30 07:09:09.171356 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:09.171249    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ada682d6-e6ed-4d78-a3cf-07072b35f4f0-istio-data\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff\" (UID: \"ada682d6-e6ed-4d78-a3cf-07072b35f4f0\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff"
Oct 30 07:09:09.171474 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:09.171404    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ada682d6-e6ed-4d78-a3cf-07072b35f4f0-workload-certs\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff\" (UID: \"ada682d6-e6ed-4d78-a3cf-07072b35f4f0\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff"
Oct 30 07:09:09.171541 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:09.171480    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ada682d6-e6ed-4d78-a3cf-07072b35f4f0-workload-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff\" (UID: \"ada682d6-e6ed-4d78-a3cf-07072b35f4f0\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff"
Oct 30 07:09:09.171714 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:09.171692    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ada682d6-e6ed-4d78-a3cf-07072b35f4f0-istiod-ca-cert\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff\" (UID: \"ada682d6-e6ed-4d78-a3cf-07072b35f4f0\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff"
Oct 30 07:09:09.173438 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:09.173403    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ada682d6-e6ed-4d78-a3cf-07072b35f4f0-istio-envoy\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff\" (UID: \"ada682d6-e6ed-4d78-a3cf-07072b35f4f0\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff"
Oct 30 07:09:09.173581 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:09.173558    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ada682d6-e6ed-4d78-a3cf-07072b35f4f0-istio-podinfo\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff\" (UID: \"ada682d6-e6ed-4d78-a3cf-07072b35f4f0\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff"
Oct 30 07:09:09.195790 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:09.195767    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ada682d6-e6ed-4d78-a3cf-07072b35f4f0-istio-token\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff\" (UID: \"ada682d6-e6ed-4d78-a3cf-07072b35f4f0\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff"
Oct 30 07:09:09.195966 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:09.195948    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwxnx\" (UniqueName: \"kubernetes.io/projected/ada682d6-e6ed-4d78-a3cf-07072b35f4f0-kube-api-access-jwxnx\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff\" (UID: \"ada682d6-e6ed-4d78-a3cf-07072b35f4f0\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff"
Oct 30 07:09:09.224760 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:09.224738    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff"
Oct 30 07:09:09.536496 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:09:09.536454    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podada682d6_e6ed_4d78_a3cf_07072b35f4f0.slice/crio-a7b678ebcf716ae709c5e480e0d392307606b92031864849ceddbfb43ea465e7 WatchSource:0}: Error finding container a7b678ebcf716ae709c5e480e0d392307606b92031864849ceddbfb43ea465e7: Status 404 returned error can't find the container with id a7b678ebcf716ae709c5e480e0d392307606b92031864849ceddbfb43ea465e7
Oct 30 07:09:09.537597 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:09.537574    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff"]
Oct 30 07:09:09.538727 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:09.538698    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:09:09.538779 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:09.538762    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:09:09.538861 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:09.538791    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:09:10.061190 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:10.061140    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff" event={"ID":"ada682d6-e6ed-4d78-a3cf-07072b35f4f0","Type":"ContainerStarted","Data":"eb6841701a7999487e6d9a19348bd93ab96f61d1408b3a845db072a1432b47b9"}
Oct 30 07:09:10.061593 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:10.061218    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff" event={"ID":"ada682d6-e6ed-4d78-a3cf-07072b35f4f0","Type":"ContainerStarted","Data":"a7b678ebcf716ae709c5e480e0d392307606b92031864849ceddbfb43ea465e7"}
Oct 30 07:09:10.139769 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:10.139714    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff" podStartSLOduration=2.139699686 podStartE2EDuration="2.139699686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:09:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:09:10.124350708 +0000 UTC m=+708.733757679" watchObservedRunningTime="2026-10-30 07:09:10.139699686 +0000 UTC m=+708.749106656"
Oct 30 07:09:10.225892 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:10.225855    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff"
Oct 30 07:09:10.227527 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:10.227496    6224 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Oct 30 07:09:10.227666 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:10.227542    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff" podUID="ada682d6-e6ed-4d78-a3cf-07072b35f4f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Oct 30 07:09:10.233363 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:10.233337    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:09:11.225630 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:11.225595    6224 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Oct 30 07:09:11.226054 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:11.225655    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff" podUID="ada682d6-e6ed-4d78-a3cf-07072b35f4f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Oct 30 07:09:12.225105 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:12.225068    6224 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Oct 30 07:09:12.225299 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:12.225134    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff" podUID="ada682d6-e6ed-4d78-a3cf-07072b35f4f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Oct 30 07:09:13.225481 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:13.225446    6224 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Oct 30 07:09:13.225893 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:13.225505    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff" podUID="ada682d6-e6ed-4d78-a3cf-07072b35f4f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Oct 30 07:09:14.225466 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:14.225431    6224 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Oct 30 07:09:14.225669 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:14.225493    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff" podUID="ada682d6-e6ed-4d78-a3cf-07072b35f4f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Oct 30 07:09:15.225061 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:15.225026    6224 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Oct 30 07:09:15.225233 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:15.225080    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff" podUID="ada682d6-e6ed-4d78-a3cf-07072b35f4f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Oct 30 07:09:16.229579 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:16.229550    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff"
Oct 30 07:09:16.230024 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:16.229836    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff"
Oct 30 07:09:16.230432 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:16.230417    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff"
Oct 30 07:09:24.087375 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:24.087339    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-2bab4c/distinct-w4-2bab4c-kserve-6868d767c9-j7wlm"]
Oct 30 07:09:24.091993 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:24.091965    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2bab4c/distinct-w4-2bab4c-kserve-6868d767c9-j7wlm"
Oct 30 07:09:24.109863 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:24.109821    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-2bab4c/distinct-w4-2bab4c-kserve-6868d767c9-j7wlm"]
Oct 30 07:09:24.121802 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:24.121774    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-2bab4c\"/\"distinct-w4-2bab4c-kserve-self-signed-certs\""
Oct 30 07:09:24.134167 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:24.134147    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00-tls-certs\") pod \"distinct-w4-2bab4c-kserve-6868d767c9-j7wlm\" (UID: \"11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00\") " pod="e2e-models-e2e-worker-w4-2bab4c/distinct-w4-2bab4c-kserve-6868d767c9-j7wlm"
Oct 30 07:09:24.134271 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:24.134206    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00-tmp-dir\") pod \"distinct-w4-2bab4c-kserve-6868d767c9-j7wlm\" (UID: \"11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00\") " pod="e2e-models-e2e-worker-w4-2bab4c/distinct-w4-2bab4c-kserve-6868d767c9-j7wlm"
Oct 30 07:09:24.134271 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:24.134256    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00-model-cache\") pod \"distinct-w4-2bab4c-kserve-6868d767c9-j7wlm\" (UID: \"11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00\") " pod="e2e-models-e2e-worker-w4-2bab4c/distinct-w4-2bab4c-kserve-6868d767c9-j7wlm"
Oct 30 07:09:24.134380 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:24.134334    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00-home\") pod \"distinct-w4-2bab4c-kserve-6868d767c9-j7wlm\" (UID: \"11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00\") " pod="e2e-models-e2e-worker-w4-2bab4c/distinct-w4-2bab4c-kserve-6868d767c9-j7wlm"
Oct 30 07:09:24.134380 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:24.134372    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00-dshm\") pod \"distinct-w4-2bab4c-kserve-6868d767c9-j7wlm\" (UID: \"11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00\") " pod="e2e-models-e2e-worker-w4-2bab4c/distinct-w4-2bab4c-kserve-6868d767c9-j7wlm"
Oct 30 07:09:24.134498 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:24.134478    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4sdz\" (UniqueName: \"kubernetes.io/projected/11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00-kube-api-access-j4sdz\") pod \"distinct-w4-2bab4c-kserve-6868d767c9-j7wlm\" (UID: \"11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00\") " pod="e2e-models-e2e-worker-w4-2bab4c/distinct-w4-2bab4c-kserve-6868d767c9-j7wlm"
Oct 30 07:09:24.236042 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:24.236003    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00-home\") pod \"distinct-w4-2bab4c-kserve-6868d767c9-j7wlm\" (UID: \"11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00\") " pod="e2e-models-e2e-worker-w4-2bab4c/distinct-w4-2bab4c-kserve-6868d767c9-j7wlm"
Oct 30 07:09:24.236042 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:24.236041    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00-dshm\") pod \"distinct-w4-2bab4c-kserve-6868d767c9-j7wlm\" (UID: \"11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00\") " pod="e2e-models-e2e-worker-w4-2bab4c/distinct-w4-2bab4c-kserve-6868d767c9-j7wlm"
Oct 30 07:09:24.236285 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:24.236090    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4sdz\" (UniqueName: \"kubernetes.io/projected/11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00-kube-api-access-j4sdz\") pod \"distinct-w4-2bab4c-kserve-6868d767c9-j7wlm\" (UID: \"11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00\") " pod="e2e-models-e2e-worker-w4-2bab4c/distinct-w4-2bab4c-kserve-6868d767c9-j7wlm"
Oct 30 07:09:24.236285 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:24.236157    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00-tls-certs\") pod \"distinct-w4-2bab4c-kserve-6868d767c9-j7wlm\" (UID: \"11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00\") " pod="e2e-models-e2e-worker-w4-2bab4c/distinct-w4-2bab4c-kserve-6868d767c9-j7wlm"
Oct 30 07:09:24.236285 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:24.236221    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00-tmp-dir\") pod \"distinct-w4-2bab4c-kserve-6868d767c9-j7wlm\" (UID: \"11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00\") " pod="e2e-models-e2e-worker-w4-2bab4c/distinct-w4-2bab4c-kserve-6868d767c9-j7wlm"
Oct 30 07:09:24.236285 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:24.236236    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00-model-cache\") pod \"distinct-w4-2bab4c-kserve-6868d767c9-j7wlm\" (UID: \"11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00\") " pod="e2e-models-e2e-worker-w4-2bab4c/distinct-w4-2bab4c-kserve-6868d767c9-j7wlm"
Oct 30 07:09:24.236532 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:24.236503    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00-home\") pod \"distinct-w4-2bab4c-kserve-6868d767c9-j7wlm\" (UID: \"11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00\") " pod="e2e-models-e2e-worker-w4-2bab4c/distinct-w4-2bab4c-kserve-6868d767c9-j7wlm"
Oct 30 07:09:24.236640 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:24.236617    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00-model-cache\") pod \"distinct-w4-2bab4c-kserve-6868d767c9-j7wlm\" (UID: \"11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00\") " pod="e2e-models-e2e-worker-w4-2bab4c/distinct-w4-2bab4c-kserve-6868d767c9-j7wlm"
Oct 30 07:09:24.236759 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:24.236727    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00-tmp-dir\") pod \"distinct-w4-2bab4c-kserve-6868d767c9-j7wlm\" (UID: \"11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00\") " pod="e2e-models-e2e-worker-w4-2bab4c/distinct-w4-2bab4c-kserve-6868d767c9-j7wlm"
Oct 30 07:09:24.238431 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:24.238411    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00-dshm\") pod \"distinct-w4-2bab4c-kserve-6868d767c9-j7wlm\" (UID: \"11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00\") " pod="e2e-models-e2e-worker-w4-2bab4c/distinct-w4-2bab4c-kserve-6868d767c9-j7wlm"
Oct 30 07:09:24.239238 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:24.239207    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00-tls-certs\") pod \"distinct-w4-2bab4c-kserve-6868d767c9-j7wlm\" (UID: \"11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00\") " pod="e2e-models-e2e-worker-w4-2bab4c/distinct-w4-2bab4c-kserve-6868d767c9-j7wlm"
Oct 30 07:09:24.261188 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:24.261164    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4sdz\" (UniqueName: \"kubernetes.io/projected/11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00-kube-api-access-j4sdz\") pod \"distinct-w4-2bab4c-kserve-6868d767c9-j7wlm\" (UID: \"11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00\") " pod="e2e-models-e2e-worker-w4-2bab4c/distinct-w4-2bab4c-kserve-6868d767c9-j7wlm"
Oct 30 07:09:24.406104 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:24.406025    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2bab4c/distinct-w4-2bab4c-kserve-6868d767c9-j7wlm"
Oct 30 07:09:24.703790 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:09:24.703755    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod11c3cd4b_c4fc_4ef6_9eb7_85f9be0b8a00.slice/crio-35b7528dc6840b47d4a32d2b951f3aa1c075441f376307817a4c8a3ace5c6739 WatchSource:0}: Error finding container 35b7528dc6840b47d4a32d2b951f3aa1c075441f376307817a4c8a3ace5c6739: Status 404 returned error can't find the container with id 35b7528dc6840b47d4a32d2b951f3aa1c075441f376307817a4c8a3ace5c6739
Oct 30 07:09:24.704877 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:24.704855    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-2bab4c/distinct-w4-2bab4c-kserve-6868d767c9-j7wlm"]
Oct 30 07:09:25.168889 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:25.168848    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2bab4c/distinct-w4-2bab4c-kserve-6868d767c9-j7wlm" event={"ID":"11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00","Type":"ContainerStarted","Data":"8fb2e28712f58d1bccf7d6e1685849c57c8975d11b5199fed4fe3ab2bb128e58"}
Oct 30 07:09:25.169339 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:25.168899    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2bab4c/distinct-w4-2bab4c-kserve-6868d767c9-j7wlm" event={"ID":"11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00","Type":"ContainerStarted","Data":"35b7528dc6840b47d4a32d2b951f3aa1c075441f376307817a4c8a3ace5c6739"}
Oct 30 07:09:25.215364 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:25.215316    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-2bab4c/distinct-w4-2bab4c-kserve-6868d767c9-j7wlm" podStartSLOduration=1.215302343 podStartE2EDuration="1.215302343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:09:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:09:25.213992466 +0000 UTC m=+723.823399437" watchObservedRunningTime="2026-10-30 07:09:25.215302343 +0000 UTC m=+723.824709313"
Oct 30 07:09:25.406407 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:25.406375    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2bab4c/distinct-w4-2bab4c-kserve-6868d767c9-j7wlm"
Oct 30 07:09:29.757358 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:29.757326    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:09:31.855556 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:09:31.855516    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:09:34.407234 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:34.407160    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-2bab4c/distinct-w4-2bab4c-kserve-6868d767c9-j7wlm"
Oct 30 07:09:34.421170 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:34.421147    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-2bab4c/distinct-w4-2bab4c-kserve-6868d767c9-j7wlm"
Oct 30 07:09:35.233198 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:35.233168    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2bab4c/distinct-w4-2bab4c-kserve-6868d767c9-j7wlm"
Oct 30 07:09:47.709901 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:47.709852    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:09:50.241358 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:50.241322    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth"]
Oct 30 07:09:50.245740 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:50.245716    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth"
Oct 30 07:09:50.276620 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:50.276582    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth"]
Oct 30 07:09:50.281430 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:50.281400    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-bb6be4-openshift-default-dockercfg-2kstl\""
Oct 30 07:09:50.301346 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:50.301311    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ec378e4-d80e-4306-b1b8-8556b80ce58d-istio-podinfo\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth\" (UID: \"9ec378e4-d80e-4306-b1b8-8556b80ce58d\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth"
Oct 30 07:09:50.301542 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:50.301371    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ec378e4-d80e-4306-b1b8-8556b80ce58d-istio-token\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth\" (UID: \"9ec378e4-d80e-4306-b1b8-8556b80ce58d\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth"
Oct 30 07:09:50.301542 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:50.301481    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ec378e4-d80e-4306-b1b8-8556b80ce58d-istio-data\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth\" (UID: \"9ec378e4-d80e-4306-b1b8-8556b80ce58d\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth"
Oct 30 07:09:50.301542 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:50.301515    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ec378e4-d80e-4306-b1b8-8556b80ce58d-istiod-ca-cert\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth\" (UID: \"9ec378e4-d80e-4306-b1b8-8556b80ce58d\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth"
Oct 30 07:09:50.301724 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:50.301551    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec378e4-d80e-4306-b1b8-8556b80ce58d-credential-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth\" (UID: \"9ec378e4-d80e-4306-b1b8-8556b80ce58d\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth"
Oct 30 07:09:50.301724 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:50.301582    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ec378e4-d80e-4306-b1b8-8556b80ce58d-istio-envoy\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth\" (UID: \"9ec378e4-d80e-4306-b1b8-8556b80ce58d\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth"
Oct 30 07:09:50.301724 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:50.301606    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec378e4-d80e-4306-b1b8-8556b80ce58d-workload-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth\" (UID: \"9ec378e4-d80e-4306-b1b8-8556b80ce58d\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth"
Oct 30 07:09:50.301724 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:50.301634    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48chc\" (UniqueName: \"kubernetes.io/projected/9ec378e4-d80e-4306-b1b8-8556b80ce58d-kube-api-access-48chc\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth\" (UID: \"9ec378e4-d80e-4306-b1b8-8556b80ce58d\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth"
Oct 30 07:09:50.301724 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:50.301669    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ec378e4-d80e-4306-b1b8-8556b80ce58d-workload-certs\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth\" (UID: \"9ec378e4-d80e-4306-b1b8-8556b80ce58d\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth"
Oct 30 07:09:50.402274 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:50.402238    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ec378e4-d80e-4306-b1b8-8556b80ce58d-istio-data\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth\" (UID: \"9ec378e4-d80e-4306-b1b8-8556b80ce58d\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth"
Oct 30 07:09:50.402447 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:50.402284    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ec378e4-d80e-4306-b1b8-8556b80ce58d-istiod-ca-cert\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth\" (UID: \"9ec378e4-d80e-4306-b1b8-8556b80ce58d\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth"
Oct 30 07:09:50.402447 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:50.402329    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec378e4-d80e-4306-b1b8-8556b80ce58d-credential-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth\" (UID: \"9ec378e4-d80e-4306-b1b8-8556b80ce58d\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth"
Oct 30 07:09:50.402447 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:50.402379    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ec378e4-d80e-4306-b1b8-8556b80ce58d-istio-envoy\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth\" (UID: \"9ec378e4-d80e-4306-b1b8-8556b80ce58d\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth"
Oct 30 07:09:50.402447 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:50.402417    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec378e4-d80e-4306-b1b8-8556b80ce58d-workload-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth\" (UID: \"9ec378e4-d80e-4306-b1b8-8556b80ce58d\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth"
Oct 30 07:09:50.402678 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:50.402462    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48chc\" (UniqueName: \"kubernetes.io/projected/9ec378e4-d80e-4306-b1b8-8556b80ce58d-kube-api-access-48chc\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth\" (UID: \"9ec378e4-d80e-4306-b1b8-8556b80ce58d\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth"
Oct 30 07:09:50.402742 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:50.402682    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ec378e4-d80e-4306-b1b8-8556b80ce58d-istio-data\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth\" (UID: \"9ec378e4-d80e-4306-b1b8-8556b80ce58d\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth"
Oct 30 07:09:50.402888 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:50.402863    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec378e4-d80e-4306-b1b8-8556b80ce58d-credential-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth\" (UID: \"9ec378e4-d80e-4306-b1b8-8556b80ce58d\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth"
Oct 30 07:09:50.402888 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:50.402877    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ec378e4-d80e-4306-b1b8-8556b80ce58d-workload-certs\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth\" (UID: \"9ec378e4-d80e-4306-b1b8-8556b80ce58d\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth"
Oct 30 07:09:50.403021 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:50.402915    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec378e4-d80e-4306-b1b8-8556b80ce58d-workload-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth\" (UID: \"9ec378e4-d80e-4306-b1b8-8556b80ce58d\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth"
Oct 30 07:09:50.403021 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:50.402941    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ec378e4-d80e-4306-b1b8-8556b80ce58d-istio-podinfo\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth\" (UID: \"9ec378e4-d80e-4306-b1b8-8556b80ce58d\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth"
Oct 30 07:09:50.403021 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:50.402963    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ec378e4-d80e-4306-b1b8-8556b80ce58d-workload-certs\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth\" (UID: \"9ec378e4-d80e-4306-b1b8-8556b80ce58d\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth"
Oct 30 07:09:50.403021 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:50.403008    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ec378e4-d80e-4306-b1b8-8556b80ce58d-istio-token\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth\" (UID: \"9ec378e4-d80e-4306-b1b8-8556b80ce58d\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth"
Oct 30 07:09:50.403223 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:50.403092    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ec378e4-d80e-4306-b1b8-8556b80ce58d-istiod-ca-cert\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth\" (UID: \"9ec378e4-d80e-4306-b1b8-8556b80ce58d\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth"
Oct 30 07:09:50.404666 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:50.404646    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ec378e4-d80e-4306-b1b8-8556b80ce58d-istio-envoy\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth\" (UID: \"9ec378e4-d80e-4306-b1b8-8556b80ce58d\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth"
Oct 30 07:09:50.405220 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:50.405200    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ec378e4-d80e-4306-b1b8-8556b80ce58d-istio-podinfo\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth\" (UID: \"9ec378e4-d80e-4306-b1b8-8556b80ce58d\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth"
Oct 30 07:09:50.428684 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:50.428656    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ec378e4-d80e-4306-b1b8-8556b80ce58d-istio-token\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth\" (UID: \"9ec378e4-d80e-4306-b1b8-8556b80ce58d\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth"
Oct 30 07:09:50.428959 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:50.428941    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48chc\" (UniqueName: \"kubernetes.io/projected/9ec378e4-d80e-4306-b1b8-8556b80ce58d-kube-api-access-48chc\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth\" (UID: \"9ec378e4-d80e-4306-b1b8-8556b80ce58d\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth"
Oct 30 07:09:50.567229 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:50.567197    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth"
Oct 30 07:09:50.882858 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:50.882829    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth"]
Oct 30 07:09:50.882989 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:09:50.882884    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ec378e4_d80e_4306_b1b8_8556b80ce58d.slice/crio-aa825ccfbb09388ab42e096ec47df174f4b1bb3f14fb16ceac90bd38b6cb118e WatchSource:0}: Error finding container aa825ccfbb09388ab42e096ec47df174f4b1bb3f14fb16ceac90bd38b6cb118e: Status 404 returned error can't find the container with id aa825ccfbb09388ab42e096ec47df174f4b1bb3f14fb16ceac90bd38b6cb118e
Oct 30 07:09:50.885298 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:50.885263    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:09:50.885411 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:50.885340    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:09:50.885411 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:50.885385    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:09:51.319346 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:51.319307    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth" event={"ID":"9ec378e4-d80e-4306-b1b8-8556b80ce58d","Type":"ContainerStarted","Data":"4b48742d8237c38f3e906322f7cc80c9b2e8555465da374f092a99f67af591b4"}
Oct 30 07:09:51.319859 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:51.319375    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth" event={"ID":"9ec378e4-d80e-4306-b1b8-8556b80ce58d","Type":"ContainerStarted","Data":"aa825ccfbb09388ab42e096ec47df174f4b1bb3f14fb16ceac90bd38b6cb118e"}
Oct 30 07:09:51.394130 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:51.394069    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth" podStartSLOduration=1.394053998 podStartE2EDuration="1.394053998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:09:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:09:51.392618326 +0000 UTC m=+750.002025298" watchObservedRunningTime="2026-10-30 07:09:51.394053998 +0000 UTC m=+750.003461038"
Oct 30 07:09:51.587949 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:51.585166    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth"
Oct 30 07:09:51.588130 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:51.587980    6224 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 30 07:09:51.588130 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:51.588033    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth" podUID="9ec378e4-d80e-4306-b1b8-8556b80ce58d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 30 07:09:52.582148 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:52.582111    6224 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 30 07:09:52.582717 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:52.582687    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth" podUID="9ec378e4-d80e-4306-b1b8-8556b80ce58d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 30 07:09:53.590580 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:53.590545    6224 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 30 07:09:53.591207 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:53.591169    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth" podUID="9ec378e4-d80e-4306-b1b8-8556b80ce58d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 30 07:09:54.578152 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:54.578115    6224 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 30 07:09:54.578396 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:54.578366    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth" podUID="9ec378e4-d80e-4306-b1b8-8556b80ce58d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 30 07:09:55.572832 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:55.568052    6224 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 30 07:09:55.572832 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:55.568110    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth" podUID="9ec378e4-d80e-4306-b1b8-8556b80ce58d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 30 07:09:56.573938 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:56.573906    6224 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 30 07:09:56.574440 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:56.574398    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth" podUID="9ec378e4-d80e-4306-b1b8-8556b80ce58d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 30 07:09:58.367083 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:58.367046    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth"
Oct 30 07:09:58.368011 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:58.367985    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth"
Oct 30 07:09:58.368854 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:58.368801    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth"
Oct 30 07:09:58.605418 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:58.605387    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq"]
Oct 30 07:09:58.609659 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:58.609635    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq"
Oct 30 07:09:58.630591 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:58.630422    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-4fe2ed-openshift-default-dockercfg-s6hbq\""
Oct 30 07:09:58.636321 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:58.636289    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq"]
Oct 30 07:09:58.688576 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:58.688549    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fd8ca77-b283-4d93-87e7-640397564926-credential-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq\" (UID: \"6fd8ca77-b283-4d93-87e7-640397564926\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq"
Oct 30 07:09:58.688576 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:58.688580    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fd8ca77-b283-4d93-87e7-640397564926-istiod-ca-cert\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq\" (UID: \"6fd8ca77-b283-4d93-87e7-640397564926\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq"
Oct 30 07:09:58.688770 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:58.688607    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97nfs\" (UniqueName: \"kubernetes.io/projected/6fd8ca77-b283-4d93-87e7-640397564926-kube-api-access-97nfs\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq\" (UID: \"6fd8ca77-b283-4d93-87e7-640397564926\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq"
Oct 30 07:09:58.688770 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:58.688743    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fd8ca77-b283-4d93-87e7-640397564926-istio-envoy\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq\" (UID: \"6fd8ca77-b283-4d93-87e7-640397564926\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq"
Oct 30 07:09:58.688906 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:58.688771    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fd8ca77-b283-4d93-87e7-640397564926-istio-data\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq\" (UID: \"6fd8ca77-b283-4d93-87e7-640397564926\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq"
Oct 30 07:09:58.688906 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:58.688884    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fd8ca77-b283-4d93-87e7-640397564926-istio-podinfo\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq\" (UID: \"6fd8ca77-b283-4d93-87e7-640397564926\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq"
Oct 30 07:09:58.688982 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:58.688906    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fd8ca77-b283-4d93-87e7-640397564926-workload-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq\" (UID: \"6fd8ca77-b283-4d93-87e7-640397564926\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq"
Oct 30 07:09:58.688982 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:58.688929    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fd8ca77-b283-4d93-87e7-640397564926-workload-certs\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq\" (UID: \"6fd8ca77-b283-4d93-87e7-640397564926\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq"
Oct 30 07:09:58.688982 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:58.688959    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fd8ca77-b283-4d93-87e7-640397564926-istio-token\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq\" (UID: \"6fd8ca77-b283-4d93-87e7-640397564926\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq"
Oct 30 07:09:58.790267 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:58.790234    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fd8ca77-b283-4d93-87e7-640397564926-credential-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq\" (UID: \"6fd8ca77-b283-4d93-87e7-640397564926\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq"
Oct 30 07:09:58.790267 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:58.790269    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fd8ca77-b283-4d93-87e7-640397564926-istiod-ca-cert\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq\" (UID: \"6fd8ca77-b283-4d93-87e7-640397564926\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq"
Oct 30 07:09:58.790468 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:58.790306    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97nfs\" (UniqueName: \"kubernetes.io/projected/6fd8ca77-b283-4d93-87e7-640397564926-kube-api-access-97nfs\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq\" (UID: \"6fd8ca77-b283-4d93-87e7-640397564926\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq"
Oct 30 07:09:58.790468 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:58.790353    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fd8ca77-b283-4d93-87e7-640397564926-istio-envoy\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq\" (UID: \"6fd8ca77-b283-4d93-87e7-640397564926\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq"
Oct 30 07:09:58.790468 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:58.790382    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fd8ca77-b283-4d93-87e7-640397564926-istio-data\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq\" (UID: \"6fd8ca77-b283-4d93-87e7-640397564926\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq"
Oct 30 07:09:58.790626 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:58.790476    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fd8ca77-b283-4d93-87e7-640397564926-istio-podinfo\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq\" (UID: \"6fd8ca77-b283-4d93-87e7-640397564926\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq"
Oct 30 07:09:58.790626 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:58.790508    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fd8ca77-b283-4d93-87e7-640397564926-workload-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq\" (UID: \"6fd8ca77-b283-4d93-87e7-640397564926\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq"
Oct 30 07:09:58.790626 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:58.790541    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fd8ca77-b283-4d93-87e7-640397564926-workload-certs\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq\" (UID: \"6fd8ca77-b283-4d93-87e7-640397564926\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq"
Oct 30 07:09:58.790626 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:58.790589    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fd8ca77-b283-4d93-87e7-640397564926-istio-token\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq\" (UID: \"6fd8ca77-b283-4d93-87e7-640397564926\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq"
Oct 30 07:09:58.790829 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:58.790629    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fd8ca77-b283-4d93-87e7-640397564926-credential-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq\" (UID: \"6fd8ca77-b283-4d93-87e7-640397564926\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq"
Oct 30 07:09:58.790964 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:58.790927    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fd8ca77-b283-4d93-87e7-640397564926-workload-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq\" (UID: \"6fd8ca77-b283-4d93-87e7-640397564926\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq"
Oct 30 07:09:58.791102 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:58.790943    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fd8ca77-b283-4d93-87e7-640397564926-istio-data\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq\" (UID: \"6fd8ca77-b283-4d93-87e7-640397564926\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq"
Oct 30 07:09:58.791162 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:58.791125    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fd8ca77-b283-4d93-87e7-640397564926-istiod-ca-cert\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq\" (UID: \"6fd8ca77-b283-4d93-87e7-640397564926\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq"
Oct 30 07:09:58.791231 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:58.791190    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fd8ca77-b283-4d93-87e7-640397564926-workload-certs\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq\" (UID: \"6fd8ca77-b283-4d93-87e7-640397564926\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq"
Oct 30 07:09:58.792681 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:58.792662    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fd8ca77-b283-4d93-87e7-640397564926-istio-envoy\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq\" (UID: \"6fd8ca77-b283-4d93-87e7-640397564926\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq"
Oct 30 07:09:58.793125 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:58.793102    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fd8ca77-b283-4d93-87e7-640397564926-istio-podinfo\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq\" (UID: \"6fd8ca77-b283-4d93-87e7-640397564926\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq"
Oct 30 07:09:58.815533 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:58.815510    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fd8ca77-b283-4d93-87e7-640397564926-istio-token\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq\" (UID: \"6fd8ca77-b283-4d93-87e7-640397564926\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq"
Oct 30 07:09:58.815750 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:58.815727    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97nfs\" (UniqueName: \"kubernetes.io/projected/6fd8ca77-b283-4d93-87e7-640397564926-kube-api-access-97nfs\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq\" (UID: \"6fd8ca77-b283-4d93-87e7-640397564926\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq"
Oct 30 07:09:58.922743 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:58.922672    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq"
Oct 30 07:09:59.234926 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:59.234897    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq"]
Oct 30 07:09:59.235888 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:09:59.235862    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fd8ca77_b283_4d93_87e7_640397564926.slice/crio-5567ca82082345865da633f31106792bc0e11fc5d5fe03b986ec1e81bbb89b73 WatchSource:0}: Error finding container 5567ca82082345865da633f31106792bc0e11fc5d5fe03b986ec1e81bbb89b73: Status 404 returned error can't find the container with id 5567ca82082345865da633f31106792bc0e11fc5d5fe03b986ec1e81bbb89b73
Oct 30 07:09:59.238125 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:59.238096    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:09:59.238227 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:59.238154    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:09:59.238227 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:59.238179    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:09:59.372875 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:59.372732    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq" event={"ID":"6fd8ca77-b283-4d93-87e7-640397564926","Type":"ContainerStarted","Data":"5cf20a309dcf4192b17baccb1b3a184451f1bdb69d4d9e8cba8b494512feda25"}
Oct 30 07:09:59.372875 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:59.372830    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq" event={"ID":"6fd8ca77-b283-4d93-87e7-640397564926","Type":"ContainerStarted","Data":"5567ca82082345865da633f31106792bc0e11fc5d5fe03b986ec1e81bbb89b73"}
Oct 30 07:09:59.422871 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:59.422789    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq" podStartSLOduration=1.422775488 podStartE2EDuration="1.422775488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:09:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:09:59.421273906 +0000 UTC m=+758.030680894" watchObservedRunningTime="2026-10-30 07:09:59.422775488 +0000 UTC m=+758.032182457"
Oct 30 07:09:59.922767 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:09:59.922723    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq"
Oct 30 07:10:00.924436 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:00.924398    6224 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 30 07:10:00.924963 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:00.924468    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq" podUID="6fd8ca77-b283-4d93-87e7-640397564926" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 30 07:10:01.855271 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:10:01.855228    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:10:01.923422 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:01.923389    6224 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 30 07:10:01.923583 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:01.923452    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq" podUID="6fd8ca77-b283-4d93-87e7-640397564926" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 30 07:10:02.923858 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:02.923824    6224 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 30 07:10:02.924246 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:02.923888    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq" podUID="6fd8ca77-b283-4d93-87e7-640397564926" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 30 07:10:03.923297 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:03.923259    6224 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 30 07:10:03.923475 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:03.923325    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq" podUID="6fd8ca77-b283-4d93-87e7-640397564926" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 30 07:10:04.923628 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:04.923590    6224 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 30 07:10:04.924036 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:04.923648    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq" podUID="6fd8ca77-b283-4d93-87e7-640397564926" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 30 07:10:05.924079 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:05.924037    6224 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 30 07:10:05.924479 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:05.924109    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq" podUID="6fd8ca77-b283-4d93-87e7-640397564926" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 30 07:10:06.927623 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:06.927586    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq"
Oct 30 07:10:06.928046 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:06.927838    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq"
Oct 30 07:10:06.928638 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:06.928619    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq"
Oct 30 07:10:08.058795 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:08.058764    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:10:13.026068 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:13.026032    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm"]
Oct 30 07:10:13.030045 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:13.030024    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm"
Oct 30 07:10:13.062714 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:13.062689    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm"]
Oct 30 07:10:13.150023 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:13.149992    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b5884ad-a370-40c8-89f5-6f260bbf234a-istio-envoy\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm\" (UID: \"5b5884ad-a370-40c8-89f5-6f260bbf234a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm"
Oct 30 07:10:13.150185 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:13.150033    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b5884ad-a370-40c8-89f5-6f260bbf234a-istio-token\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm\" (UID: \"5b5884ad-a370-40c8-89f5-6f260bbf234a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm"
Oct 30 07:10:13.150185 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:13.150053    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b5884ad-a370-40c8-89f5-6f260bbf234a-credential-socket\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm\" (UID: \"5b5884ad-a370-40c8-89f5-6f260bbf234a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm"
Oct 30 07:10:13.150259 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:13.150184    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjt2h\" (UniqueName: \"kubernetes.io/projected/5b5884ad-a370-40c8-89f5-6f260bbf234a-kube-api-access-xjt2h\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm\" (UID: \"5b5884ad-a370-40c8-89f5-6f260bbf234a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm"
Oct 30 07:10:13.150297 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:13.150280    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b5884ad-a370-40c8-89f5-6f260bbf234a-workload-socket\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm\" (UID: \"5b5884ad-a370-40c8-89f5-6f260bbf234a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm"
Oct 30 07:10:13.150339 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:13.150325    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b5884ad-a370-40c8-89f5-6f260bbf234a-istio-data\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm\" (UID: \"5b5884ad-a370-40c8-89f5-6f260bbf234a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm"
Oct 30 07:10:13.150432 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:13.150420    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b5884ad-a370-40c8-89f5-6f260bbf234a-workload-certs\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm\" (UID: \"5b5884ad-a370-40c8-89f5-6f260bbf234a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm"
Oct 30 07:10:13.150469 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:13.150444    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b5884ad-a370-40c8-89f5-6f260bbf234a-istio-podinfo\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm\" (UID: \"5b5884ad-a370-40c8-89f5-6f260bbf234a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm"
Oct 30 07:10:13.150469 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:13.150462    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b5884ad-a370-40c8-89f5-6f260bbf234a-istiod-ca-cert\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm\" (UID: \"5b5884ad-a370-40c8-89f5-6f260bbf234a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm"
Oct 30 07:10:13.251512 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:13.251486    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b5884ad-a370-40c8-89f5-6f260bbf234a-workload-certs\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm\" (UID: \"5b5884ad-a370-40c8-89f5-6f260bbf234a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm"
Oct 30 07:10:13.251512 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:13.251514    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b5884ad-a370-40c8-89f5-6f260bbf234a-istio-podinfo\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm\" (UID: \"5b5884ad-a370-40c8-89f5-6f260bbf234a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm"
Oct 30 07:10:13.251718 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:13.251537    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b5884ad-a370-40c8-89f5-6f260bbf234a-istiod-ca-cert\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm\" (UID: \"5b5884ad-a370-40c8-89f5-6f260bbf234a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm"
Oct 30 07:10:13.251718 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:13.251583    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b5884ad-a370-40c8-89f5-6f260bbf234a-istio-envoy\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm\" (UID: \"5b5884ad-a370-40c8-89f5-6f260bbf234a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm"
Oct 30 07:10:13.251718 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:13.251616    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b5884ad-a370-40c8-89f5-6f260bbf234a-istio-token\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm\" (UID: \"5b5884ad-a370-40c8-89f5-6f260bbf234a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm"
Oct 30 07:10:13.251718 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:13.251643    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b5884ad-a370-40c8-89f5-6f260bbf234a-credential-socket\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm\" (UID: \"5b5884ad-a370-40c8-89f5-6f260bbf234a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm"
Oct 30 07:10:13.251718 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:13.251689    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjt2h\" (UniqueName: \"kubernetes.io/projected/5b5884ad-a370-40c8-89f5-6f260bbf234a-kube-api-access-xjt2h\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm\" (UID: \"5b5884ad-a370-40c8-89f5-6f260bbf234a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm"
Oct 30 07:10:13.251994 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:13.251911    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b5884ad-a370-40c8-89f5-6f260bbf234a-workload-socket\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm\" (UID: \"5b5884ad-a370-40c8-89f5-6f260bbf234a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm"
Oct 30 07:10:13.251994 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:13.251982    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b5884ad-a370-40c8-89f5-6f260bbf234a-workload-certs\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm\" (UID: \"5b5884ad-a370-40c8-89f5-6f260bbf234a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm"
Oct 30 07:10:13.252087 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:13.251983    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b5884ad-a370-40c8-89f5-6f260bbf234a-istio-data\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm\" (UID: \"5b5884ad-a370-40c8-89f5-6f260bbf234a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm"
Oct 30 07:10:13.252159 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:13.252131    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b5884ad-a370-40c8-89f5-6f260bbf234a-credential-socket\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm\" (UID: \"5b5884ad-a370-40c8-89f5-6f260bbf234a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm"
Oct 30 07:10:13.252278 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:13.252256    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b5884ad-a370-40c8-89f5-6f260bbf234a-workload-socket\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm\" (UID: \"5b5884ad-a370-40c8-89f5-6f260bbf234a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm"
Oct 30 07:10:13.252394 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:13.252370    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b5884ad-a370-40c8-89f5-6f260bbf234a-istio-data\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm\" (UID: \"5b5884ad-a370-40c8-89f5-6f260bbf234a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm"
Oct 30 07:10:13.252538 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:13.252520    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b5884ad-a370-40c8-89f5-6f260bbf234a-istiod-ca-cert\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm\" (UID: \"5b5884ad-a370-40c8-89f5-6f260bbf234a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm"
Oct 30 07:10:13.253880 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:13.253857    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b5884ad-a370-40c8-89f5-6f260bbf234a-istio-envoy\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm\" (UID: \"5b5884ad-a370-40c8-89f5-6f260bbf234a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm"
Oct 30 07:10:13.254088 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:13.254072    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b5884ad-a370-40c8-89f5-6f260bbf234a-istio-podinfo\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm\" (UID: \"5b5884ad-a370-40c8-89f5-6f260bbf234a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm"
Oct 30 07:10:13.276347 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:13.276298    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b5884ad-a370-40c8-89f5-6f260bbf234a-istio-token\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm\" (UID: \"5b5884ad-a370-40c8-89f5-6f260bbf234a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm"
Oct 30 07:10:13.276427 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:13.276393    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjt2h\" (UniqueName: \"kubernetes.io/projected/5b5884ad-a370-40c8-89f5-6f260bbf234a-kube-api-access-xjt2h\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm\" (UID: \"5b5884ad-a370-40c8-89f5-6f260bbf234a\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm"
Oct 30 07:10:13.345490 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:13.345462    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm"
Oct 30 07:10:13.659503 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:13.659471    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm"]
Oct 30 07:10:13.660509 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:10:13.660484    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b5884ad_a370_40c8_89f5_6f260bbf234a.slice/crio-a4954a9ae47c36e411c3220113a4537816e055aea72f4eee562c5a8944d43255 WatchSource:0}: Error finding container a4954a9ae47c36e411c3220113a4537816e055aea72f4eee562c5a8944d43255: Status 404 returned error can't find the container with id a4954a9ae47c36e411c3220113a4537816e055aea72f4eee562c5a8944d43255
Oct 30 07:10:13.662682 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:13.662648    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:10:13.662769 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:13.662710    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:10:13.662769 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:13.662740    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:10:14.478783 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:14.478739    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm" event={"ID":"5b5884ad-a370-40c8-89f5-6f260bbf234a","Type":"ContainerStarted","Data":"49b36688e9b6f9c2939fdb838afb3b74f4a069e6a0c550933a7d65189a962bf7"}
Oct 30 07:10:14.479216 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:14.478801    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm" event={"ID":"5b5884ad-a370-40c8-89f5-6f260bbf234a","Type":"ContainerStarted","Data":"a4954a9ae47c36e411c3220113a4537816e055aea72f4eee562c5a8944d43255"}
Oct 30 07:10:14.544441 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:14.544372    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm" podStartSLOduration=2.54435649 podStartE2EDuration="2.54435649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:10:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:10:14.54213901 +0000 UTC m=+773.151545981" watchObservedRunningTime="2026-10-30 07:10:14.54435649 +0000 UTC m=+773.153763459"
Oct 30 07:10:15.346494 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:15.346461    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm"
Oct 30 07:10:15.348230 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:15.348203    6224 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 30 07:10:15.348356 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:15.348253    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm" podUID="5b5884ad-a370-40c8-89f5-6f260bbf234a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 30 07:10:16.346321 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:16.346282    6224 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 30 07:10:16.346713 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:16.346344    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm" podUID="5b5884ad-a370-40c8-89f5-6f260bbf234a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 30 07:10:17.346215 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:17.346163    6224 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 30 07:10:17.346433 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:17.346243    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm" podUID="5b5884ad-a370-40c8-89f5-6f260bbf234a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 30 07:10:18.346727 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:18.346688    6224 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 30 07:10:18.347353 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:18.347313    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm" podUID="5b5884ad-a370-40c8-89f5-6f260bbf234a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 30 07:10:19.346758 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:19.346725    6224 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 30 07:10:19.347193 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:19.346778    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm" podUID="5b5884ad-a370-40c8-89f5-6f260bbf234a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 30 07:10:20.346261 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:20.346223    6224 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 30 07:10:20.346428 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:20.346281    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm" podUID="5b5884ad-a370-40c8-89f5-6f260bbf234a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 30 07:10:22.145743 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:22.145697    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm"
Oct 30 07:10:22.146240 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:22.146173    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm"
Oct 30 07:10:22.147470 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:22.147451    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm"
Oct 30 07:10:22.657351 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:22.657304    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wkjb2"]
Oct 30 07:10:22.657578 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:22.657531    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wkjb2" podUID="ac450c4b-8d00-4efb-9023-1681c843a993" containerName="istio-proxy" containerID="cri-o://35a9ccadaaf89a0a11fd2aef81e111c7f460e4df5c346f97faa4edd7567dce01" gracePeriod=30
Oct 30 07:10:27.911112 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:27.911083    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wkjb2"
Oct 30 07:10:27.944129 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:27.944058    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac450c4b-8d00-4efb-9023-1681c843a993-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac450c4b-8d00-4efb-9023-1681c843a993-workload-certs\") pod \"ac450c4b-8d00-4efb-9023-1681c843a993\" (UID: \"ac450c4b-8d00-4efb-9023-1681c843a993\") "
Oct 30 07:10:27.944129 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:27.944109    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac450c4b-8d00-4efb-9023-1681c843a993-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac450c4b-8d00-4efb-9023-1681c843a993-istio-data\") pod \"ac450c4b-8d00-4efb-9023-1681c843a993\" (UID: \"ac450c4b-8d00-4efb-9023-1681c843a993\") "
Oct 30 07:10:27.944338 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:27.944134    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac450c4b-8d00-4efb-9023-1681c843a993-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac450c4b-8d00-4efb-9023-1681c843a993-workload-socket\") pod \"ac450c4b-8d00-4efb-9023-1681c843a993\" (UID: \"ac450c4b-8d00-4efb-9023-1681c843a993\") "
Oct 30 07:10:27.944338 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:27.944206    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac450c4b-8d00-4efb-9023-1681c843a993-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac450c4b-8d00-4efb-9023-1681c843a993-istiod-ca-cert\") pod \"ac450c4b-8d00-4efb-9023-1681c843a993\" (UID: \"ac450c4b-8d00-4efb-9023-1681c843a993\") "
Oct 30 07:10:27.944338 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:27.944264    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac450c4b-8d00-4efb-9023-1681c843a993-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac450c4b-8d00-4efb-9023-1681c843a993-credential-socket\") pod \"ac450c4b-8d00-4efb-9023-1681c843a993\" (UID: \"ac450c4b-8d00-4efb-9023-1681c843a993\") "
Oct 30 07:10:27.944515 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:27.944316    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac450c4b-8d00-4efb-9023-1681c843a993-workload-certs" pod "ac450c4b-8d00-4efb-9023-1681c843a993" (UID: "ac450c4b-8d00-4efb-9023-1681c843a993"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:10:27.944515 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:27.944370    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac450c4b-8d00-4efb-9023-1681c843a993-kube-api-access-h8kdz\" (UniqueName: \"kubernetes.io/projected/ac450c4b-8d00-4efb-9023-1681c843a993-kube-api-access-h8kdz\") pod \"ac450c4b-8d00-4efb-9023-1681c843a993\" (UID: \"ac450c4b-8d00-4efb-9023-1681c843a993\") "
Oct 30 07:10:27.944515 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:27.944388    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac450c4b-8d00-4efb-9023-1681c843a993-istio-data" pod "ac450c4b-8d00-4efb-9023-1681c843a993" (UID: "ac450c4b-8d00-4efb-9023-1681c843a993"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:10:27.944515 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:27.944414    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac450c4b-8d00-4efb-9023-1681c843a993-istio-token\" (UniqueName: \"kubernetes.io/projected/ac450c4b-8d00-4efb-9023-1681c843a993-istio-token\") pod \"ac450c4b-8d00-4efb-9023-1681c843a993\" (UID: \"ac450c4b-8d00-4efb-9023-1681c843a993\") "
Oct 30 07:10:27.944515 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:27.944452    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac450c4b-8d00-4efb-9023-1681c843a993-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac450c4b-8d00-4efb-9023-1681c843a993-istio-envoy\") pod \"ac450c4b-8d00-4efb-9023-1681c843a993\" (UID: \"ac450c4b-8d00-4efb-9023-1681c843a993\") "
Oct 30 07:10:27.944515 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:27.944460    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac450c4b-8d00-4efb-9023-1681c843a993-workload-socket" pod "ac450c4b-8d00-4efb-9023-1681c843a993" (UID: "ac450c4b-8d00-4efb-9023-1681c843a993"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:10:27.944515 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:27.944504    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac450c4b-8d00-4efb-9023-1681c843a993-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac450c4b-8d00-4efb-9023-1681c843a993-istio-podinfo\") pod \"ac450c4b-8d00-4efb-9023-1681c843a993\" (UID: \"ac450c4b-8d00-4efb-9023-1681c843a993\") "
Oct 30 07:10:27.944920 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:27.944532    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac450c4b-8d00-4efb-9023-1681c843a993-credential-socket" pod "ac450c4b-8d00-4efb-9023-1681c843a993" (UID: "ac450c4b-8d00-4efb-9023-1681c843a993"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:10:27.944920 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:27.944702    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac450c4b-8d00-4efb-9023-1681c843a993-istiod-ca-cert" pod "ac450c4b-8d00-4efb-9023-1681c843a993" (UID: "ac450c4b-8d00-4efb-9023-1681c843a993"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:10:27.945038 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:27.944997    6224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac450c4b-8d00-4efb-9023-1681c843a993-workload-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:10:27.945038 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:27.945031    6224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac450c4b-8d00-4efb-9023-1681c843a993-istio-data\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:10:27.945148 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:27.945044    6224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac450c4b-8d00-4efb-9023-1681c843a993-workload-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:10:27.945148 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:27.945058    6224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac450c4b-8d00-4efb-9023-1681c843a993-istiod-ca-cert\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:10:27.945148 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:27.945073    6224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac450c4b-8d00-4efb-9023-1681c843a993-credential-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:10:27.946696 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:27.946673    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac450c4b-8d00-4efb-9023-1681c843a993-kube-api-access-h8kdz" pod "ac450c4b-8d00-4efb-9023-1681c843a993" (UID: "ac450c4b-8d00-4efb-9023-1681c843a993"). InnerVolumeSpecName "kube-api-access-h8kdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:10:27.946786 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:27.946755    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac450c4b-8d00-4efb-9023-1681c843a993-istio-token" pod "ac450c4b-8d00-4efb-9023-1681c843a993" (UID: "ac450c4b-8d00-4efb-9023-1681c843a993"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:10:27.947016 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:27.946989    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac450c4b-8d00-4efb-9023-1681c843a993-istio-envoy" pod "ac450c4b-8d00-4efb-9023-1681c843a993" (UID: "ac450c4b-8d00-4efb-9023-1681c843a993"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:10:27.947105 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:27.946997    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac450c4b-8d00-4efb-9023-1681c843a993-istio-podinfo" pod "ac450c4b-8d00-4efb-9023-1681c843a993" (UID: "ac450c4b-8d00-4efb-9023-1681c843a993"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:10:28.046017 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:28.045982    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8kdz\" (UniqueName: \"kubernetes.io/projected/ac450c4b-8d00-4efb-9023-1681c843a993-kube-api-access-h8kdz\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:10:28.046017 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:28.046011    6224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac450c4b-8d00-4efb-9023-1681c843a993-istio-token\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:10:28.046017 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:28.046021    6224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac450c4b-8d00-4efb-9023-1681c843a993-istio-envoy\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:10:28.046246 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:28.046028    6224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac450c4b-8d00-4efb-9023-1681c843a993-istio-podinfo\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:10:28.440951 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:28.440916    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:10:28.573653 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:28.573613    6224 generic.go:356] "Generic (PLEG): container finished" podID="ac450c4b-8d00-4efb-9023-1681c843a993" containerID="35a9ccadaaf89a0a11fd2aef81e111c7f460e4df5c346f97faa4edd7567dce01" exitCode=0
Oct 30 07:10:28.573818 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:28.573663    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wkjb2"
Oct 30 07:10:28.573818 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:28.573697    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wkjb2" event={"ID":"ac450c4b-8d00-4efb-9023-1681c843a993","Type":"ContainerDied","Data":"35a9ccadaaf89a0a11fd2aef81e111c7f460e4df5c346f97faa4edd7567dce01"}
Oct 30 07:10:28.573818 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:28.573750    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wkjb2" event={"ID":"ac450c4b-8d00-4efb-9023-1681c843a993","Type":"ContainerDied","Data":"98448383271d45d9f7afdbec2aad39b3b2f61457747dfda2ce31116ccb409e2c"}
Oct 30 07:10:28.573818 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:28.573773    6224 scope.go:122] "RemoveContainer" containerID="35a9ccadaaf89a0a11fd2aef81e111c7f460e4df5c346f97faa4edd7567dce01"
Oct 30 07:10:28.583943 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:28.583926    6224 scope.go:122] "RemoveContainer" containerID="35a9ccadaaf89a0a11fd2aef81e111c7f460e4df5c346f97faa4edd7567dce01"
Oct 30 07:10:28.584218 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:10:28.584182    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35a9ccadaaf89a0a11fd2aef81e111c7f460e4df5c346f97faa4edd7567dce01\": container with ID starting with 35a9ccadaaf89a0a11fd2aef81e111c7f460e4df5c346f97faa4edd7567dce01 not found: ID does not exist" containerID="35a9ccadaaf89a0a11fd2aef81e111c7f460e4df5c346f97faa4edd7567dce01"
Oct 30 07:10:28.584274 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:28.584226    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35a9ccadaaf89a0a11fd2aef81e111c7f460e4df5c346f97faa4edd7567dce01"} err="failed to get container status \"35a9ccadaaf89a0a11fd2aef81e111c7f460e4df5c346f97faa4edd7567dce01\": rpc error: code = NotFound desc = could not find container \"35a9ccadaaf89a0a11fd2aef81e111c7f460e4df5c346f97faa4edd7567dce01\": container with ID starting with 35a9ccadaaf89a0a11fd2aef81e111c7f460e4df5c346f97faa4edd7567dce01 not found: ID does not exist"
Oct 30 07:10:28.632503 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:28.632475    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wkjb2"]
Oct 30 07:10:28.645706 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:28.645685    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wkjb2"]
Oct 30 07:10:29.522443 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:29.522407    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac450c4b-8d00-4efb-9023-1681c843a993" path="/var/lib/kubelet/pods/ac450c4b-8d00-4efb-9023-1681c843a993/volumes"
Oct 30 07:10:31.856769 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:10:31.856732    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:10:39.207840 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:39.207792    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2"]
Oct 30 07:10:39.208294 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:39.208279    6224 cpu_manager.go:414] "containerMap: removing container" podUID="ac450c4b-8d00-4efb-9023-1681c843a993" containerName="istio-proxy"
Oct 30 07:10:39.208337 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:39.208295    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac450c4b-8d00-4efb-9023-1681c843a993" containerName="istio-proxy"
Oct 30 07:10:39.208386 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:39.208376    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac450c4b-8d00-4efb-9023-1681c843a993" containerName="istio-proxy"
Oct 30 07:10:39.211466 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:39.211447    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2"
Oct 30 07:10:39.245293 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:39.245261    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2"]
Oct 30 07:10:39.379251 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:39.379214    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1871ac3-4279-4cd5-8fc2-5891514a0dff-workload-certs\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2\" (UID: \"e1871ac3-4279-4cd5-8fc2-5891514a0dff\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2"
Oct 30 07:10:39.379412 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:39.379300    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1871ac3-4279-4cd5-8fc2-5891514a0dff-istio-envoy\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2\" (UID: \"e1871ac3-4279-4cd5-8fc2-5891514a0dff\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2"
Oct 30 07:10:39.379412 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:39.379353    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1871ac3-4279-4cd5-8fc2-5891514a0dff-istio-token\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2\" (UID: \"e1871ac3-4279-4cd5-8fc2-5891514a0dff\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2"
Oct 30 07:10:39.379412 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:39.379407    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1871ac3-4279-4cd5-8fc2-5891514a0dff-istiod-ca-cert\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2\" (UID: \"e1871ac3-4279-4cd5-8fc2-5891514a0dff\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2"
Oct 30 07:10:39.379540 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:39.379525    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1871ac3-4279-4cd5-8fc2-5891514a0dff-credential-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2\" (UID: \"e1871ac3-4279-4cd5-8fc2-5891514a0dff\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2"
Oct 30 07:10:39.379585 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:39.379573    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1871ac3-4279-4cd5-8fc2-5891514a0dff-workload-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2\" (UID: \"e1871ac3-4279-4cd5-8fc2-5891514a0dff\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2"
Oct 30 07:10:39.379658 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:39.379640    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4m88\" (UniqueName: \"kubernetes.io/projected/e1871ac3-4279-4cd5-8fc2-5891514a0dff-kube-api-access-g4m88\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2\" (UID: \"e1871ac3-4279-4cd5-8fc2-5891514a0dff\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2"
Oct 30 07:10:39.379699 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:39.379674    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1871ac3-4279-4cd5-8fc2-5891514a0dff-istio-data\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2\" (UID: \"e1871ac3-4279-4cd5-8fc2-5891514a0dff\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2"
Oct 30 07:10:39.379699 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:39.379693    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1871ac3-4279-4cd5-8fc2-5891514a0dff-istio-podinfo\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2\" (UID: \"e1871ac3-4279-4cd5-8fc2-5891514a0dff\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2"
Oct 30 07:10:39.480320 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:39.480242    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1871ac3-4279-4cd5-8fc2-5891514a0dff-credential-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2\" (UID: \"e1871ac3-4279-4cd5-8fc2-5891514a0dff\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2"
Oct 30 07:10:39.480320 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:39.480319    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1871ac3-4279-4cd5-8fc2-5891514a0dff-workload-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2\" (UID: \"e1871ac3-4279-4cd5-8fc2-5891514a0dff\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2"
Oct 30 07:10:39.480535 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:39.480346    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4m88\" (UniqueName: \"kubernetes.io/projected/e1871ac3-4279-4cd5-8fc2-5891514a0dff-kube-api-access-g4m88\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2\" (UID: \"e1871ac3-4279-4cd5-8fc2-5891514a0dff\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2"
Oct 30 07:10:39.480535 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:39.480381    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1871ac3-4279-4cd5-8fc2-5891514a0dff-istio-data\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2\" (UID: \"e1871ac3-4279-4cd5-8fc2-5891514a0dff\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2"
Oct 30 07:10:39.480535 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:39.480436    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1871ac3-4279-4cd5-8fc2-5891514a0dff-istio-podinfo\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2\" (UID: \"e1871ac3-4279-4cd5-8fc2-5891514a0dff\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2"
Oct 30 07:10:39.480535 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:39.480490    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1871ac3-4279-4cd5-8fc2-5891514a0dff-workload-certs\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2\" (UID: \"e1871ac3-4279-4cd5-8fc2-5891514a0dff\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2"
Oct 30 07:10:39.480771 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:39.480535    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1871ac3-4279-4cd5-8fc2-5891514a0dff-istio-envoy\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2\" (UID: \"e1871ac3-4279-4cd5-8fc2-5891514a0dff\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2"
Oct 30 07:10:39.480771 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:39.480566    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1871ac3-4279-4cd5-8fc2-5891514a0dff-istio-token\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2\" (UID: \"e1871ac3-4279-4cd5-8fc2-5891514a0dff\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2"
Oct 30 07:10:39.480771 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:39.480593    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1871ac3-4279-4cd5-8fc2-5891514a0dff-istiod-ca-cert\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2\" (UID: \"e1871ac3-4279-4cd5-8fc2-5891514a0dff\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2"
Oct 30 07:10:39.480771 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:39.480682    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1871ac3-4279-4cd5-8fc2-5891514a0dff-credential-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2\" (UID: \"e1871ac3-4279-4cd5-8fc2-5891514a0dff\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2"
Oct 30 07:10:39.481024 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:39.480851    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1871ac3-4279-4cd5-8fc2-5891514a0dff-istio-data\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2\" (UID: \"e1871ac3-4279-4cd5-8fc2-5891514a0dff\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2"
Oct 30 07:10:39.481117 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:39.481086    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1871ac3-4279-4cd5-8fc2-5891514a0dff-workload-certs\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2\" (UID: \"e1871ac3-4279-4cd5-8fc2-5891514a0dff\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2"
Oct 30 07:10:39.481309 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:39.481286    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1871ac3-4279-4cd5-8fc2-5891514a0dff-workload-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2\" (UID: \"e1871ac3-4279-4cd5-8fc2-5891514a0dff\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2"
Oct 30 07:10:39.481446 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:39.481423    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1871ac3-4279-4cd5-8fc2-5891514a0dff-istiod-ca-cert\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2\" (UID: \"e1871ac3-4279-4cd5-8fc2-5891514a0dff\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2"
Oct 30 07:10:39.483130 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:39.483096    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1871ac3-4279-4cd5-8fc2-5891514a0dff-istio-envoy\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2\" (UID: \"e1871ac3-4279-4cd5-8fc2-5891514a0dff\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2"
Oct 30 07:10:39.483235 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:39.483213    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1871ac3-4279-4cd5-8fc2-5891514a0dff-istio-podinfo\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2\" (UID: \"e1871ac3-4279-4cd5-8fc2-5891514a0dff\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2"
Oct 30 07:10:39.505643 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:39.505619    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1871ac3-4279-4cd5-8fc2-5891514a0dff-istio-token\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2\" (UID: \"e1871ac3-4279-4cd5-8fc2-5891514a0dff\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2"
Oct 30 07:10:39.505767 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:39.505747    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4m88\" (UniqueName: \"kubernetes.io/projected/e1871ac3-4279-4cd5-8fc2-5891514a0dff-kube-api-access-g4m88\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2\" (UID: \"e1871ac3-4279-4cd5-8fc2-5891514a0dff\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2"
Oct 30 07:10:39.527670 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:39.527652    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2"
Oct 30 07:10:39.841867 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:10:39.841832    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1871ac3_4279_4cd5_8fc2_5891514a0dff.slice/crio-4f0a4a8282bad04c39bc121b8a036d821cb4c5b07e9adbc42bb5149a1a85d129 WatchSource:0}: Error finding container 4f0a4a8282bad04c39bc121b8a036d821cb4c5b07e9adbc42bb5149a1a85d129: Status 404 returned error can't find the container with id 4f0a4a8282bad04c39bc121b8a036d821cb4c5b07e9adbc42bb5149a1a85d129
Oct 30 07:10:39.842141 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:39.842115    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2"]
Oct 30 07:10:39.844313 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:39.844280    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:10:39.844419 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:39.844359    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:10:39.844419 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:39.844405    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:10:40.643295 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:40.643259    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2" event={"ID":"e1871ac3-4279-4cd5-8fc2-5891514a0dff","Type":"ContainerStarted","Data":"0115fda8f7fa3c133561ba9c4b1fe1d6cbbff866a9cd8d71d13b0c17d8c083df"}
Oct 30 07:10:40.643696 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:40.643314    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2" event={"ID":"e1871ac3-4279-4cd5-8fc2-5891514a0dff","Type":"ContainerStarted","Data":"4f0a4a8282bad04c39bc121b8a036d821cb4c5b07e9adbc42bb5149a1a85d129"}
Oct 30 07:10:40.720139 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:40.720089    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2" podStartSLOduration=1.720075458 podStartE2EDuration="1.720075458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:10:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:10:40.718603993 +0000 UTC m=+799.328010962" watchObservedRunningTime="2026-10-30 07:10:40.720075458 +0000 UTC m=+799.329482418"
Oct 30 07:10:41.527876 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:41.527840    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2"
Oct 30 07:10:41.529082 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:41.529059    6224 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 30 07:10:41.529209 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:41.529100    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2" podUID="e1871ac3-4279-4cd5-8fc2-5891514a0dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 30 07:10:42.528296 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:42.528262    6224 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 30 07:10:42.528698 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:42.528314    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2" podUID="e1871ac3-4279-4cd5-8fc2-5891514a0dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 30 07:10:43.528912 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:43.528879    6224 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 30 07:10:43.529294 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:43.528940    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2" podUID="e1871ac3-4279-4cd5-8fc2-5891514a0dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 30 07:10:44.528302 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:44.528268    6224 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 30 07:10:44.528468 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:44.528317    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2" podUID="e1871ac3-4279-4cd5-8fc2-5891514a0dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 30 07:10:45.528467 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:45.528417    6224 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 30 07:10:45.528974 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:45.528509    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2" podUID="e1871ac3-4279-4cd5-8fc2-5891514a0dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 30 07:10:46.530241 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:46.530193    6224 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 30 07:10:46.530729 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:46.530263    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2" podUID="e1871ac3-4279-4cd5-8fc2-5891514a0dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 30 07:10:47.533448 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:47.533416    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2"
Oct 30 07:10:47.533950 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:47.533817    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2"
Oct 30 07:10:47.534465 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:47.534447    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2"
Oct 30 07:10:54.266951 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:54.266916    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj"]
Oct 30 07:10:54.270790 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:54.270773    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj"
Oct 30 07:10:54.294359 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:54.294327    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj"]
Oct 30 07:10:54.441420 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:54.441379    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12729552-3e12-4b64-8779-a1584734c16e-istio-data\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj\" (UID: \"12729552-3e12-4b64-8779-a1584734c16e\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj"
Oct 30 07:10:54.441420 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:54.441417    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dhbr\" (UniqueName: \"kubernetes.io/projected/12729552-3e12-4b64-8779-a1584734c16e-kube-api-access-5dhbr\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj\" (UID: \"12729552-3e12-4b64-8779-a1584734c16e\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj"
Oct 30 07:10:54.441664 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:54.441457    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12729552-3e12-4b64-8779-a1584734c16e-workload-certs\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj\" (UID: \"12729552-3e12-4b64-8779-a1584734c16e\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj"
Oct 30 07:10:54.441664 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:54.441571    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12729552-3e12-4b64-8779-a1584734c16e-istio-envoy\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj\" (UID: \"12729552-3e12-4b64-8779-a1584734c16e\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj"
Oct 30 07:10:54.441664 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:54.441628    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12729552-3e12-4b64-8779-a1584734c16e-istio-token\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj\" (UID: \"12729552-3e12-4b64-8779-a1584734c16e\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj"
Oct 30 07:10:54.441664 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:54.441650    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12729552-3e12-4b64-8779-a1584734c16e-istio-podinfo\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj\" (UID: \"12729552-3e12-4b64-8779-a1584734c16e\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj"
Oct 30 07:10:54.441887 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:54.441752    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12729552-3e12-4b64-8779-a1584734c16e-workload-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj\" (UID: \"12729552-3e12-4b64-8779-a1584734c16e\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj"
Oct 30 07:10:54.441887 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:54.441770    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12729552-3e12-4b64-8779-a1584734c16e-istiod-ca-cert\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj\" (UID: \"12729552-3e12-4b64-8779-a1584734c16e\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj"
Oct 30 07:10:54.441887 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:54.441839    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12729552-3e12-4b64-8779-a1584734c16e-credential-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj\" (UID: \"12729552-3e12-4b64-8779-a1584734c16e\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj"
Oct 30 07:10:54.543086 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:54.543044    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12729552-3e12-4b64-8779-a1584734c16e-istio-data\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj\" (UID: \"12729552-3e12-4b64-8779-a1584734c16e\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj"
Oct 30 07:10:54.543272 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:54.543092    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dhbr\" (UniqueName: \"kubernetes.io/projected/12729552-3e12-4b64-8779-a1584734c16e-kube-api-access-5dhbr\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj\" (UID: \"12729552-3e12-4b64-8779-a1584734c16e\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj"
Oct 30 07:10:54.543272 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:54.543145    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12729552-3e12-4b64-8779-a1584734c16e-workload-certs\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj\" (UID: \"12729552-3e12-4b64-8779-a1584734c16e\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj"
Oct 30 07:10:54.543272 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:54.543184    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12729552-3e12-4b64-8779-a1584734c16e-istio-envoy\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj\" (UID: \"12729552-3e12-4b64-8779-a1584734c16e\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj"
Oct 30 07:10:54.543272 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:54.543206    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12729552-3e12-4b64-8779-a1584734c16e-istio-token\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj\" (UID: \"12729552-3e12-4b64-8779-a1584734c16e\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj"
Oct 30 07:10:54.543272 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:54.543232    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12729552-3e12-4b64-8779-a1584734c16e-istio-podinfo\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj\" (UID: \"12729552-3e12-4b64-8779-a1584734c16e\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj"
Oct 30 07:10:54.543530 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:54.543330    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12729552-3e12-4b64-8779-a1584734c16e-workload-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj\" (UID: \"12729552-3e12-4b64-8779-a1584734c16e\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj"
Oct 30 07:10:54.543530 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:54.543360    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12729552-3e12-4b64-8779-a1584734c16e-istiod-ca-cert\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj\" (UID: \"12729552-3e12-4b64-8779-a1584734c16e\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj"
Oct 30 07:10:54.543530 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:54.543391    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12729552-3e12-4b64-8779-a1584734c16e-credential-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj\" (UID: \"12729552-3e12-4b64-8779-a1584734c16e\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj"
Oct 30 07:10:54.543530 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:54.543499    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12729552-3e12-4b64-8779-a1584734c16e-istio-data\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj\" (UID: \"12729552-3e12-4b64-8779-a1584734c16e\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj"
Oct 30 07:10:54.543720 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:54.543562    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12729552-3e12-4b64-8779-a1584734c16e-workload-certs\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj\" (UID: \"12729552-3e12-4b64-8779-a1584734c16e\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj"
Oct 30 07:10:54.543720 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:54.543681    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12729552-3e12-4b64-8779-a1584734c16e-workload-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj\" (UID: \"12729552-3e12-4b64-8779-a1584734c16e\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj"
Oct 30 07:10:54.543878 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:54.543854    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12729552-3e12-4b64-8779-a1584734c16e-credential-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj\" (UID: \"12729552-3e12-4b64-8779-a1584734c16e\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj"
Oct 30 07:10:54.544179 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:54.544158    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12729552-3e12-4b64-8779-a1584734c16e-istiod-ca-cert\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj\" (UID: \"12729552-3e12-4b64-8779-a1584734c16e\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj"
Oct 30 07:10:54.545513 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:54.545495    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12729552-3e12-4b64-8779-a1584734c16e-istio-envoy\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj\" (UID: \"12729552-3e12-4b64-8779-a1584734c16e\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj"
Oct 30 07:10:54.545780 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:54.545762    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12729552-3e12-4b64-8779-a1584734c16e-istio-podinfo\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj\" (UID: \"12729552-3e12-4b64-8779-a1584734c16e\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj"
Oct 30 07:10:54.568913 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:54.568883    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12729552-3e12-4b64-8779-a1584734c16e-istio-token\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj\" (UID: \"12729552-3e12-4b64-8779-a1584734c16e\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj"
Oct 30 07:10:54.569031 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:54.569010    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dhbr\" (UniqueName: \"kubernetes.io/projected/12729552-3e12-4b64-8779-a1584734c16e-kube-api-access-5dhbr\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj\" (UID: \"12729552-3e12-4b64-8779-a1584734c16e\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj"
Oct 30 07:10:54.586894 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:54.586872    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj"
Oct 30 07:10:54.959727 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:10:54.959696    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12729552_3e12_4b64_8779_a1584734c16e.slice/crio-f1407b4cf0a8f71b251e66d4888bb178c4c57851385169730ba3f3d86068c855 WatchSource:0}: Error finding container f1407b4cf0a8f71b251e66d4888bb178c4c57851385169730ba3f3d86068c855: Status 404 returned error can't find the container with id f1407b4cf0a8f71b251e66d4888bb178c4c57851385169730ba3f3d86068c855
Oct 30 07:10:54.960700 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:54.960677    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj"]
Oct 30 07:10:54.962420 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:54.962386    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:10:54.962526 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:54.962489    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:10:54.962588 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:54.962533    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:10:55.055733 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:55.055694    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-06b26e/distinct-w0-06b26e-kserve-545cf57898-2fhhd"]
Oct 30 07:10:55.060041 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:55.059690    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-06b26e/distinct-w0-06b26e-kserve-545cf57898-2fhhd"
Oct 30 07:10:55.088602 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:55.088572    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-06b26e\"/\"distinct-w0-06b26e-kserve-self-signed-certs\""
Oct 30 07:10:55.091946 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:55.091922    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-06b26e/distinct-w0-06b26e-kserve-545cf57898-2fhhd"]
Oct 30 07:10:55.150661 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:55.150613    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71294bfa-ac2c-40b7-b889-0527daa07f63-home\") pod \"distinct-w0-06b26e-kserve-545cf57898-2fhhd\" (UID: \"71294bfa-ac2c-40b7-b889-0527daa07f63\") " pod="e2e-models-e2e-worker-w0-06b26e/distinct-w0-06b26e-kserve-545cf57898-2fhhd"
Oct 30 07:10:55.150661 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:55.150650    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71294bfa-ac2c-40b7-b889-0527daa07f63-tls-certs\") pod \"distinct-w0-06b26e-kserve-545cf57898-2fhhd\" (UID: \"71294bfa-ac2c-40b7-b889-0527daa07f63\") " pod="e2e-models-e2e-worker-w0-06b26e/distinct-w0-06b26e-kserve-545cf57898-2fhhd"
Oct 30 07:10:55.150946 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:55.150751    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71294bfa-ac2c-40b7-b889-0527daa07f63-model-cache\") pod \"distinct-w0-06b26e-kserve-545cf57898-2fhhd\" (UID: \"71294bfa-ac2c-40b7-b889-0527daa07f63\") " pod="e2e-models-e2e-worker-w0-06b26e/distinct-w0-06b26e-kserve-545cf57898-2fhhd"
Oct 30 07:10:55.150946 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:55.150870    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dscrd\" (UniqueName: \"kubernetes.io/projected/71294bfa-ac2c-40b7-b889-0527daa07f63-kube-api-access-dscrd\") pod \"distinct-w0-06b26e-kserve-545cf57898-2fhhd\" (UID: \"71294bfa-ac2c-40b7-b889-0527daa07f63\") " pod="e2e-models-e2e-worker-w0-06b26e/distinct-w0-06b26e-kserve-545cf57898-2fhhd"
Oct 30 07:10:55.150946 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:55.150924    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71294bfa-ac2c-40b7-b889-0527daa07f63-tmp-dir\") pod \"distinct-w0-06b26e-kserve-545cf57898-2fhhd\" (UID: \"71294bfa-ac2c-40b7-b889-0527daa07f63\") " pod="e2e-models-e2e-worker-w0-06b26e/distinct-w0-06b26e-kserve-545cf57898-2fhhd"
Oct 30 07:10:55.151110 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:55.151030    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71294bfa-ac2c-40b7-b889-0527daa07f63-dshm\") pod \"distinct-w0-06b26e-kserve-545cf57898-2fhhd\" (UID: \"71294bfa-ac2c-40b7-b889-0527daa07f63\") " pod="e2e-models-e2e-worker-w0-06b26e/distinct-w0-06b26e-kserve-545cf57898-2fhhd"
Oct 30 07:10:55.251895 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:55.251779    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dscrd\" (UniqueName: \"kubernetes.io/projected/71294bfa-ac2c-40b7-b889-0527daa07f63-kube-api-access-dscrd\") pod \"distinct-w0-06b26e-kserve-545cf57898-2fhhd\" (UID: \"71294bfa-ac2c-40b7-b889-0527daa07f63\") " pod="e2e-models-e2e-worker-w0-06b26e/distinct-w0-06b26e-kserve-545cf57898-2fhhd"
Oct 30 07:10:55.251895 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:55.251869    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71294bfa-ac2c-40b7-b889-0527daa07f63-tmp-dir\") pod \"distinct-w0-06b26e-kserve-545cf57898-2fhhd\" (UID: \"71294bfa-ac2c-40b7-b889-0527daa07f63\") " pod="e2e-models-e2e-worker-w0-06b26e/distinct-w0-06b26e-kserve-545cf57898-2fhhd"
Oct 30 07:10:55.252128 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:55.251929    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71294bfa-ac2c-40b7-b889-0527daa07f63-dshm\") pod \"distinct-w0-06b26e-kserve-545cf57898-2fhhd\" (UID: \"71294bfa-ac2c-40b7-b889-0527daa07f63\") " pod="e2e-models-e2e-worker-w0-06b26e/distinct-w0-06b26e-kserve-545cf57898-2fhhd"
Oct 30 07:10:55.252128 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:55.251961    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71294bfa-ac2c-40b7-b889-0527daa07f63-home\") pod \"distinct-w0-06b26e-kserve-545cf57898-2fhhd\" (UID: \"71294bfa-ac2c-40b7-b889-0527daa07f63\") " pod="e2e-models-e2e-worker-w0-06b26e/distinct-w0-06b26e-kserve-545cf57898-2fhhd"
Oct 30 07:10:55.252128 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:55.251987    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71294bfa-ac2c-40b7-b889-0527daa07f63-tls-certs\") pod \"distinct-w0-06b26e-kserve-545cf57898-2fhhd\" (UID: \"71294bfa-ac2c-40b7-b889-0527daa07f63\") " pod="e2e-models-e2e-worker-w0-06b26e/distinct-w0-06b26e-kserve-545cf57898-2fhhd"
Oct 30 07:10:55.252128 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:55.252055    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71294bfa-ac2c-40b7-b889-0527daa07f63-model-cache\") pod \"distinct-w0-06b26e-kserve-545cf57898-2fhhd\" (UID: \"71294bfa-ac2c-40b7-b889-0527daa07f63\") " pod="e2e-models-e2e-worker-w0-06b26e/distinct-w0-06b26e-kserve-545cf57898-2fhhd"
Oct 30 07:10:55.252368 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:55.252340    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71294bfa-ac2c-40b7-b889-0527daa07f63-home\") pod \"distinct-w0-06b26e-kserve-545cf57898-2fhhd\" (UID: \"71294bfa-ac2c-40b7-b889-0527daa07f63\") " pod="e2e-models-e2e-worker-w0-06b26e/distinct-w0-06b26e-kserve-545cf57898-2fhhd"
Oct 30 07:10:55.252498 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:55.252473    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71294bfa-ac2c-40b7-b889-0527daa07f63-model-cache\") pod \"distinct-w0-06b26e-kserve-545cf57898-2fhhd\" (UID: \"71294bfa-ac2c-40b7-b889-0527daa07f63\") " pod="e2e-models-e2e-worker-w0-06b26e/distinct-w0-06b26e-kserve-545cf57898-2fhhd"
Oct 30 07:10:55.252586 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:55.252566    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71294bfa-ac2c-40b7-b889-0527daa07f63-tmp-dir\") pod \"distinct-w0-06b26e-kserve-545cf57898-2fhhd\" (UID: \"71294bfa-ac2c-40b7-b889-0527daa07f63\") " pod="e2e-models-e2e-worker-w0-06b26e/distinct-w0-06b26e-kserve-545cf57898-2fhhd"
Oct 30 07:10:55.255238 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:55.255208    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71294bfa-ac2c-40b7-b889-0527daa07f63-tls-certs\") pod \"distinct-w0-06b26e-kserve-545cf57898-2fhhd\" (UID: \"71294bfa-ac2c-40b7-b889-0527daa07f63\") " pod="e2e-models-e2e-worker-w0-06b26e/distinct-w0-06b26e-kserve-545cf57898-2fhhd"
Oct 30 07:10:55.255468 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:55.255448    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71294bfa-ac2c-40b7-b889-0527daa07f63-dshm\") pod \"distinct-w0-06b26e-kserve-545cf57898-2fhhd\" (UID: \"71294bfa-ac2c-40b7-b889-0527daa07f63\") " pod="e2e-models-e2e-worker-w0-06b26e/distinct-w0-06b26e-kserve-545cf57898-2fhhd"
Oct 30 07:10:55.280844 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:55.278713    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dscrd\" (UniqueName: \"kubernetes.io/projected/71294bfa-ac2c-40b7-b889-0527daa07f63-kube-api-access-dscrd\") pod \"distinct-w0-06b26e-kserve-545cf57898-2fhhd\" (UID: \"71294bfa-ac2c-40b7-b889-0527daa07f63\") " pod="e2e-models-e2e-worker-w0-06b26e/distinct-w0-06b26e-kserve-545cf57898-2fhhd"
Oct 30 07:10:55.375323 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:55.375284    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-06b26e/distinct-w0-06b26e-kserve-545cf57898-2fhhd"
Oct 30 07:10:55.697356 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:55.697226    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-06b26e/distinct-w0-06b26e-kserve-545cf57898-2fhhd"]
Oct 30 07:10:55.701130 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:10:55.701101    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod71294bfa_ac2c_40b7_b889_0527daa07f63.slice/crio-228559ab6cb1ca0c3a65955f2efeda3ab0ba508cb9792f9f38ef0ebe23c8804f WatchSource:0}: Error finding container 228559ab6cb1ca0c3a65955f2efeda3ab0ba508cb9792f9f38ef0ebe23c8804f: Status 404 returned error can't find the container with id 228559ab6cb1ca0c3a65955f2efeda3ab0ba508cb9792f9f38ef0ebe23c8804f
Oct 30 07:10:55.795409 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:55.795364    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-06b26e/distinct-w0-06b26e-kserve-545cf57898-2fhhd" event={"ID":"71294bfa-ac2c-40b7-b889-0527daa07f63","Type":"ContainerStarted","Data":"baca22d2e895843d07e8828e9c7a0a2643a15d069661c99636a1e08a4e6374d2"}
Oct 30 07:10:55.795580 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:55.795419    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-06b26e/distinct-w0-06b26e-kserve-545cf57898-2fhhd" event={"ID":"71294bfa-ac2c-40b7-b889-0527daa07f63","Type":"ContainerStarted","Data":"228559ab6cb1ca0c3a65955f2efeda3ab0ba508cb9792f9f38ef0ebe23c8804f"}
Oct 30 07:10:55.798035 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:55.798002    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj" event={"ID":"12729552-3e12-4b64-8779-a1584734c16e","Type":"ContainerStarted","Data":"2f914474e5c2adc796e469d46b532e410e51805ff14b3566ef7c8bc8ea2adce0"}
Oct 30 07:10:55.798188 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:55.798058    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj" event={"ID":"12729552-3e12-4b64-8779-a1584734c16e","Type":"ContainerStarted","Data":"f1407b4cf0a8f71b251e66d4888bb178c4c57851385169730ba3f3d86068c855"}
Oct 30 07:10:55.849547 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:55.849493    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj" podStartSLOduration=1.849480536 podStartE2EDuration="1.849480536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:10:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:10:55.847020442 +0000 UTC m=+814.456427412" watchObservedRunningTime="2026-10-30 07:10:55.849480536 +0000 UTC m=+814.458887529"
Oct 30 07:10:56.588169 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:56.587991    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj"
Oct 30 07:10:56.589763 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:56.589719    6224 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 30 07:10:56.589911 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:56.589783    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj" podUID="12729552-3e12-4b64-8779-a1584734c16e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 30 07:10:56.849588 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:56.849497    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-06b26e/distinct-w0-06b26e-kserve-545cf57898-2fhhd" podStartSLOduration=1.849483306 podStartE2EDuration="1.849483306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:10:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:10:56.848359362 +0000 UTC m=+815.457766332" watchObservedRunningTime="2026-10-30 07:10:56.849483306 +0000 UTC m=+815.458890312"
Oct 30 07:10:57.382950 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:57.382884    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-06b26e/distinct-w0-06b26e-kserve-545cf57898-2fhhd"
Oct 30 07:10:57.588262 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:57.588230    6224 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 30 07:10:57.588713 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:57.588286    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj" podUID="12729552-3e12-4b64-8779-a1584734c16e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 30 07:10:58.588199 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:58.588163    6224 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 30 07:10:58.588431 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:58.588222    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj" podUID="12729552-3e12-4b64-8779-a1584734c16e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 30 07:10:59.587627 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:59.587589    6224 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 30 07:10:59.588073 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:10:59.588031    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj" podUID="12729552-3e12-4b64-8779-a1584734c16e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 30 07:11:00.588248 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:00.588208    6224 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 30 07:11:00.588743 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:00.588278    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj" podUID="12729552-3e12-4b64-8779-a1584734c16e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 30 07:11:01.587474 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:01.587438    6224 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 30 07:11:01.587642 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:01.587506    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj" podUID="12729552-3e12-4b64-8779-a1584734c16e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 30 07:11:01.861271 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:11:01.861183    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:11:02.316309 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:02.316274    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:11:02.592080 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:02.591999    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj"
Oct 30 07:11:02.592422 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:02.592403    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj"
Oct 30 07:11:02.592973 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:02.592952    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj"
Oct 30 07:11:05.376837 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:05.376108    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-06b26e/distinct-w0-06b26e-kserve-545cf57898-2fhhd"
Oct 30 07:11:05.429084 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:05.429052    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-06b26e/distinct-w0-06b26e-kserve-545cf57898-2fhhd"
Oct 30 07:11:05.963329 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:05.963072    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-06b26e/distinct-w0-06b26e-kserve-545cf57898-2fhhd"
Oct 30 07:11:16.977062 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:16.977016    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-2eda2c/distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw"]
Oct 30 07:11:16.981227 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:16.981206    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw"
Oct 30 07:11:17.007407 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:17.007382    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-2eda2c\"/\"distinct-2-w3-2eda2c-kserve-self-signed-certs\""
Oct 30 07:11:17.023944 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:17.023917    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2eda2c/distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw"]
Oct 30 07:11:17.106490 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:17.106455    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0597696e-26b8-4ef8-8697-64d49b2b9398-tmp-dir\") pod \"distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw\" (UID: \"0597696e-26b8-4ef8-8697-64d49b2b9398\") " pod="e2e-models-e2e-worker-w3-2eda2c/distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw"
Oct 30 07:11:17.106639 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:17.106506    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0597696e-26b8-4ef8-8697-64d49b2b9398-model-cache\") pod \"distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw\" (UID: \"0597696e-26b8-4ef8-8697-64d49b2b9398\") " pod="e2e-models-e2e-worker-w3-2eda2c/distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw"
Oct 30 07:11:17.106639 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:17.106622    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0597696e-26b8-4ef8-8697-64d49b2b9398-dshm\") pod \"distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw\" (UID: \"0597696e-26b8-4ef8-8697-64d49b2b9398\") " pod="e2e-models-e2e-worker-w3-2eda2c/distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw"
Oct 30 07:11:17.106726 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:17.106694    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0597696e-26b8-4ef8-8697-64d49b2b9398-home\") pod \"distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw\" (UID: \"0597696e-26b8-4ef8-8697-64d49b2b9398\") " pod="e2e-models-e2e-worker-w3-2eda2c/distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw"
Oct 30 07:11:17.106761 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:17.106745    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0597696e-26b8-4ef8-8697-64d49b2b9398-tls-certs\") pod \"distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw\" (UID: \"0597696e-26b8-4ef8-8697-64d49b2b9398\") " pod="e2e-models-e2e-worker-w3-2eda2c/distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw"
Oct 30 07:11:17.106926 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:17.106899    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f8gl\" (UniqueName: \"kubernetes.io/projected/0597696e-26b8-4ef8-8697-64d49b2b9398-kube-api-access-9f8gl\") pod \"distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw\" (UID: \"0597696e-26b8-4ef8-8697-64d49b2b9398\") " pod="e2e-models-e2e-worker-w3-2eda2c/distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw"
Oct 30 07:11:17.210830 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:17.207860    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0597696e-26b8-4ef8-8697-64d49b2b9398-tmp-dir\") pod \"distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw\" (UID: \"0597696e-26b8-4ef8-8697-64d49b2b9398\") " pod="e2e-models-e2e-worker-w3-2eda2c/distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw"
Oct 30 07:11:17.210830 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:17.207939    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0597696e-26b8-4ef8-8697-64d49b2b9398-model-cache\") pod \"distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw\" (UID: \"0597696e-26b8-4ef8-8697-64d49b2b9398\") " pod="e2e-models-e2e-worker-w3-2eda2c/distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw"
Oct 30 07:11:17.210830 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:17.208004    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0597696e-26b8-4ef8-8697-64d49b2b9398-dshm\") pod \"distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw\" (UID: \"0597696e-26b8-4ef8-8697-64d49b2b9398\") " pod="e2e-models-e2e-worker-w3-2eda2c/distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw"
Oct 30 07:11:17.210830 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:17.208036    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0597696e-26b8-4ef8-8697-64d49b2b9398-home\") pod \"distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw\" (UID: \"0597696e-26b8-4ef8-8697-64d49b2b9398\") " pod="e2e-models-e2e-worker-w3-2eda2c/distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw"
Oct 30 07:11:17.210830 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:17.208096    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0597696e-26b8-4ef8-8697-64d49b2b9398-tls-certs\") pod \"distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw\" (UID: \"0597696e-26b8-4ef8-8697-64d49b2b9398\") " pod="e2e-models-e2e-worker-w3-2eda2c/distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw"
Oct 30 07:11:17.210830 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:17.208169    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f8gl\" (UniqueName: \"kubernetes.io/projected/0597696e-26b8-4ef8-8697-64d49b2b9398-kube-api-access-9f8gl\") pod \"distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw\" (UID: \"0597696e-26b8-4ef8-8697-64d49b2b9398\") " pod="e2e-models-e2e-worker-w3-2eda2c/distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw"
Oct 30 07:11:17.210830 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:17.209119    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0597696e-26b8-4ef8-8697-64d49b2b9398-tmp-dir\") pod \"distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw\" (UID: \"0597696e-26b8-4ef8-8697-64d49b2b9398\") " pod="e2e-models-e2e-worker-w3-2eda2c/distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw"
Oct 30 07:11:17.210830 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:17.209551    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0597696e-26b8-4ef8-8697-64d49b2b9398-model-cache\") pod \"distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw\" (UID: \"0597696e-26b8-4ef8-8697-64d49b2b9398\") " pod="e2e-models-e2e-worker-w3-2eda2c/distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw"
Oct 30 07:11:17.210830 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:17.210166    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0597696e-26b8-4ef8-8697-64d49b2b9398-home\") pod \"distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw\" (UID: \"0597696e-26b8-4ef8-8697-64d49b2b9398\") " pod="e2e-models-e2e-worker-w3-2eda2c/distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw"
Oct 30 07:11:17.220825 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:17.213646    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0597696e-26b8-4ef8-8697-64d49b2b9398-tls-certs\") pod \"distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw\" (UID: \"0597696e-26b8-4ef8-8697-64d49b2b9398\") " pod="e2e-models-e2e-worker-w3-2eda2c/distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw"
Oct 30 07:11:17.237247 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:17.237144    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0597696e-26b8-4ef8-8697-64d49b2b9398-dshm\") pod \"distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw\" (UID: \"0597696e-26b8-4ef8-8697-64d49b2b9398\") " pod="e2e-models-e2e-worker-w3-2eda2c/distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw"
Oct 30 07:11:17.238064 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:17.238039    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f8gl\" (UniqueName: \"kubernetes.io/projected/0597696e-26b8-4ef8-8697-64d49b2b9398-kube-api-access-9f8gl\") pod \"distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw\" (UID: \"0597696e-26b8-4ef8-8697-64d49b2b9398\") " pod="e2e-models-e2e-worker-w3-2eda2c/distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw"
Oct 30 07:11:17.294908 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:17.294878    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw"
Oct 30 07:11:17.602038 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:17.602006    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2eda2c/distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw"]
Oct 30 07:11:17.958546 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:17.958461    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw" event={"ID":"0597696e-26b8-4ef8-8697-64d49b2b9398","Type":"ContainerStarted","Data":"1e1e6c1f02aac62165ac73f472cc11e8a2f238417bdd3b38f2a5d72aac48d5aa"}
Oct 30 07:11:17.958546 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:17.958505    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw" event={"ID":"0597696e-26b8-4ef8-8697-64d49b2b9398","Type":"ContainerStarted","Data":"cc16343a5fee439198c92efcb77fb41d9b058ffcf78fc347091fe748fcb3cc3b"}
Oct 30 07:11:18.013283 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:18.013232    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw" podStartSLOduration=2.013220342 podStartE2EDuration="2.013220342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:11:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:11:18.012903038 +0000 UTC m=+836.622310007" watchObservedRunningTime="2026-10-30 07:11:18.013220342 +0000 UTC m=+836.622627311"
Oct 30 07:11:18.295468 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:18.295438    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw"
Oct 30 07:11:27.295461 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:27.295426    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw"
Oct 30 07:11:27.307774 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:27.307750    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw"
Oct 30 07:11:28.040765 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:28.040738    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw"
Oct 30 07:11:28.787722 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:28.787681    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2"]
Oct 30 07:11:28.792914 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:28.792894    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2"
Oct 30 07:11:28.827444 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:28.827419    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2"]
Oct 30 07:11:28.910379 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:28.910346    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5583b983-1645-4e19-acb3-8e2f498f6b5b-workload-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2\" (UID: \"5583b983-1645-4e19-acb3-8e2f498f6b5b\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2"
Oct 30 07:11:28.910533 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:28.910384    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5583b983-1645-4e19-acb3-8e2f498f6b5b-credential-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2\" (UID: \"5583b983-1645-4e19-acb3-8e2f498f6b5b\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2"
Oct 30 07:11:28.910533 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:28.910462    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5583b983-1645-4e19-acb3-8e2f498f6b5b-istiod-ca-cert\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2\" (UID: \"5583b983-1645-4e19-acb3-8e2f498f6b5b\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2"
Oct 30 07:11:28.910625 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:28.910543    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5583b983-1645-4e19-acb3-8e2f498f6b5b-istio-data\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2\" (UID: \"5583b983-1645-4e19-acb3-8e2f498f6b5b\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2"
Oct 30 07:11:28.910669 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:28.910618    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5583b983-1645-4e19-acb3-8e2f498f6b5b-workload-certs\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2\" (UID: \"5583b983-1645-4e19-acb3-8e2f498f6b5b\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2"
Oct 30 07:11:28.910669 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:28.910660    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5583b983-1645-4e19-acb3-8e2f498f6b5b-istio-envoy\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2\" (UID: \"5583b983-1645-4e19-acb3-8e2f498f6b5b\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2"
Oct 30 07:11:28.910735 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:28.910687    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5583b983-1645-4e19-acb3-8e2f498f6b5b-istio-podinfo\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2\" (UID: \"5583b983-1645-4e19-acb3-8e2f498f6b5b\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2"
Oct 30 07:11:28.910797 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:28.910780    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ljtx\" (UniqueName: \"kubernetes.io/projected/5583b983-1645-4e19-acb3-8e2f498f6b5b-kube-api-access-2ljtx\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2\" (UID: \"5583b983-1645-4e19-acb3-8e2f498f6b5b\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2"
Oct 30 07:11:28.910870 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:28.910846    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5583b983-1645-4e19-acb3-8e2f498f6b5b-istio-token\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2\" (UID: \"5583b983-1645-4e19-acb3-8e2f498f6b5b\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2"
Oct 30 07:11:29.011700 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:29.011670    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5583b983-1645-4e19-acb3-8e2f498f6b5b-istiod-ca-cert\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2\" (UID: \"5583b983-1645-4e19-acb3-8e2f498f6b5b\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2"
Oct 30 07:11:29.011898 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:29.011742    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5583b983-1645-4e19-acb3-8e2f498f6b5b-istio-data\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2\" (UID: \"5583b983-1645-4e19-acb3-8e2f498f6b5b\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2"
Oct 30 07:11:29.011898 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:29.011781    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5583b983-1645-4e19-acb3-8e2f498f6b5b-workload-certs\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2\" (UID: \"5583b983-1645-4e19-acb3-8e2f498f6b5b\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2"
Oct 30 07:11:29.011898 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:29.011832    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5583b983-1645-4e19-acb3-8e2f498f6b5b-istio-envoy\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2\" (UID: \"5583b983-1645-4e19-acb3-8e2f498f6b5b\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2"
Oct 30 07:11:29.011898 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:29.011853    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5583b983-1645-4e19-acb3-8e2f498f6b5b-istio-podinfo\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2\" (UID: \"5583b983-1645-4e19-acb3-8e2f498f6b5b\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2"
Oct 30 07:11:29.012126 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:29.011903    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ljtx\" (UniqueName: \"kubernetes.io/projected/5583b983-1645-4e19-acb3-8e2f498f6b5b-kube-api-access-2ljtx\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2\" (UID: \"5583b983-1645-4e19-acb3-8e2f498f6b5b\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2"
Oct 30 07:11:29.012126 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:29.011924    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5583b983-1645-4e19-acb3-8e2f498f6b5b-istio-token\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2\" (UID: \"5583b983-1645-4e19-acb3-8e2f498f6b5b\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2"
Oct 30 07:11:29.012126 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:29.011970    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5583b983-1645-4e19-acb3-8e2f498f6b5b-workload-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2\" (UID: \"5583b983-1645-4e19-acb3-8e2f498f6b5b\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2"
Oct 30 07:11:29.012126 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:29.012002    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5583b983-1645-4e19-acb3-8e2f498f6b5b-credential-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2\" (UID: \"5583b983-1645-4e19-acb3-8e2f498f6b5b\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2"
Oct 30 07:11:29.012325 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:29.012193    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5583b983-1645-4e19-acb3-8e2f498f6b5b-istio-data\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2\" (UID: \"5583b983-1645-4e19-acb3-8e2f498f6b5b\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2"
Oct 30 07:11:29.012382 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:29.012365    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5583b983-1645-4e19-acb3-8e2f498f6b5b-credential-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2\" (UID: \"5583b983-1645-4e19-acb3-8e2f498f6b5b\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2"
Oct 30 07:11:29.012444 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:29.012382    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5583b983-1645-4e19-acb3-8e2f498f6b5b-workload-certs\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2\" (UID: \"5583b983-1645-4e19-acb3-8e2f498f6b5b\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2"
Oct 30 07:11:29.012607 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:29.012582    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5583b983-1645-4e19-acb3-8e2f498f6b5b-istiod-ca-cert\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2\" (UID: \"5583b983-1645-4e19-acb3-8e2f498f6b5b\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2"
Oct 30 07:11:29.012661 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:29.012605    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5583b983-1645-4e19-acb3-8e2f498f6b5b-workload-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2\" (UID: \"5583b983-1645-4e19-acb3-8e2f498f6b5b\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2"
Oct 30 07:11:29.014318 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:29.014298    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5583b983-1645-4e19-acb3-8e2f498f6b5b-istio-envoy\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2\" (UID: \"5583b983-1645-4e19-acb3-8e2f498f6b5b\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2"
Oct 30 07:11:29.014396 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:29.014365    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5583b983-1645-4e19-acb3-8e2f498f6b5b-istio-podinfo\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2\" (UID: \"5583b983-1645-4e19-acb3-8e2f498f6b5b\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2"
Oct 30 07:11:29.038359 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:29.038282    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ljtx\" (UniqueName: \"kubernetes.io/projected/5583b983-1645-4e19-acb3-8e2f498f6b5b-kube-api-access-2ljtx\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2\" (UID: \"5583b983-1645-4e19-acb3-8e2f498f6b5b\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2"
Oct 30 07:11:29.039454 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:29.039420    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5583b983-1645-4e19-acb3-8e2f498f6b5b-istio-token\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2\" (UID: \"5583b983-1645-4e19-acb3-8e2f498f6b5b\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2"
Oct 30 07:11:29.106866 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:29.106836    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2"
Oct 30 07:11:29.435371 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:11:29.435339    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5583b983_1645_4e19_acb3_8e2f498f6b5b.slice/crio-844fb8ffa43b5a45161e1c4ecb230b5a148abfcaad60f043d26f304c76c37087 WatchSource:0}: Error finding container 844fb8ffa43b5a45161e1c4ecb230b5a148abfcaad60f043d26f304c76c37087: Status 404 returned error can't find the container with id 844fb8ffa43b5a45161e1c4ecb230b5a148abfcaad60f043d26f304c76c37087
Oct 30 07:11:29.436039 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:29.436012    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2"]
Oct 30 07:11:29.437769 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:29.437740    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:11:29.437862 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:29.437831    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:11:29.437907 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:29.437868    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:11:30.045210 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:30.045175    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2" event={"ID":"5583b983-1645-4e19-acb3-8e2f498f6b5b","Type":"ContainerStarted","Data":"b0569e021d4e302ca6c6f736e53810818ccb1c88692bd2746bfe157eabab1629"}
Oct 30 07:11:30.045620 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:30.045238    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2" event={"ID":"5583b983-1645-4e19-acb3-8e2f498f6b5b","Type":"ContainerStarted","Data":"844fb8ffa43b5a45161e1c4ecb230b5a148abfcaad60f043d26f304c76c37087"}
Oct 30 07:11:30.111268 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:30.108239    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2"
Oct 30 07:11:30.111268 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:30.110139    6224 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 30 07:11:30.111268 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:30.110238    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2" podUID="5583b983-1645-4e19-acb3-8e2f498f6b5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 30 07:11:30.111268 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:30.110726    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2" podStartSLOduration=2.110708893 podStartE2EDuration="2.110708893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:11:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:11:30.103155357 +0000 UTC m=+848.712562354" watchObservedRunningTime="2026-10-30 07:11:30.110708893 +0000 UTC m=+848.720115863"
Oct 30 07:11:31.107715 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:31.107684    6224 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 30 07:11:31.108203 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:31.107740    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2" podUID="5583b983-1645-4e19-acb3-8e2f498f6b5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 30 07:11:31.859040 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:11:31.859002    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:11:32.107573 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:32.107538    6224 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 30 07:11:32.107775 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:32.107605    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2" podUID="5583b983-1645-4e19-acb3-8e2f498f6b5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 30 07:11:33.107268 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:33.107230    6224 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 30 07:11:33.107454 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:33.107308    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2" podUID="5583b983-1645-4e19-acb3-8e2f498f6b5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 30 07:11:34.108195 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:34.108152    6224 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 30 07:11:34.108602 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:34.108214    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2" podUID="5583b983-1645-4e19-acb3-8e2f498f6b5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 30 07:11:35.107332 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:35.107297    6224 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 30 07:11:35.107522 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:35.107358    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2" podUID="5583b983-1645-4e19-acb3-8e2f498f6b5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 30 07:11:35.612714 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:35.612675    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5"]
Oct 30 07:11:35.616609 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:35.616590    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5"
Oct 30 07:11:35.643058 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:35.643029    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5"]
Oct 30 07:11:35.710386 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:35.710345    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c358f8-4625-4498-8322-c843e5eeb017-workload-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5\" (UID: \"d0c358f8-4625-4498-8322-c843e5eeb017\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5"
Oct 30 07:11:35.710539 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:35.710463    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c358f8-4625-4498-8322-c843e5eeb017-credential-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5\" (UID: \"d0c358f8-4625-4498-8322-c843e5eeb017\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5"
Oct 30 07:11:35.710539 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:35.710492    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0c358f8-4625-4498-8322-c843e5eeb017-istio-token\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5\" (UID: \"d0c358f8-4625-4498-8322-c843e5eeb017\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5"
Oct 30 07:11:35.710611 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:35.710555    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0c358f8-4625-4498-8322-c843e5eeb017-istio-podinfo\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5\" (UID: \"d0c358f8-4625-4498-8322-c843e5eeb017\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5"
Oct 30 07:11:35.710645 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:35.710619    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0c358f8-4625-4498-8322-c843e5eeb017-istiod-ca-cert\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5\" (UID: \"d0c358f8-4625-4498-8322-c843e5eeb017\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5"
Oct 30 07:11:35.710727 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:35.710647    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpszx\" (UniqueName: \"kubernetes.io/projected/d0c358f8-4625-4498-8322-c843e5eeb017-kube-api-access-dpszx\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5\" (UID: \"d0c358f8-4625-4498-8322-c843e5eeb017\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5"
Oct 30 07:11:35.710761 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:35.710740    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0c358f8-4625-4498-8322-c843e5eeb017-workload-certs\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5\" (UID: \"d0c358f8-4625-4498-8322-c843e5eeb017\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5"
Oct 30 07:11:35.710843 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:35.710826    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0c358f8-4625-4498-8322-c843e5eeb017-istio-envoy\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5\" (UID: \"d0c358f8-4625-4498-8322-c843e5eeb017\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5"
Oct 30 07:11:35.710926 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:35.710905    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0c358f8-4625-4498-8322-c843e5eeb017-istio-data\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5\" (UID: \"d0c358f8-4625-4498-8322-c843e5eeb017\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5"
Oct 30 07:11:35.812208 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:35.812172    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c358f8-4625-4498-8322-c843e5eeb017-workload-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5\" (UID: \"d0c358f8-4625-4498-8322-c843e5eeb017\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5"
Oct 30 07:11:35.812385 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:35.812328    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c358f8-4625-4498-8322-c843e5eeb017-credential-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5\" (UID: \"d0c358f8-4625-4498-8322-c843e5eeb017\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5"
Oct 30 07:11:35.812385 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:35.812361    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0c358f8-4625-4498-8322-c843e5eeb017-istio-token\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5\" (UID: \"d0c358f8-4625-4498-8322-c843e5eeb017\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5"
Oct 30 07:11:35.812502 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:35.812398    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0c358f8-4625-4498-8322-c843e5eeb017-istio-podinfo\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5\" (UID: \"d0c358f8-4625-4498-8322-c843e5eeb017\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5"
Oct 30 07:11:35.812502 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:35.812433    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0c358f8-4625-4498-8322-c843e5eeb017-istiod-ca-cert\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5\" (UID: \"d0c358f8-4625-4498-8322-c843e5eeb017\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5"
Oct 30 07:11:35.812502 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:35.812476    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpszx\" (UniqueName: \"kubernetes.io/projected/d0c358f8-4625-4498-8322-c843e5eeb017-kube-api-access-dpszx\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5\" (UID: \"d0c358f8-4625-4498-8322-c843e5eeb017\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5"
Oct 30 07:11:35.812661 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:35.812536    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0c358f8-4625-4498-8322-c843e5eeb017-workload-certs\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5\" (UID: \"d0c358f8-4625-4498-8322-c843e5eeb017\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5"
Oct 30 07:11:35.812661 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:35.812623    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0c358f8-4625-4498-8322-c843e5eeb017-istio-envoy\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5\" (UID: \"d0c358f8-4625-4498-8322-c843e5eeb017\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5"
Oct 30 07:11:35.812661 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:35.812652    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0c358f8-4625-4498-8322-c843e5eeb017-istio-data\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5\" (UID: \"d0c358f8-4625-4498-8322-c843e5eeb017\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5"
Oct 30 07:11:35.813006 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:35.812975    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0c358f8-4625-4498-8322-c843e5eeb017-workload-certs\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5\" (UID: \"d0c358f8-4625-4498-8322-c843e5eeb017\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5"
Oct 30 07:11:35.813113 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:35.813072    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0c358f8-4625-4498-8322-c843e5eeb017-istio-data\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5\" (UID: \"d0c358f8-4625-4498-8322-c843e5eeb017\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5"
Oct 30 07:11:35.813113 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:35.813096    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c358f8-4625-4498-8322-c843e5eeb017-workload-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5\" (UID: \"d0c358f8-4625-4498-8322-c843e5eeb017\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5"
Oct 30 07:11:35.813333 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:35.813304    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0c358f8-4625-4498-8322-c843e5eeb017-istiod-ca-cert\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5\" (UID: \"d0c358f8-4625-4498-8322-c843e5eeb017\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5"
Oct 30 07:11:35.813445 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:35.813384    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c358f8-4625-4498-8322-c843e5eeb017-credential-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5\" (UID: \"d0c358f8-4625-4498-8322-c843e5eeb017\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5"
Oct 30 07:11:35.814855 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:35.814829    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0c358f8-4625-4498-8322-c843e5eeb017-istio-envoy\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5\" (UID: \"d0c358f8-4625-4498-8322-c843e5eeb017\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5"
Oct 30 07:11:35.815152 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:35.815133    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0c358f8-4625-4498-8322-c843e5eeb017-istio-podinfo\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5\" (UID: \"d0c358f8-4625-4498-8322-c843e5eeb017\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5"
Oct 30 07:11:35.837720 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:35.837687    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0c358f8-4625-4498-8322-c843e5eeb017-istio-token\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5\" (UID: \"d0c358f8-4625-4498-8322-c843e5eeb017\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5"
Oct 30 07:11:35.838529 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:35.838503    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpszx\" (UniqueName: \"kubernetes.io/projected/d0c358f8-4625-4498-8322-c843e5eeb017-kube-api-access-dpszx\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5\" (UID: \"d0c358f8-4625-4498-8322-c843e5eeb017\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5"
Oct 30 07:11:35.932960 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:35.932889    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5"
Oct 30 07:11:36.107455 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:36.107420    6224 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 30 07:11:36.107638 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:36.107482    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2" podUID="5583b983-1645-4e19-acb3-8e2f498f6b5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 30 07:11:36.269400 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:36.269369    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5"]
Oct 30 07:11:36.270603 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:11:36.270570    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0c358f8_4625_4498_8322_c843e5eeb017.slice/crio-b7e9027551366c54782fb6711b13bac524e212650bf4dd5179c9b94a8c78f85c WatchSource:0}: Error finding container b7e9027551366c54782fb6711b13bac524e212650bf4dd5179c9b94a8c78f85c: Status 404 returned error can't find the container with id b7e9027551366c54782fb6711b13bac524e212650bf4dd5179c9b94a8c78f85c
Oct 30 07:11:36.273674 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:36.273634    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:11:36.273784 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:36.273706    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:11:36.273784 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:36.273754    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:11:37.114498 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:37.114466    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2"
Oct 30 07:11:37.115279 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:37.115259    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2"
Oct 30 07:11:37.116176 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:37.116149    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2"
Oct 30 07:11:37.116401 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:37.116373    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5" event={"ID":"d0c358f8-4625-4498-8322-c843e5eeb017","Type":"ContainerStarted","Data":"2da2762b04296e23f620839c5d9e9fa750a7720bbfac1ec922a59d0807d43909"}
Oct 30 07:11:37.116517 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:37.116416    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5" event={"ID":"d0c358f8-4625-4498-8322-c843e5eeb017","Type":"ContainerStarted","Data":"b7e9027551366c54782fb6711b13bac524e212650bf4dd5179c9b94a8c78f85c"}
Oct 30 07:11:37.245177 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:37.245127    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5" podStartSLOduration=2.245113625 podStartE2EDuration="2.245113625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:11:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:11:37.244606983 +0000 UTC m=+855.854013958" watchObservedRunningTime="2026-10-30 07:11:37.245113625 +0000 UTC m=+855.854520595"
Oct 30 07:11:37.933957 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:37.933926    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5"
Oct 30 07:11:37.934674 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:37.934646    6224 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Oct 30 07:11:37.934792 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:37.934694    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5" podUID="d0c358f8-4625-4498-8322-c843e5eeb017" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Oct 30 07:11:38.933742 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:38.933704    6224 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Oct 30 07:11:38.934151 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:38.933760    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5" podUID="d0c358f8-4625-4498-8322-c843e5eeb017" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Oct 30 07:11:39.934101 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:39.934067    6224 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Oct 30 07:11:39.934474 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:39.934129    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5" podUID="d0c358f8-4625-4498-8322-c843e5eeb017" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Oct 30 07:11:40.933952 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:40.933914    6224 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Oct 30 07:11:40.934124 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:40.933983    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5" podUID="d0c358f8-4625-4498-8322-c843e5eeb017" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Oct 30 07:11:41.933492 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:41.933447    6224 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Oct 30 07:11:41.933660 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:41.933528    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5" podUID="d0c358f8-4625-4498-8322-c843e5eeb017" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Oct 30 07:11:42.934111 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:42.934079    6224 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Oct 30 07:11:42.934487 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:42.934134    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5" podUID="d0c358f8-4625-4498-8322-c843e5eeb017" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Oct 30 07:11:43.300757 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:43.300719    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm"]
Oct 30 07:11:43.305387 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:43.305356    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm"
Oct 30 07:11:43.327554 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:43.327526    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm"]
Oct 30 07:11:43.432386 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:43.432349    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68dd39f6-f527-431d-94d4-52a6101d08de-istio-token\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm\" (UID: \"68dd39f6-f527-431d-94d4-52a6101d08de\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm"
Oct 30 07:11:43.432592 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:43.432397    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68dd39f6-f527-431d-94d4-52a6101d08de-istio-data\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm\" (UID: \"68dd39f6-f527-431d-94d4-52a6101d08de\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm"
Oct 30 07:11:43.432592 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:43.432497    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdt5z\" (UniqueName: \"kubernetes.io/projected/68dd39f6-f527-431d-94d4-52a6101d08de-kube-api-access-fdt5z\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm\" (UID: \"68dd39f6-f527-431d-94d4-52a6101d08de\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm"
Oct 30 07:11:43.432592 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:43.432584    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68dd39f6-f527-431d-94d4-52a6101d08de-istio-envoy\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm\" (UID: \"68dd39f6-f527-431d-94d4-52a6101d08de\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm"
Oct 30 07:11:43.432778 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:43.432630    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68dd39f6-f527-431d-94d4-52a6101d08de-istiod-ca-cert\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm\" (UID: \"68dd39f6-f527-431d-94d4-52a6101d08de\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm"
Oct 30 07:11:43.432778 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:43.432746    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68dd39f6-f527-431d-94d4-52a6101d08de-workload-socket\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm\" (UID: \"68dd39f6-f527-431d-94d4-52a6101d08de\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm"
Oct 30 07:11:43.432921 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:43.432835    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68dd39f6-f527-431d-94d4-52a6101d08de-istio-podinfo\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm\" (UID: \"68dd39f6-f527-431d-94d4-52a6101d08de\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm"
Oct 30 07:11:43.433003 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:43.432981    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68dd39f6-f527-431d-94d4-52a6101d08de-credential-socket\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm\" (UID: \"68dd39f6-f527-431d-94d4-52a6101d08de\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm"
Oct 30 07:11:43.433079 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:43.433033    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68dd39f6-f527-431d-94d4-52a6101d08de-workload-certs\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm\" (UID: \"68dd39f6-f527-431d-94d4-52a6101d08de\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm"
Oct 30 07:11:43.534123 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:43.534082    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68dd39f6-f527-431d-94d4-52a6101d08de-istio-envoy\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm\" (UID: \"68dd39f6-f527-431d-94d4-52a6101d08de\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm"
Oct 30 07:11:43.534288 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:43.534145    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68dd39f6-f527-431d-94d4-52a6101d08de-istiod-ca-cert\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm\" (UID: \"68dd39f6-f527-431d-94d4-52a6101d08de\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm"
Oct 30 07:11:43.534288 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:43.534181    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68dd39f6-f527-431d-94d4-52a6101d08de-workload-socket\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm\" (UID: \"68dd39f6-f527-431d-94d4-52a6101d08de\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm"
Oct 30 07:11:43.534288 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:43.534231    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68dd39f6-f527-431d-94d4-52a6101d08de-istio-podinfo\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm\" (UID: \"68dd39f6-f527-431d-94d4-52a6101d08de\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm"
Oct 30 07:11:43.534496 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:43.534294    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68dd39f6-f527-431d-94d4-52a6101d08de-credential-socket\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm\" (UID: \"68dd39f6-f527-431d-94d4-52a6101d08de\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm"
Oct 30 07:11:43.534496 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:43.534332    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68dd39f6-f527-431d-94d4-52a6101d08de-workload-certs\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm\" (UID: \"68dd39f6-f527-431d-94d4-52a6101d08de\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm"
Oct 30 07:11:43.534496 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:43.534411    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68dd39f6-f527-431d-94d4-52a6101d08de-istio-token\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm\" (UID: \"68dd39f6-f527-431d-94d4-52a6101d08de\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm"
Oct 30 07:11:43.534496 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:43.534440    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68dd39f6-f527-431d-94d4-52a6101d08de-istio-data\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm\" (UID: \"68dd39f6-f527-431d-94d4-52a6101d08de\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm"
Oct 30 07:11:43.534496 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:43.534471    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdt5z\" (UniqueName: \"kubernetes.io/projected/68dd39f6-f527-431d-94d4-52a6101d08de-kube-api-access-fdt5z\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm\" (UID: \"68dd39f6-f527-431d-94d4-52a6101d08de\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm"
Oct 30 07:11:43.534864 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:43.534795    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68dd39f6-f527-431d-94d4-52a6101d08de-workload-certs\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm\" (UID: \"68dd39f6-f527-431d-94d4-52a6101d08de\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm"
Oct 30 07:11:43.535051 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:43.534970    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68dd39f6-f527-431d-94d4-52a6101d08de-istiod-ca-cert\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm\" (UID: \"68dd39f6-f527-431d-94d4-52a6101d08de\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm"
Oct 30 07:11:43.535160 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:43.535123    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68dd39f6-f527-431d-94d4-52a6101d08de-istio-data\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm\" (UID: \"68dd39f6-f527-431d-94d4-52a6101d08de\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm"
Oct 30 07:11:43.535356 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:43.535329    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68dd39f6-f527-431d-94d4-52a6101d08de-credential-socket\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm\" (UID: \"68dd39f6-f527-431d-94d4-52a6101d08de\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm"
Oct 30 07:11:43.535710 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:43.535676    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68dd39f6-f527-431d-94d4-52a6101d08de-workload-socket\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm\" (UID: \"68dd39f6-f527-431d-94d4-52a6101d08de\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm"
Oct 30 07:11:43.536915 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:43.536890    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68dd39f6-f527-431d-94d4-52a6101d08de-istio-envoy\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm\" (UID: \"68dd39f6-f527-431d-94d4-52a6101d08de\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm"
Oct 30 07:11:43.537404 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:43.537381    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68dd39f6-f527-431d-94d4-52a6101d08de-istio-podinfo\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm\" (UID: \"68dd39f6-f527-431d-94d4-52a6101d08de\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm"
Oct 30 07:11:43.559684 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:43.559616    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdt5z\" (UniqueName: \"kubernetes.io/projected/68dd39f6-f527-431d-94d4-52a6101d08de-kube-api-access-fdt5z\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm\" (UID: \"68dd39f6-f527-431d-94d4-52a6101d08de\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm"
Oct 30 07:11:43.559787 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:43.559746    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68dd39f6-f527-431d-94d4-52a6101d08de-istio-token\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm\" (UID: \"68dd39f6-f527-431d-94d4-52a6101d08de\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm"
Oct 30 07:11:43.626299 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:43.626269    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm"
Oct 30 07:11:43.939139 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:43.939064    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5"
Oct 30 07:11:43.939599 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:43.939461    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5"
Oct 30 07:11:43.940522 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:43.940496    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5"
Oct 30 07:11:43.949194 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:43.949170    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm"]
Oct 30 07:11:43.949939 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:11:43.949917    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68dd39f6_f527_431d_94d4_52a6101d08de.slice/crio-070b42e2a384314fd3b3c3127e9c7fb4e381d69db541cfaadc159187ecc49303 WatchSource:0}: Error finding container 070b42e2a384314fd3b3c3127e9c7fb4e381d69db541cfaadc159187ecc49303: Status 404 returned error can't find the container with id 070b42e2a384314fd3b3c3127e9c7fb4e381d69db541cfaadc159187ecc49303
Oct 30 07:11:43.952343 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:43.952315    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:11:43.952440 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:43.952381    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:11:43.952440 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:43.952422    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:11:44.170282 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:44.170228    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm" event={"ID":"68dd39f6-f527-431d-94d4-52a6101d08de","Type":"ContainerStarted","Data":"9bf9195a84314feb2f56ae23b330b25f169c170076cfa5a010775827cedde575"}
Oct 30 07:11:44.170464 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:44.170330    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm" event={"ID":"68dd39f6-f527-431d-94d4-52a6101d08de","Type":"ContainerStarted","Data":"070b42e2a384314fd3b3c3127e9c7fb4e381d69db541cfaadc159187ecc49303"}
Oct 30 07:11:44.225273 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:44.223460    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm" podStartSLOduration=1.22344668 podStartE2EDuration="1.22344668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:11:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:11:44.219558246 +0000 UTC m=+862.828965217" watchObservedRunningTime="2026-10-30 07:11:44.22344668 +0000 UTC m=+862.832853649"
Oct 30 07:11:44.626470 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:44.626438    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm"
Oct 30 07:11:45.489830 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:45.489602    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:11:45.628199 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:45.628166    6224 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 07:11:45.628365 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:45.628218    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm" podUID="68dd39f6-f527-431d-94d4-52a6101d08de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 07:11:46.627329 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:46.627289    6224 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 07:11:46.627820 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:46.627343    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm" podUID="68dd39f6-f527-431d-94d4-52a6101d08de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 07:11:47.630830 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:47.628844    6224 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 07:11:47.630830 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:47.629000    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm" podUID="68dd39f6-f527-431d-94d4-52a6101d08de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 07:11:47.936084 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:47.935978    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-06b26e/distinct-w0-06b26e-kserve-545cf57898-2fhhd" podUID="71294bfa-ac2c-40b7-b889-0527daa07f63" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:11:48.235930 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:48.233989    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2bab4c/distinct-w4-2bab4c-kserve-6868d767c9-j7wlm" podUID="11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:11:48.345995 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:48.345934    6224 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-bb6be4/premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t" podUID="7dd5133c-480f-4fa0-9dd1-27662004aae9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:11:48.393015 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:48.392970    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/simulator-w3-2eda2c-kserve-56776b9849-9bqzm" podUID="482e6b7c-2231-423c-b54f-a54e56f16ab7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:11:48.484840 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:48.482236    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2bab4c/simulator-w4-2bab4c-kserve-6db9444549-7nrk8" podUID="c7b09178-9ac7-492f-890b-5ff3da130981" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:11:48.627319 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:48.627282    6224 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 07:11:48.627643 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:48.627597    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm" podUID="68dd39f6-f527-431d-94d4-52a6101d08de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 07:11:49.011882 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:48.941273    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-06b26e/distinct-w0-06b26e-kserve-545cf57898-2fhhd" podUID="71294bfa-ac2c-40b7-b889-0527daa07f63" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Oct 30 07:11:49.011882 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:48.941925    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-06b26e/distinct-w0-06b26e-kserve-545cf57898-2fhhd"
Oct 30 07:11:49.047833 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:49.046679    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-06b26e/distinct-w0-06b26e-kserve-545cf57898-2fhhd"
Oct 30 07:11:49.081846 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:49.054926    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw" podUID="0597696e-26b8-4ef8-8697-64d49b2b9398" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 30 07:11:49.724835 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:49.723951    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb6be4/premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t" podUID="7dd5133c-480f-4fa0-9dd1-27662004aae9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:11:49.833835 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:49.832430    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-06b26e/premium-w0-06b26e-kserve-c9dbd9879-b4jlk" podUID="1230dcae-e1cd-460d-86fc-bd40c8427b5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:11:49.871831 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:49.871227    6224 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 07:11:49.871831 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:49.871299    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm" podUID="68dd39f6-f527-431d-94d4-52a6101d08de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 07:11:50.068856 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:50.068512    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw" podUID="0597696e-26b8-4ef8-8697-64d49b2b9398" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:11:50.068856 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:50.068743    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw"
Oct 30 07:11:50.138841 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:50.124289    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw"
Oct 30 07:11:50.627814 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:50.627773    6224 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 07:11:50.627984 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:50.627846    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm" podUID="68dd39f6-f527-431d-94d4-52a6101d08de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 07:11:51.627061 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:51.627027    6224 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 07:11:51.627434 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:51.627098    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm" podUID="68dd39f6-f527-431d-94d4-52a6101d08de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 07:11:52.894452 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:52.894417    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm"
Oct 30 07:11:52.895003 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:52.894981    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm"
Oct 30 07:11:52.895922 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:11:52.895902    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm"
Oct 30 07:12:01.858773 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:12:01.858728    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:12:04.990849 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:04.990065    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv"]
Oct 30 07:12:04.996176 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:04.996151    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv"
Oct 30 07:12:05.016537 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:05.016514    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv"]
Oct 30 07:12:05.026643 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:05.026615    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-a163b2-openshift-default-dockercfg-j92kn\""
Oct 30 07:12:05.112929 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:05.112890    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14cba440-5936-4d19-bc3d-6d3aabc70f38-workload-socket\") pod \"e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv\" (UID: \"14cba440-5936-4d19-bc3d-6d3aabc70f38\") " pod="openshift-ingress/e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv"
Oct 30 07:12:05.113116 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:05.113087    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14cba440-5936-4d19-bc3d-6d3aabc70f38-istio-podinfo\") pod \"e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv\" (UID: \"14cba440-5936-4d19-bc3d-6d3aabc70f38\") " pod="openshift-ingress/e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv"
Oct 30 07:12:05.113190 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:05.113155    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14cba440-5936-4d19-bc3d-6d3aabc70f38-istio-envoy\") pod \"e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv\" (UID: \"14cba440-5936-4d19-bc3d-6d3aabc70f38\") " pod="openshift-ingress/e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv"
Oct 30 07:12:05.113239 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:05.113190    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14cba440-5936-4d19-bc3d-6d3aabc70f38-credential-socket\") pod \"e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv\" (UID: \"14cba440-5936-4d19-bc3d-6d3aabc70f38\") " pod="openshift-ingress/e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv"
Oct 30 07:12:05.113327 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:05.113313    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14cba440-5936-4d19-bc3d-6d3aabc70f38-istiod-ca-cert\") pod \"e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv\" (UID: \"14cba440-5936-4d19-bc3d-6d3aabc70f38\") " pod="openshift-ingress/e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv"
Oct 30 07:12:05.113387 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:05.113372    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14cba440-5936-4d19-bc3d-6d3aabc70f38-istio-data\") pod \"e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv\" (UID: \"14cba440-5936-4d19-bc3d-6d3aabc70f38\") " pod="openshift-ingress/e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv"
Oct 30 07:12:05.113499 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:05.113475    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14cba440-5936-4d19-bc3d-6d3aabc70f38-istio-token\") pod \"e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv\" (UID: \"14cba440-5936-4d19-bc3d-6d3aabc70f38\") " pod="openshift-ingress/e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv"
Oct 30 07:12:05.113549 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:05.113534    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14cba440-5936-4d19-bc3d-6d3aabc70f38-workload-certs\") pod \"e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv\" (UID: \"14cba440-5936-4d19-bc3d-6d3aabc70f38\") " pod="openshift-ingress/e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv"
Oct 30 07:12:05.113593 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:05.113580    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hfr7\" (UniqueName: \"kubernetes.io/projected/14cba440-5936-4d19-bc3d-6d3aabc70f38-kube-api-access-8hfr7\") pod \"e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv\" (UID: \"14cba440-5936-4d19-bc3d-6d3aabc70f38\") " pod="openshift-ingress/e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv"
Oct 30 07:12:05.215446 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:05.215403    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14cba440-5936-4d19-bc3d-6d3aabc70f38-istio-podinfo\") pod \"e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv\" (UID: \"14cba440-5936-4d19-bc3d-6d3aabc70f38\") " pod="openshift-ingress/e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv"
Oct 30 07:12:05.215645 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:05.215484    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14cba440-5936-4d19-bc3d-6d3aabc70f38-istio-envoy\") pod \"e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv\" (UID: \"14cba440-5936-4d19-bc3d-6d3aabc70f38\") " pod="openshift-ingress/e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv"
Oct 30 07:12:05.215645 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:05.215518    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14cba440-5936-4d19-bc3d-6d3aabc70f38-credential-socket\") pod \"e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv\" (UID: \"14cba440-5936-4d19-bc3d-6d3aabc70f38\") " pod="openshift-ingress/e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv"
Oct 30 07:12:05.215645 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:05.215582    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14cba440-5936-4d19-bc3d-6d3aabc70f38-istiod-ca-cert\") pod \"e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv\" (UID: \"14cba440-5936-4d19-bc3d-6d3aabc70f38\") " pod="openshift-ingress/e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv"
Oct 30 07:12:05.215645 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:05.215635    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14cba440-5936-4d19-bc3d-6d3aabc70f38-istio-data\") pod \"e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv\" (UID: \"14cba440-5936-4d19-bc3d-6d3aabc70f38\") " pod="openshift-ingress/e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv"
Oct 30 07:12:05.215895 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:05.215661    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14cba440-5936-4d19-bc3d-6d3aabc70f38-istio-token\") pod \"e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv\" (UID: \"14cba440-5936-4d19-bc3d-6d3aabc70f38\") " pod="openshift-ingress/e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv"
Oct 30 07:12:05.215895 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:05.215703    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14cba440-5936-4d19-bc3d-6d3aabc70f38-workload-certs\") pod \"e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv\" (UID: \"14cba440-5936-4d19-bc3d-6d3aabc70f38\") " pod="openshift-ingress/e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv"
Oct 30 07:12:05.215895 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:05.215743    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hfr7\" (UniqueName: \"kubernetes.io/projected/14cba440-5936-4d19-bc3d-6d3aabc70f38-kube-api-access-8hfr7\") pod \"e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv\" (UID: \"14cba440-5936-4d19-bc3d-6d3aabc70f38\") " pod="openshift-ingress/e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv"
Oct 30 07:12:05.215895 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:05.215787    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14cba440-5936-4d19-bc3d-6d3aabc70f38-workload-socket\") pod \"e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv\" (UID: \"14cba440-5936-4d19-bc3d-6d3aabc70f38\") " pod="openshift-ingress/e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv"
Oct 30 07:12:05.216387 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:05.216353    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14cba440-5936-4d19-bc3d-6d3aabc70f38-workload-socket\") pod \"e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv\" (UID: \"14cba440-5936-4d19-bc3d-6d3aabc70f38\") " pod="openshift-ingress/e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv"
Oct 30 07:12:05.217416 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:05.217388    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14cba440-5936-4d19-bc3d-6d3aabc70f38-istio-data\") pod \"e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv\" (UID: \"14cba440-5936-4d19-bc3d-6d3aabc70f38\") " pod="openshift-ingress/e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv"
Oct 30 07:12:05.220689 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:05.218005    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14cba440-5936-4d19-bc3d-6d3aabc70f38-workload-certs\") pod \"e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv\" (UID: \"14cba440-5936-4d19-bc3d-6d3aabc70f38\") " pod="openshift-ingress/e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv"
Oct 30 07:12:05.220689 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:05.218411    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14cba440-5936-4d19-bc3d-6d3aabc70f38-credential-socket\") pod \"e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv\" (UID: \"14cba440-5936-4d19-bc3d-6d3aabc70f38\") " pod="openshift-ingress/e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv"
Oct 30 07:12:05.220689 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:05.218797    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14cba440-5936-4d19-bc3d-6d3aabc70f38-istiod-ca-cert\") pod \"e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv\" (UID: \"14cba440-5936-4d19-bc3d-6d3aabc70f38\") " pod="openshift-ingress/e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv"
Oct 30 07:12:05.221159 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:05.221135    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14cba440-5936-4d19-bc3d-6d3aabc70f38-istio-envoy\") pod \"e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv\" (UID: \"14cba440-5936-4d19-bc3d-6d3aabc70f38\") " pod="openshift-ingress/e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv"
Oct 30 07:12:05.222238 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:05.222216    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14cba440-5936-4d19-bc3d-6d3aabc70f38-istio-podinfo\") pod \"e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv\" (UID: \"14cba440-5936-4d19-bc3d-6d3aabc70f38\") " pod="openshift-ingress/e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv"
Oct 30 07:12:05.243777 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:05.243710    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14cba440-5936-4d19-bc3d-6d3aabc70f38-istio-token\") pod \"e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv\" (UID: \"14cba440-5936-4d19-bc3d-6d3aabc70f38\") " pod="openshift-ingress/e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv"
Oct 30 07:12:05.245527 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:05.245504    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hfr7\" (UniqueName: \"kubernetes.io/projected/14cba440-5936-4d19-bc3d-6d3aabc70f38-kube-api-access-8hfr7\") pod \"e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv\" (UID: \"14cba440-5936-4d19-bc3d-6d3aabc70f38\") " pod="openshift-ingress/e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv"
Oct 30 07:12:05.312820 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:05.312767    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv"
Oct 30 07:12:05.641266 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:12:05.641235    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14cba440_5936_4d19_bc3d_6d3aabc70f38.slice/crio-30aa2ef05fa7c0f1727cf9d3f0082f4afe5aa4aa684f274fd65ecfadbc888c3b WatchSource:0}: Error finding container 30aa2ef05fa7c0f1727cf9d3f0082f4afe5aa4aa684f274fd65ecfadbc888c3b: Status 404 returned error can't find the container with id 30aa2ef05fa7c0f1727cf9d3f0082f4afe5aa4aa684f274fd65ecfadbc888c3b
Oct 30 07:12:05.641424 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:05.641334    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv"]
Oct 30 07:12:05.643895 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:05.643863    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:12:05.644011 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:05.643935    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:12:05.644011 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:05.644000    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:12:06.442260 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:06.442219    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv" event={"ID":"14cba440-5936-4d19-bc3d-6d3aabc70f38","Type":"ContainerStarted","Data":"423d154d81c95109780fe710d18c7badf83807d6bd2dcba26247f232cf62c783"}
Oct 30 07:12:06.442846 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:06.442823    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv" event={"ID":"14cba440-5936-4d19-bc3d-6d3aabc70f38","Type":"ContainerStarted","Data":"30aa2ef05fa7c0f1727cf9d3f0082f4afe5aa4aa684f274fd65ecfadbc888c3b"}
Oct 30 07:12:06.493500 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:06.493440    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv" podStartSLOduration=2.493421445 podStartE2EDuration="2.493421445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:12:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:12:06.492118343 +0000 UTC m=+885.101525314" watchObservedRunningTime="2026-10-30 07:12:06.493421445 +0000 UTC m=+885.102828419"
Oct 30 07:12:07.341832 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:07.321154    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv"
Oct 30 07:12:07.341832 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:07.338124    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv"
Oct 30 07:12:07.498833 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:07.492493    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv"
Oct 30 07:12:07.504836 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:07.503063    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv"
Oct 30 07:12:08.058085 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:08.056914    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-06b26e/distinct-w0-06b26e-kserve-545cf57898-2fhhd" podUID="71294bfa-ac2c-40b7-b889-0527daa07f63" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Oct 30 07:12:08.162026 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:08.161972    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw" podUID="0597696e-26b8-4ef8-8697-64d49b2b9398" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:08.197078 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:08.196917    6224 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 07:12:08.197078 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:08.196976    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2" podUID="5583b983-1645-4e19-acb3-8e2f498f6b5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:08.239729 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:08.238942    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2bab4c/distinct-w4-2bab4c-kserve-6868d767c9-j7wlm" podUID="11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Oct 30 07:12:08.314367 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:08.314036    6224 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw" podUID="0597696e-26b8-4ef8-8697-64d49b2b9398" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": context deadline exceeded"
Oct 30 07:12:08.400104 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:08.398921    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/simulator-w3-2eda2c-kserve-56776b9849-9bqzm" podUID="482e6b7c-2231-423c-b54f-a54e56f16ab7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:08.435972 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:08.434920    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb6be4/simulator-w2-bb6be4-kserve-8485bbf954-dnbbz" podUID="6370f7e2-d6c8-4fb5-8198-32b38837ca31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:08.495283 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:08.495240    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2bab4c/simulator-w4-2bab4c-kserve-6db9444549-7nrk8" podUID="c7b09178-9ac7-492f-890b-5ff3da130981" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:08.701919 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:08.696226    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2bab4c/premium-w4-2bab4c-kserve-665887fc68-c6dgf" podUID="f9463476-fcf5-4d66-bfd2-b85ce4ed651f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Oct 30 07:12:08.760830 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:08.760762    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-06b26e/premium-w0-06b26e-kserve-c9dbd9879-b4jlk" podUID="1230dcae-e1cd-460d-86fc-bd40c8427b5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 30 07:12:08.965884 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:08.964938    6224 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 07:12:08.965884 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:08.965004    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm" podUID="68dd39f6-f527-431d-94d4-52a6101d08de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:09.062834 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:09.061904    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-06b26e/distinct-w0-06b26e-kserve-545cf57898-2fhhd" podUID="71294bfa-ac2c-40b7-b889-0527daa07f63" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:09.062834 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:09.062538    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-06b26e/distinct-w0-06b26e-kserve-545cf57898-2fhhd"
Oct 30 07:12:09.206832 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:09.206640    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw" podUID="0597696e-26b8-4ef8-8697-64d49b2b9398" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 30 07:12:09.223878 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:09.221268    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw"
Oct 30 07:12:09.605833 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:09.593999    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/simulator-w3-2eda2c-kserve-56776b9849-9bqzm" podUID="482e6b7c-2231-423c-b54f-a54e56f16ab7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded"
Oct 30 07:12:09.630831 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:09.606473    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb6be4/simulator-w2-bb6be4-kserve-8485bbf954-dnbbz" podUID="6370f7e2-d6c8-4fb5-8198-32b38837ca31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Oct 30 07:12:09.679830 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:09.653703    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb6be4/premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t" podUID="7dd5133c-480f-4fa0-9dd1-27662004aae9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:09.698635 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:09.698593    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2bab4c/premium-w4-2bab4c-kserve-665887fc68-c6dgf" podUID="f9463476-fcf5-4d66-bfd2-b85ce4ed651f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Oct 30 07:12:09.761840 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:09.761183    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-06b26e/premium-w0-06b26e-kserve-c9dbd9879-b4jlk" podUID="1230dcae-e1cd-460d-86fc-bd40c8427b5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:09.922501 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:09.917648    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2eda2c/simulator-w3-2eda2c-kserve-56776b9849-9bqzm"
Oct 30 07:12:09.922501 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:09.917713    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb6be4/simulator-w2-bb6be4-kserve-8485bbf954-dnbbz"
Oct 30 07:12:09.922501 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:09.917749    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2bab4c/premium-w4-2bab4c-kserve-665887fc68-c6dgf"
Oct 30 07:12:09.922501 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:09.917781    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-06b26e/premium-w0-06b26e-kserve-c9dbd9879-b4jlk"
Oct 30 07:12:10.149838 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:10.141500    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-06b26e/distinct-w0-06b26e-kserve-545cf57898-2fhhd" podUID="71294bfa-ac2c-40b7-b889-0527daa07f63" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:10.222895 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:10.207203    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw" podUID="0597696e-26b8-4ef8-8697-64d49b2b9398" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 30 07:12:10.270832 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:10.259618    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2bab4c/distinct-w4-2bab4c-kserve-6868d767c9-j7wlm" podUID="11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:10.614954 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:10.614911    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/simulator-w3-2eda2c-kserve-56776b9849-9bqzm" podUID="482e6b7c-2231-423c-b54f-a54e56f16ab7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded"
Oct 30 07:12:10.638830 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:10.628268    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb6be4/simulator-w2-bb6be4-kserve-8485bbf954-dnbbz" podUID="6370f7e2-d6c8-4fb5-8198-32b38837ca31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:10.651142 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:10.651058    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2bab4c/simulator-w4-2bab4c-kserve-6db9444549-7nrk8" podUID="c7b09178-9ac7-492f-890b-5ff3da130981" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:10.654797 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:10.654754    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb6be4/premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t" podUID="7dd5133c-480f-4fa0-9dd1-27662004aae9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Oct 30 07:12:10.656587 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:10.656562    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb6be4/premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t"
Oct 30 07:12:10.733151 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:10.733091    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2bab4c/premium-w4-2bab4c-kserve-665887fc68-c6dgf" podUID="f9463476-fcf5-4d66-bfd2-b85ce4ed651f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:10.835832 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:10.815929    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-06b26e/premium-w0-06b26e-kserve-c9dbd9879-b4jlk" podUID="1230dcae-e1cd-460d-86fc-bd40c8427b5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:11.173953 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:11.157888    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-06b26e/distinct-w0-06b26e-kserve-545cf57898-2fhhd" podUID="71294bfa-ac2c-40b7-b889-0527daa07f63" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:11.196044 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:11.196015    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2bab4c/premium-w4-2bab4c-kserve-665887fc68-c6dgf"
Oct 30 07:12:11.199712 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:11.199691    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-06b26e/premium-w0-06b26e-kserve-c9dbd9879-b4jlk"
Oct 30 07:12:11.208253 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:11.206485    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb6be4/premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t"
Oct 30 07:12:11.208253 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:11.207387    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw" podUID="0597696e-26b8-4ef8-8697-64d49b2b9398" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:11.217013 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:11.216995    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb6be4/simulator-w2-bb6be4-kserve-8485bbf954-dnbbz"
Oct 30 07:12:11.235865 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:11.232000    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw"
Oct 30 07:12:11.260178 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:11.260148    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2bab4c/distinct-w4-2bab4c-kserve-6868d767c9-j7wlm" podUID="11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Oct 30 07:12:11.261188 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:11.261169    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2bab4c/distinct-w4-2bab4c-kserve-6868d767c9-j7wlm"
Oct 30 07:12:11.266179 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:11.266163    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-06b26e/distinct-w0-06b26e-kserve-545cf57898-2fhhd"
Oct 30 07:12:11.291681 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:11.291117    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2eda2c/simulator-w3-2eda2c-kserve-56776b9849-9bqzm"
Oct 30 07:12:11.305773 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:11.305752    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2bab4c/distinct-w4-2bab4c-kserve-6868d767c9-j7wlm"
Oct 30 07:12:13.957543 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:13.957509    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq"]
Oct 30 07:12:13.962576 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:13.962550    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq"
Oct 30 07:12:13.984075 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:13.984049    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq"]
Oct 30 07:12:14.060926 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.060873    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/717ebd45-f957-47c5-a710-d091051fa4eb-istio-token\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq\" (UID: \"717ebd45-f957-47c5-a710-d091051fa4eb\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq"
Oct 30 07:12:14.060926 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.060927    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/717ebd45-f957-47c5-a710-d091051fa4eb-istio-data\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq\" (UID: \"717ebd45-f957-47c5-a710-d091051fa4eb\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq"
Oct 30 07:12:14.061768 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.060980    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/717ebd45-f957-47c5-a710-d091051fa4eb-istio-envoy\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq\" (UID: \"717ebd45-f957-47c5-a710-d091051fa4eb\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq"
Oct 30 07:12:14.062739 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.062710    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/717ebd45-f957-47c5-a710-d091051fa4eb-workload-certs\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq\" (UID: \"717ebd45-f957-47c5-a710-d091051fa4eb\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq"
Oct 30 07:12:14.062894 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.062853    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/717ebd45-f957-47c5-a710-d091051fa4eb-istio-podinfo\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq\" (UID: \"717ebd45-f957-47c5-a710-d091051fa4eb\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq"
Oct 30 07:12:14.062983 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.062968    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/717ebd45-f957-47c5-a710-d091051fa4eb-workload-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq\" (UID: \"717ebd45-f957-47c5-a710-d091051fa4eb\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq"
Oct 30 07:12:14.063044 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.063031    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/717ebd45-f957-47c5-a710-d091051fa4eb-istiod-ca-cert\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq\" (UID: \"717ebd45-f957-47c5-a710-d091051fa4eb\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq"
Oct 30 07:12:14.063101 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.063072    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/717ebd45-f957-47c5-a710-d091051fa4eb-credential-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq\" (UID: \"717ebd45-f957-47c5-a710-d091051fa4eb\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq"
Oct 30 07:12:14.063231 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.063205    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8fdc\" (UniqueName: \"kubernetes.io/projected/717ebd45-f957-47c5-a710-d091051fa4eb-kube-api-access-c8fdc\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq\" (UID: \"717ebd45-f957-47c5-a710-d091051fa4eb\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq"
Oct 30 07:12:14.131364 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.131336    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2"]
Oct 30 07:12:14.135165 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.135147    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2"
Oct 30 07:12:14.151633 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.151601    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-06b26e-openshift-default-dockercfg-t2k6l\""
Oct 30 07:12:14.164556 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.164528    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a24e154-72f7-48d6-9d93-2f9fee0de386-istiod-ca-cert\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2\" (UID: \"3a24e154-72f7-48d6-9d93-2f9fee0de386\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2"
Oct 30 07:12:14.164696 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.164647    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/717ebd45-f957-47c5-a710-d091051fa4eb-workload-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq\" (UID: \"717ebd45-f957-47c5-a710-d091051fa4eb\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq"
Oct 30 07:12:14.164696 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.164686    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrpj8\" (UniqueName: \"kubernetes.io/projected/3a24e154-72f7-48d6-9d93-2f9fee0de386-kube-api-access-nrpj8\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2\" (UID: \"3a24e154-72f7-48d6-9d93-2f9fee0de386\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2"
Oct 30 07:12:14.164924 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.164726    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/717ebd45-f957-47c5-a710-d091051fa4eb-istiod-ca-cert\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq\" (UID: \"717ebd45-f957-47c5-a710-d091051fa4eb\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq"
Oct 30 07:12:14.164924 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.164765    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/717ebd45-f957-47c5-a710-d091051fa4eb-credential-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq\" (UID: \"717ebd45-f957-47c5-a710-d091051fa4eb\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq"
Oct 30 07:12:14.164924 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.164794    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a24e154-72f7-48d6-9d93-2f9fee0de386-istio-data\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2\" (UID: \"3a24e154-72f7-48d6-9d93-2f9fee0de386\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2"
Oct 30 07:12:14.164924 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.164858    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8fdc\" (UniqueName: \"kubernetes.io/projected/717ebd45-f957-47c5-a710-d091051fa4eb-kube-api-access-c8fdc\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq\" (UID: \"717ebd45-f957-47c5-a710-d091051fa4eb\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq"
Oct 30 07:12:14.164924 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.164917    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a24e154-72f7-48d6-9d93-2f9fee0de386-workload-certs\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2\" (UID: \"3a24e154-72f7-48d6-9d93-2f9fee0de386\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2"
Oct 30 07:12:14.165195 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.164944    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/717ebd45-f957-47c5-a710-d091051fa4eb-istio-token\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq\" (UID: \"717ebd45-f957-47c5-a710-d091051fa4eb\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq"
Oct 30 07:12:14.165195 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.164975    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/717ebd45-f957-47c5-a710-d091051fa4eb-istio-data\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq\" (UID: \"717ebd45-f957-47c5-a710-d091051fa4eb\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq"
Oct 30 07:12:14.165195 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.164999    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a24e154-72f7-48d6-9d93-2f9fee0de386-istio-podinfo\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2\" (UID: \"3a24e154-72f7-48d6-9d93-2f9fee0de386\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2"
Oct 30 07:12:14.165195 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.165016    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/717ebd45-f957-47c5-a710-d091051fa4eb-workload-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq\" (UID: \"717ebd45-f957-47c5-a710-d091051fa4eb\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq"
Oct 30 07:12:14.165195 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.165058    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/717ebd45-f957-47c5-a710-d091051fa4eb-istio-envoy\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq\" (UID: \"717ebd45-f957-47c5-a710-d091051fa4eb\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq"
Oct 30 07:12:14.165195 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.165084    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a24e154-72f7-48d6-9d93-2f9fee0de386-credential-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2\" (UID: \"3a24e154-72f7-48d6-9d93-2f9fee0de386\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2"
Oct 30 07:12:14.165195 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.165151    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/717ebd45-f957-47c5-a710-d091051fa4eb-workload-certs\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq\" (UID: \"717ebd45-f957-47c5-a710-d091051fa4eb\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq"
Oct 30 07:12:14.165554 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.165204    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a24e154-72f7-48d6-9d93-2f9fee0de386-istio-envoy\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2\" (UID: \"3a24e154-72f7-48d6-9d93-2f9fee0de386\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2"
Oct 30 07:12:14.165554 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.165233    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a24e154-72f7-48d6-9d93-2f9fee0de386-workload-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2\" (UID: \"3a24e154-72f7-48d6-9d93-2f9fee0de386\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2"
Oct 30 07:12:14.165554 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.165280    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/717ebd45-f957-47c5-a710-d091051fa4eb-istio-podinfo\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq\" (UID: \"717ebd45-f957-47c5-a710-d091051fa4eb\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq"
Oct 30 07:12:14.165554 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.165304    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a24e154-72f7-48d6-9d93-2f9fee0de386-istio-token\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2\" (UID: \"3a24e154-72f7-48d6-9d93-2f9fee0de386\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2"
Oct 30 07:12:14.166049 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.166025    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/717ebd45-f957-47c5-a710-d091051fa4eb-workload-certs\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq\" (UID: \"717ebd45-f957-47c5-a710-d091051fa4eb\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq"
Oct 30 07:12:14.166296 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.166271    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/717ebd45-f957-47c5-a710-d091051fa4eb-istiod-ca-cert\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq\" (UID: \"717ebd45-f957-47c5-a710-d091051fa4eb\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq"
Oct 30 07:12:14.166373 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.166276    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/717ebd45-f957-47c5-a710-d091051fa4eb-istio-data\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq\" (UID: \"717ebd45-f957-47c5-a710-d091051fa4eb\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq"
Oct 30 07:12:14.166373 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.166333    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/717ebd45-f957-47c5-a710-d091051fa4eb-credential-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq\" (UID: \"717ebd45-f957-47c5-a710-d091051fa4eb\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq"
Oct 30 07:12:14.168857 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.168565    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2"]
Oct 30 07:12:14.169185 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.169160    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/717ebd45-f957-47c5-a710-d091051fa4eb-istio-envoy\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq\" (UID: \"717ebd45-f957-47c5-a710-d091051fa4eb\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq"
Oct 30 07:12:14.172651 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.172622    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/717ebd45-f957-47c5-a710-d091051fa4eb-istio-podinfo\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq\" (UID: \"717ebd45-f957-47c5-a710-d091051fa4eb\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq"
Oct 30 07:12:14.191159 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.191128    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/717ebd45-f957-47c5-a710-d091051fa4eb-istio-token\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq\" (UID: \"717ebd45-f957-47c5-a710-d091051fa4eb\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq"
Oct 30 07:12:14.191290 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.191262    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8fdc\" (UniqueName: \"kubernetes.io/projected/717ebd45-f957-47c5-a710-d091051fa4eb-kube-api-access-c8fdc\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq\" (UID: \"717ebd45-f957-47c5-a710-d091051fa4eb\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq"
Oct 30 07:12:14.266970 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.266887    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a24e154-72f7-48d6-9d93-2f9fee0de386-istio-token\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2\" (UID: \"3a24e154-72f7-48d6-9d93-2f9fee0de386\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2"
Oct 30 07:12:14.266970 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.266949    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a24e154-72f7-48d6-9d93-2f9fee0de386-istiod-ca-cert\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2\" (UID: \"3a24e154-72f7-48d6-9d93-2f9fee0de386\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2"
Oct 30 07:12:14.267211 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.267183    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrpj8\" (UniqueName: \"kubernetes.io/projected/3a24e154-72f7-48d6-9d93-2f9fee0de386-kube-api-access-nrpj8\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2\" (UID: \"3a24e154-72f7-48d6-9d93-2f9fee0de386\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2"
Oct 30 07:12:14.267333 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.267245    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a24e154-72f7-48d6-9d93-2f9fee0de386-istio-data\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2\" (UID: \"3a24e154-72f7-48d6-9d93-2f9fee0de386\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2"
Oct 30 07:12:14.267333 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.267310    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a24e154-72f7-48d6-9d93-2f9fee0de386-workload-certs\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2\" (UID: \"3a24e154-72f7-48d6-9d93-2f9fee0de386\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2"
Oct 30 07:12:14.267458 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.267337    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a24e154-72f7-48d6-9d93-2f9fee0de386-istio-podinfo\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2\" (UID: \"3a24e154-72f7-48d6-9d93-2f9fee0de386\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2"
Oct 30 07:12:14.267458 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.267387    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a24e154-72f7-48d6-9d93-2f9fee0de386-credential-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2\" (UID: \"3a24e154-72f7-48d6-9d93-2f9fee0de386\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2"
Oct 30 07:12:14.267581 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.267463    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a24e154-72f7-48d6-9d93-2f9fee0de386-istio-envoy\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2\" (UID: \"3a24e154-72f7-48d6-9d93-2f9fee0de386\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2"
Oct 30 07:12:14.267581 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.267490    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a24e154-72f7-48d6-9d93-2f9fee0de386-workload-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2\" (UID: \"3a24e154-72f7-48d6-9d93-2f9fee0de386\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2"
Oct 30 07:12:14.267682 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.267622    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a24e154-72f7-48d6-9d93-2f9fee0de386-istio-data\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2\" (UID: \"3a24e154-72f7-48d6-9d93-2f9fee0de386\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2"
Oct 30 07:12:14.267682 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.267673    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a24e154-72f7-48d6-9d93-2f9fee0de386-workload-certs\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2\" (UID: \"3a24e154-72f7-48d6-9d93-2f9fee0de386\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2"
Oct 30 07:12:14.267786 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.267730    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a24e154-72f7-48d6-9d93-2f9fee0de386-istiod-ca-cert\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2\" (UID: \"3a24e154-72f7-48d6-9d93-2f9fee0de386\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2"
Oct 30 07:12:14.267786 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.267740    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a24e154-72f7-48d6-9d93-2f9fee0de386-credential-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2\" (UID: \"3a24e154-72f7-48d6-9d93-2f9fee0de386\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2"
Oct 30 07:12:14.267961 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.267941    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a24e154-72f7-48d6-9d93-2f9fee0de386-workload-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2\" (UID: \"3a24e154-72f7-48d6-9d93-2f9fee0de386\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2"
Oct 30 07:12:14.269683 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.269658    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a24e154-72f7-48d6-9d93-2f9fee0de386-istio-envoy\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2\" (UID: \"3a24e154-72f7-48d6-9d93-2f9fee0de386\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2"
Oct 30 07:12:14.269932 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.269911    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a24e154-72f7-48d6-9d93-2f9fee0de386-istio-podinfo\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2\" (UID: \"3a24e154-72f7-48d6-9d93-2f9fee0de386\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2"
Oct 30 07:12:14.276722 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.276700    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq"
Oct 30 07:12:14.294140 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.294112    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrpj8\" (UniqueName: \"kubernetes.io/projected/3a24e154-72f7-48d6-9d93-2f9fee0de386-kube-api-access-nrpj8\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2\" (UID: \"3a24e154-72f7-48d6-9d93-2f9fee0de386\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2"
Oct 30 07:12:14.294581 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.294561    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a24e154-72f7-48d6-9d93-2f9fee0de386-istio-token\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2\" (UID: \"3a24e154-72f7-48d6-9d93-2f9fee0de386\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2"
Oct 30 07:12:14.448605 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.448570    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2"
Oct 30 07:12:14.613631 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.613599    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq"]
Oct 30 07:12:14.613882 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:12:14.613857    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod717ebd45_f957_47c5_a710_d091051fa4eb.slice/crio-10afb0de805bd5a30b412de7ea796c14823111412f9a959b1bc0e650cf2d680f WatchSource:0}: Error finding container 10afb0de805bd5a30b412de7ea796c14823111412f9a959b1bc0e650cf2d680f: Status 404 returned error can't find the container with id 10afb0de805bd5a30b412de7ea796c14823111412f9a959b1bc0e650cf2d680f
Oct 30 07:12:14.616547 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.616502    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:12:14.616653 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.616581    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:12:14.616653 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.616627    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:12:14.754706 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:12:14.754675    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a24e154_72f7_48d6_9d93_2f9fee0de386.slice/crio-76544539c080b8ec1bb08371125eb3229090d02d6a87153cce22c32f7cf006df WatchSource:0}: Error finding container 76544539c080b8ec1bb08371125eb3229090d02d6a87153cce22c32f7cf006df: Status 404 returned error can't find the container with id 76544539c080b8ec1bb08371125eb3229090d02d6a87153cce22c32f7cf006df
Oct 30 07:12:14.756358 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.756330    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2"]
Oct 30 07:12:14.757331 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.757301    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:12:14.757386 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.757364    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:12:14.757431 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:14.757396    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:12:15.210449 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:15.210406    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq" event={"ID":"717ebd45-f957-47c5-a710-d091051fa4eb","Type":"ContainerStarted","Data":"4ebe55dec33f95d5261d16444187347773d09fbb8c54ededdcf301de2a15312f"}
Oct 30 07:12:15.210975 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:15.210476    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq" event={"ID":"717ebd45-f957-47c5-a710-d091051fa4eb","Type":"ContainerStarted","Data":"10afb0de805bd5a30b412de7ea796c14823111412f9a959b1bc0e650cf2d680f"}
Oct 30 07:12:15.212734 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:15.212708    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2" event={"ID":"3a24e154-72f7-48d6-9d93-2f9fee0de386","Type":"ContainerStarted","Data":"462e3e73493d0d3ac4b08e04c48d1a80970a2cbc34da0a76315df502e1dd13f6"}
Oct 30 07:12:15.212929 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:15.212749    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2" event={"ID":"3a24e154-72f7-48d6-9d93-2f9fee0de386","Type":"ContainerStarted","Data":"76544539c080b8ec1bb08371125eb3229090d02d6a87153cce22c32f7cf006df"}
Oct 30 07:12:15.266884 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:15.263850    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq" podStartSLOduration=2.263830546 podStartE2EDuration="2.263830546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:12:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:12:15.260903219 +0000 UTC m=+893.870310190" watchObservedRunningTime="2026-10-30 07:12:15.263830546 +0000 UTC m=+893.873237513"
Oct 30 07:12:15.282522 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:15.279189    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq"
Oct 30 07:12:15.282909 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:15.282865    6224 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 07:12:15.283109 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:15.283083    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq" podUID="717ebd45-f957-47c5-a710-d091051fa4eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 07:12:15.449176 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:15.449146    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2"
Oct 30 07:12:15.450293 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:15.450263    6224 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 30 07:12:15.450441 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:15.450320    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2" podUID="3a24e154-72f7-48d6-9d93-2f9fee0de386" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 30 07:12:16.285275 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:16.285237    6224 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 07:12:16.285876 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:16.285840    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq" podUID="717ebd45-f957-47c5-a710-d091051fa4eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 07:12:16.448910 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:16.448863    6224 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 30 07:12:16.449064 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:16.448933    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2" podUID="3a24e154-72f7-48d6-9d93-2f9fee0de386" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 30 07:12:17.291211 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:17.291177    6224 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 07:12:17.291821 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:17.291773    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq" podUID="717ebd45-f957-47c5-a710-d091051fa4eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 07:12:17.449468 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:17.449431    6224 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 30 07:12:17.449699 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:17.449675    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2" podUID="3a24e154-72f7-48d6-9d93-2f9fee0de386" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 30 07:12:18.285275 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:18.285234    6224 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 07:12:18.286395 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:18.286362    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq" podUID="717ebd45-f957-47c5-a710-d091051fa4eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 07:12:18.449964 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:18.449920    6224 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 30 07:12:18.450351 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:18.449987    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2" podUID="3a24e154-72f7-48d6-9d93-2f9fee0de386" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 30 07:12:19.283224 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:19.283187    6224 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 07:12:19.283501 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:19.283470    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq" podUID="717ebd45-f957-47c5-a710-d091051fa4eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 07:12:19.449007 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:19.448969    6224 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 30 07:12:19.449170 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:19.449044    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2" podUID="3a24e154-72f7-48d6-9d93-2f9fee0de386" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 30 07:12:20.280831 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:20.278167    6224 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 07:12:20.280831 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:20.278225    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq" podUID="717ebd45-f957-47c5-a710-d091051fa4eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 07:12:20.449447 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:20.449411    6224 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 30 07:12:20.449607 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:20.449467    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2" podUID="3a24e154-72f7-48d6-9d93-2f9fee0de386" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 30 07:12:20.897164 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:20.897104    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2" podStartSLOduration=6.897090849 podStartE2EDuration="6.897090849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:12:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:12:15.315868771 +0000 UTC m=+893.925275742" watchObservedRunningTime="2026-10-30 07:12:20.897090849 +0000 UTC m=+899.506497818"
Oct 30 07:12:20.900074 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:20.900044    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz"]
Oct 30 07:12:20.904383 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:20.904358    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz"
Oct 30 07:12:20.931027 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:20.930993    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz"]
Oct 30 07:12:21.029753 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:21.027067    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac672938-faf4-44e5-98ca-ef87de6e3437-istio-podinfo\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz\" (UID: \"ac672938-faf4-44e5-98ca-ef87de6e3437\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz"
Oct 30 07:12:21.029753 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:21.027174    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac672938-faf4-44e5-98ca-ef87de6e3437-workload-certs\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz\" (UID: \"ac672938-faf4-44e5-98ca-ef87de6e3437\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz"
Oct 30 07:12:21.029753 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:21.027223    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac672938-faf4-44e5-98ca-ef87de6e3437-istio-token\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz\" (UID: \"ac672938-faf4-44e5-98ca-ef87de6e3437\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz"
Oct 30 07:12:21.029753 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:21.027283    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb75n\" (UniqueName: \"kubernetes.io/projected/ac672938-faf4-44e5-98ca-ef87de6e3437-kube-api-access-kb75n\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz\" (UID: \"ac672938-faf4-44e5-98ca-ef87de6e3437\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz"
Oct 30 07:12:21.029753 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:21.027347    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac672938-faf4-44e5-98ca-ef87de6e3437-credential-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz\" (UID: \"ac672938-faf4-44e5-98ca-ef87de6e3437\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz"
Oct 30 07:12:21.029753 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:21.027404    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac672938-faf4-44e5-98ca-ef87de6e3437-istiod-ca-cert\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz\" (UID: \"ac672938-faf4-44e5-98ca-ef87de6e3437\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz"
Oct 30 07:12:21.029753 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:21.027530    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac672938-faf4-44e5-98ca-ef87de6e3437-workload-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz\" (UID: \"ac672938-faf4-44e5-98ca-ef87de6e3437\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz"
Oct 30 07:12:21.029753 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:21.027577    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac672938-faf4-44e5-98ca-ef87de6e3437-istio-data\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz\" (UID: \"ac672938-faf4-44e5-98ca-ef87de6e3437\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz"
Oct 30 07:12:21.029753 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:21.027648    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac672938-faf4-44e5-98ca-ef87de6e3437-istio-envoy\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz\" (UID: \"ac672938-faf4-44e5-98ca-ef87de6e3437\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz"
Oct 30 07:12:21.129140 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:21.129092    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac672938-faf4-44e5-98ca-ef87de6e3437-istio-data\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz\" (UID: \"ac672938-faf4-44e5-98ca-ef87de6e3437\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz"
Oct 30 07:12:21.129334 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:21.129167    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac672938-faf4-44e5-98ca-ef87de6e3437-istio-envoy\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz\" (UID: \"ac672938-faf4-44e5-98ca-ef87de6e3437\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz"
Oct 30 07:12:21.129334 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:21.129217    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac672938-faf4-44e5-98ca-ef87de6e3437-istio-podinfo\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz\" (UID: \"ac672938-faf4-44e5-98ca-ef87de6e3437\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz"
Oct 30 07:12:21.129334 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:21.129293    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac672938-faf4-44e5-98ca-ef87de6e3437-workload-certs\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz\" (UID: \"ac672938-faf4-44e5-98ca-ef87de6e3437\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz"
Oct 30 07:12:21.129334 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:21.129331    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac672938-faf4-44e5-98ca-ef87de6e3437-istio-token\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz\" (UID: \"ac672938-faf4-44e5-98ca-ef87de6e3437\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz"
Oct 30 07:12:21.129567 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:21.129380    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb75n\" (UniqueName: \"kubernetes.io/projected/ac672938-faf4-44e5-98ca-ef87de6e3437-kube-api-access-kb75n\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz\" (UID: \"ac672938-faf4-44e5-98ca-ef87de6e3437\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz"
Oct 30 07:12:21.129567 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:21.129445    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac672938-faf4-44e5-98ca-ef87de6e3437-credential-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz\" (UID: \"ac672938-faf4-44e5-98ca-ef87de6e3437\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz"
Oct 30 07:12:21.129567 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:21.129484    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac672938-faf4-44e5-98ca-ef87de6e3437-istiod-ca-cert\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz\" (UID: \"ac672938-faf4-44e5-98ca-ef87de6e3437\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz"
Oct 30 07:12:21.129567 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:21.129537    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac672938-faf4-44e5-98ca-ef87de6e3437-istio-data\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz\" (UID: \"ac672938-faf4-44e5-98ca-ef87de6e3437\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz"
Oct 30 07:12:21.129751 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:21.129572    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac672938-faf4-44e5-98ca-ef87de6e3437-workload-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz\" (UID: \"ac672938-faf4-44e5-98ca-ef87de6e3437\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz"
Oct 30 07:12:21.130613 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:21.129961    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac672938-faf4-44e5-98ca-ef87de6e3437-workload-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz\" (UID: \"ac672938-faf4-44e5-98ca-ef87de6e3437\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz"
Oct 30 07:12:21.130613 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:21.130563    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac672938-faf4-44e5-98ca-ef87de6e3437-istiod-ca-cert\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz\" (UID: \"ac672938-faf4-44e5-98ca-ef87de6e3437\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz"
Oct 30 07:12:21.132577 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:21.132549    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac672938-faf4-44e5-98ca-ef87de6e3437-istio-podinfo\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz\" (UID: \"ac672938-faf4-44e5-98ca-ef87de6e3437\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz"
Oct 30 07:12:21.133960 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:21.133915    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac672938-faf4-44e5-98ca-ef87de6e3437-workload-certs\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz\" (UID: \"ac672938-faf4-44e5-98ca-ef87de6e3437\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz"
Oct 30 07:12:21.134065 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:21.134039    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac672938-faf4-44e5-98ca-ef87de6e3437-credential-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz\" (UID: \"ac672938-faf4-44e5-98ca-ef87de6e3437\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz"
Oct 30 07:12:21.135993 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:21.135970    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac672938-faf4-44e5-98ca-ef87de6e3437-istio-envoy\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz\" (UID: \"ac672938-faf4-44e5-98ca-ef87de6e3437\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz"
Oct 30 07:12:21.155055 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:21.154990    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac672938-faf4-44e5-98ca-ef87de6e3437-istio-token\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz\" (UID: \"ac672938-faf4-44e5-98ca-ef87de6e3437\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz"
Oct 30 07:12:21.155348 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:21.155331    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb75n\" (UniqueName: \"kubernetes.io/projected/ac672938-faf4-44e5-98ca-ef87de6e3437-kube-api-access-kb75n\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz\" (UID: \"ac672938-faf4-44e5-98ca-ef87de6e3437\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz"
Oct 30 07:12:21.221254 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:21.221227    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz"
Oct 30 07:12:21.277371 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:21.277004    6224 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 07:12:21.277371 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:21.277078    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq" podUID="717ebd45-f957-47c5-a710-d091051fa4eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 07:12:21.449595 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:21.449132    6224 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 30 07:12:21.449595 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:21.449188    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2" podUID="3a24e154-72f7-48d6-9d93-2f9fee0de386" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 30 07:12:21.564046 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:21.564011    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz"]
Oct 30 07:12:21.580231 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:21.580173    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:12:21.693472 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:21.693423    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:12:21.693675 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:21.693517    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:12:21.693675 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:21.693560    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:12:22.265745 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:22.265703    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz" event={"ID":"ac672938-faf4-44e5-98ca-ef87de6e3437","Type":"ContainerStarted","Data":"a564e6bd67a47cf2c049350c73939efe44422062ba3d2789c423e06a55320524"}
Oct 30 07:12:22.266072 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:22.266044    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz" event={"ID":"ac672938-faf4-44e5-98ca-ef87de6e3437","Type":"ContainerStarted","Data":"348e35c907071667cf9297001a361e5e15463c6c107b7a92413692453350074f"}
Oct 30 07:12:22.410829 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:22.403119    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq"
Oct 30 07:12:22.410829 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:22.404595    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq"
Oct 30 07:12:22.415765 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:22.415099    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq"
Oct 30 07:12:22.458165 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:22.458119    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz" podStartSLOduration=2.458106685 podStartE2EDuration="2.458106685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:12:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:12:22.330170076 +0000 UTC m=+900.939577048" watchObservedRunningTime="2026-10-30 07:12:22.458106685 +0000 UTC m=+901.067513655"
Oct 30 07:12:22.616742 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:22.616714    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2"
Oct 30 07:12:22.617394 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:22.617369    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2"
Oct 30 07:12:22.618369 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:22.618348    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2"
Oct 30 07:12:23.222829 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:23.222701    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz"
Oct 30 07:12:23.262410 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:23.262379    6224 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 07:12:23.262690 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:23.262659    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz" podUID="ac672938-faf4-44e5-98ca-ef87de6e3437" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 07:12:24.238832 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:24.238743    6224 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 07:12:24.238832 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:24.238820    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz" podUID="ac672938-faf4-44e5-98ca-ef87de6e3437" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 07:12:25.225838 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:25.218256    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb6be4/premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t" podUID="7dd5133c-480f-4fa0-9dd1-27662004aae9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Oct 30 07:12:25.233960 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:25.233904    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-06b26e/distinct-w0-06b26e-kserve-545cf57898-2fhhd" podUID="71294bfa-ac2c-40b7-b889-0527daa07f63" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:25.245836 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:25.234516    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb6be4/simulator-w2-bb6be4-kserve-8485bbf954-dnbbz" podUID="6370f7e2-d6c8-4fb5-8198-32b38837ca31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:25.273880 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:25.248569    6224 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 07:12:25.273880 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:25.248620    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz" podUID="ac672938-faf4-44e5-98ca-ef87de6e3437" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 07:12:25.273880 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:25.248685    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw" podUID="0597696e-26b8-4ef8-8697-64d49b2b9398" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:25.333841 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:25.328058    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2bab4c/distinct-w4-2bab4c-kserve-6868d767c9-j7wlm" podUID="11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:25.422781 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:25.422462    6224 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-2bab4c/distinct-w4-2bab4c-kserve-6868d767c9-j7wlm" podUID="11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:25.532878 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:25.528149    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2bab4c/simulator-w4-2bab4c-kserve-6db9444549-7nrk8" podUID="c7b09178-9ac7-492f-890b-5ff3da130981" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:25.755830 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:25.749453    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2bab4c/premium-w4-2bab4c-kserve-665887fc68-c6dgf" podUID="f9463476-fcf5-4d66-bfd2-b85ce4ed651f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Oct 30 07:12:25.935089 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:25.934987    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-06b26e/premium-w0-06b26e-kserve-c9dbd9879-b4jlk" podUID="1230dcae-e1cd-460d-86fc-bd40c8427b5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:26.227543 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:26.227459    6224 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 07:12:26.228243 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:26.228209    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz" podUID="ac672938-faf4-44e5-98ca-ef87de6e3437" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 07:12:27.221595 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:27.221562    6224 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 07:12:27.221964 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:27.221630    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz" podUID="ac672938-faf4-44e5-98ca-ef87de6e3437" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 07:12:28.223372 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:28.223332    6224 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 07:12:28.223788 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:28.223396    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz" podUID="ac672938-faf4-44e5-98ca-ef87de6e3437" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 07:12:29.188753 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:29.188718    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl"]
Oct 30 07:12:29.193234 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:29.193212    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl"
Oct 30 07:12:29.223104 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:29.223070    6224 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 07:12:29.223284 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:29.223156    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz" podUID="ac672938-faf4-44e5-98ca-ef87de6e3437" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 07:12:29.231065 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:29.231032    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl"]
Oct 30 07:12:29.231544 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:29.231521    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cb25921-0688-433d-90ec-e8ea626437b8-istio-envoy\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl\" (UID: \"9cb25921-0688-433d-90ec-e8ea626437b8\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl"
Oct 30 07:12:29.231613 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:29.231555    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cb25921-0688-433d-90ec-e8ea626437b8-istio-podinfo\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl\" (UID: \"9cb25921-0688-433d-90ec-e8ea626437b8\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl"
Oct 30 07:12:29.231613 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:29.231573    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cb25921-0688-433d-90ec-e8ea626437b8-istio-token\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl\" (UID: \"9cb25921-0688-433d-90ec-e8ea626437b8\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl"
Oct 30 07:12:29.231726 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:29.231642    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb25921-0688-433d-90ec-e8ea626437b8-credential-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl\" (UID: \"9cb25921-0688-433d-90ec-e8ea626437b8\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl"
Oct 30 07:12:29.231868 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:29.231799    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cb25921-0688-433d-90ec-e8ea626437b8-istio-data\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl\" (UID: \"9cb25921-0688-433d-90ec-e8ea626437b8\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl"
Oct 30 07:12:29.231986 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:29.231908    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljzmw\" (UniqueName: \"kubernetes.io/projected/9cb25921-0688-433d-90ec-e8ea626437b8-kube-api-access-ljzmw\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl\" (UID: \"9cb25921-0688-433d-90ec-e8ea626437b8\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl"
Oct 30 07:12:29.231986 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:29.231944    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cb25921-0688-433d-90ec-e8ea626437b8-workload-certs\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl\" (UID: \"9cb25921-0688-433d-90ec-e8ea626437b8\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl"
Oct 30 07:12:29.231986 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:29.231961    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cb25921-0688-433d-90ec-e8ea626437b8-istiod-ca-cert\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl\" (UID: \"9cb25921-0688-433d-90ec-e8ea626437b8\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl"
Oct 30 07:12:29.232171 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:29.232041    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb25921-0688-433d-90ec-e8ea626437b8-workload-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl\" (UID: \"9cb25921-0688-433d-90ec-e8ea626437b8\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl"
Oct 30 07:12:29.333020 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:29.332981    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cb25921-0688-433d-90ec-e8ea626437b8-istio-data\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl\" (UID: \"9cb25921-0688-433d-90ec-e8ea626437b8\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl"
Oct 30 07:12:29.333234 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:29.333087    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljzmw\" (UniqueName: \"kubernetes.io/projected/9cb25921-0688-433d-90ec-e8ea626437b8-kube-api-access-ljzmw\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl\" (UID: \"9cb25921-0688-433d-90ec-e8ea626437b8\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl"
Oct 30 07:12:29.333234 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:29.333151    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cb25921-0688-433d-90ec-e8ea626437b8-workload-certs\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl\" (UID: \"9cb25921-0688-433d-90ec-e8ea626437b8\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl"
Oct 30 07:12:29.333234 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:29.333176    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cb25921-0688-433d-90ec-e8ea626437b8-istiod-ca-cert\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl\" (UID: \"9cb25921-0688-433d-90ec-e8ea626437b8\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl"
Oct 30 07:12:29.333234 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:29.333229    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb25921-0688-433d-90ec-e8ea626437b8-workload-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl\" (UID: \"9cb25921-0688-433d-90ec-e8ea626437b8\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl"
Oct 30 07:12:29.333472 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:29.333290    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cb25921-0688-433d-90ec-e8ea626437b8-istio-envoy\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl\" (UID: \"9cb25921-0688-433d-90ec-e8ea626437b8\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl"
Oct 30 07:12:29.333472 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:29.333310    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cb25921-0688-433d-90ec-e8ea626437b8-istio-podinfo\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl\" (UID: \"9cb25921-0688-433d-90ec-e8ea626437b8\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl"
Oct 30 07:12:29.333472 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:29.333325    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cb25921-0688-433d-90ec-e8ea626437b8-istio-token\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl\" (UID: \"9cb25921-0688-433d-90ec-e8ea626437b8\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl"
Oct 30 07:12:29.333472 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:29.333349    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb25921-0688-433d-90ec-e8ea626437b8-credential-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl\" (UID: \"9cb25921-0688-433d-90ec-e8ea626437b8\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl"
Oct 30 07:12:29.333472 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:29.333460    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cb25921-0688-433d-90ec-e8ea626437b8-istio-data\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl\" (UID: \"9cb25921-0688-433d-90ec-e8ea626437b8\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl"
Oct 30 07:12:29.333743 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:29.333667    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb25921-0688-433d-90ec-e8ea626437b8-credential-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl\" (UID: \"9cb25921-0688-433d-90ec-e8ea626437b8\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl"
Oct 30 07:12:29.333918 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:29.333888    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cb25921-0688-433d-90ec-e8ea626437b8-workload-certs\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl\" (UID: \"9cb25921-0688-433d-90ec-e8ea626437b8\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl"
Oct 30 07:12:29.334060 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:29.333969    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cb25921-0688-433d-90ec-e8ea626437b8-istiod-ca-cert\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl\" (UID: \"9cb25921-0688-433d-90ec-e8ea626437b8\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl"
Oct 30 07:12:29.334125 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:29.334096    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb25921-0688-433d-90ec-e8ea626437b8-workload-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl\" (UID: \"9cb25921-0688-433d-90ec-e8ea626437b8\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl"
Oct 30 07:12:29.335865 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:29.335838    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cb25921-0688-433d-90ec-e8ea626437b8-istio-envoy\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl\" (UID: \"9cb25921-0688-433d-90ec-e8ea626437b8\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl"
Oct 30 07:12:29.336503 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:29.336476    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cb25921-0688-433d-90ec-e8ea626437b8-istio-podinfo\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl\" (UID: \"9cb25921-0688-433d-90ec-e8ea626437b8\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl"
Oct 30 07:12:29.359262 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:29.359230    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljzmw\" (UniqueName: \"kubernetes.io/projected/9cb25921-0688-433d-90ec-e8ea626437b8-kube-api-access-ljzmw\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl\" (UID: \"9cb25921-0688-433d-90ec-e8ea626437b8\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl"
Oct 30 07:12:29.359444 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:29.359413    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cb25921-0688-433d-90ec-e8ea626437b8-istio-token\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl\" (UID: \"9cb25921-0688-433d-90ec-e8ea626437b8\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl"
Oct 30 07:12:29.509408 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:29.509322    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl"
Oct 30 07:12:29.830796 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:12:29.830759    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cb25921_0688_433d_90ec_e8ea626437b8.slice/crio-a40fb3ff0297de113ad336b7313d3303d19e1e0b6cb4dd528ddbab125c79c9e3 WatchSource:0}: Error finding container a40fb3ff0297de113ad336b7313d3303d19e1e0b6cb4dd528ddbab125c79c9e3: Status 404 returned error can't find the container with id a40fb3ff0297de113ad336b7313d3303d19e1e0b6cb4dd528ddbab125c79c9e3
Oct 30 07:12:29.831487 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:29.831464    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl"]
Oct 30 07:12:29.833655 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:29.833618    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:12:29.833769 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:29.833681    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:12:29.833769 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:29.833714    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:12:29.927983 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:29.927933    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl" event={"ID":"9cb25921-0688-433d-90ec-e8ea626437b8","Type":"ContainerStarted","Data":"c7733c3c8cccb7a27ca9753c29eda16382457b61e2d23e953a4026530ee00515"}
Oct 30 07:12:29.928147 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:29.928001    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl" event={"ID":"9cb25921-0688-433d-90ec-e8ea626437b8","Type":"ContainerStarted","Data":"a40fb3ff0297de113ad336b7313d3303d19e1e0b6cb4dd528ddbab125c79c9e3"}
Oct 30 07:12:29.978759 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:29.978689    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl" podStartSLOduration=0.978669794 podStartE2EDuration="978.669794ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:12:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:12:29.977671494 +0000 UTC m=+908.587078487" watchObservedRunningTime="2026-10-30 07:12:29.978669794 +0000 UTC m=+908.588076764"
Oct 30 07:12:30.226691 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:30.226616    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz"
Oct 30 07:12:30.226953 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:30.226932    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz"
Oct 30 07:12:30.227985 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:30.227955    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz"
Oct 30 07:12:30.509866 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:30.509778    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl"
Oct 30 07:12:30.511186 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:30.511161    6224 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 30 07:12:30.511299 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:30.511209    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl" podUID="9cb25921-0688-433d-90ec-e8ea626437b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 30 07:12:31.411086 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:31.411050    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a163b2-5d955f44dd-z4xbd"]
Oct 30 07:12:31.415534 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:31.415515    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a163b2-5d955f44dd-z4xbd"
Oct 30 07:12:31.448387 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:31.448350    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-a163b2-serving-cert\""
Oct 30 07:12:31.451471 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:31.451442    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a163b2-5d955f44dd-z4xbd"]
Oct 30 07:12:31.493239 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:31.493198    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5175e233-cb13-4706-a43f-f51f03927a87-metrics-certs\") pod \"maas-api-e2e-api-a-a163b2-5d955f44dd-z4xbd\" (UID: \"5175e233-cb13-4706-a43f-f51f03927a87\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a163b2-5d955f44dd-z4xbd"
Oct 30 07:12:31.493573 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:31.493548    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5175e233-cb13-4706-a43f-f51f03927a87-openshift-service-ca\") pod \"maas-api-e2e-api-a-a163b2-5d955f44dd-z4xbd\" (UID: \"5175e233-cb13-4706-a43f-f51f03927a87\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a163b2-5d955f44dd-z4xbd"
Oct 30 07:12:31.493818 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:31.493749    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5175e233-cb13-4706-a43f-f51f03927a87-maas-api-tls\") pod \"maas-api-e2e-api-a-a163b2-5d955f44dd-z4xbd\" (UID: \"5175e233-cb13-4706-a43f-f51f03927a87\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a163b2-5d955f44dd-z4xbd"
Oct 30 07:12:31.493977 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:31.493958    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7km6v\" (UniqueName: \"kubernetes.io/projected/5175e233-cb13-4706-a43f-f51f03927a87-kube-api-access-7km6v\") pod \"maas-api-e2e-api-a-a163b2-5d955f44dd-z4xbd\" (UID: \"5175e233-cb13-4706-a43f-f51f03927a87\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a163b2-5d955f44dd-z4xbd"
Oct 30 07:12:31.509759 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:31.509729    6224 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 30 07:12:31.509908 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:31.509785    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl" podUID="9cb25921-0688-433d-90ec-e8ea626437b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 30 07:12:31.595788 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:31.595752    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5175e233-cb13-4706-a43f-f51f03927a87-metrics-certs\") pod \"maas-api-e2e-api-a-a163b2-5d955f44dd-z4xbd\" (UID: \"5175e233-cb13-4706-a43f-f51f03927a87\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a163b2-5d955f44dd-z4xbd"
Oct 30 07:12:31.595972 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:31.595953    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5175e233-cb13-4706-a43f-f51f03927a87-openshift-service-ca\") pod \"maas-api-e2e-api-a-a163b2-5d955f44dd-z4xbd\" (UID: \"5175e233-cb13-4706-a43f-f51f03927a87\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a163b2-5d955f44dd-z4xbd"
Oct 30 07:12:31.596081 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:31.596058    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5175e233-cb13-4706-a43f-f51f03927a87-maas-api-tls\") pod \"maas-api-e2e-api-a-a163b2-5d955f44dd-z4xbd\" (UID: \"5175e233-cb13-4706-a43f-f51f03927a87\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a163b2-5d955f44dd-z4xbd"
Oct 30 07:12:31.596243 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:31.596120    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7km6v\" (UniqueName: \"kubernetes.io/projected/5175e233-cb13-4706-a43f-f51f03927a87-kube-api-access-7km6v\") pod \"maas-api-e2e-api-a-a163b2-5d955f44dd-z4xbd\" (UID: \"5175e233-cb13-4706-a43f-f51f03927a87\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a163b2-5d955f44dd-z4xbd"
Oct 30 07:12:31.596672 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:31.596647    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5175e233-cb13-4706-a43f-f51f03927a87-openshift-service-ca\") pod \"maas-api-e2e-api-a-a163b2-5d955f44dd-z4xbd\" (UID: \"5175e233-cb13-4706-a43f-f51f03927a87\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a163b2-5d955f44dd-z4xbd"
Oct 30 07:12:31.598279 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:31.598256    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5175e233-cb13-4706-a43f-f51f03927a87-metrics-certs\") pod \"maas-api-e2e-api-a-a163b2-5d955f44dd-z4xbd\" (UID: \"5175e233-cb13-4706-a43f-f51f03927a87\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a163b2-5d955f44dd-z4xbd"
Oct 30 07:12:31.598352 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:31.598317    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5175e233-cb13-4706-a43f-f51f03927a87-maas-api-tls\") pod \"maas-api-e2e-api-a-a163b2-5d955f44dd-z4xbd\" (UID: \"5175e233-cb13-4706-a43f-f51f03927a87\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a163b2-5d955f44dd-z4xbd"
Oct 30 07:12:31.622132 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:31.622099    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7km6v\" (UniqueName: \"kubernetes.io/projected/5175e233-cb13-4706-a43f-f51f03927a87-kube-api-access-7km6v\") pod \"maas-api-e2e-api-a-a163b2-5d955f44dd-z4xbd\" (UID: \"5175e233-cb13-4706-a43f-f51f03927a87\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a163b2-5d955f44dd-z4xbd"
Oct 30 07:12:31.732072 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:31.731972    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a163b2-5d955f44dd-z4xbd"
Oct 30 07:12:31.941992 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:12:31.941958    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:12:32.091264 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:12:32.091228    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5175e233_cb13_4706_a43f_f51f03927a87.slice/crio-cbcf448a5e97a880979e7df63e06f982d98829cc635a06e82eba6d977fd7cfb8 WatchSource:0}: Error finding container cbcf448a5e97a880979e7df63e06f982d98829cc635a06e82eba6d977fd7cfb8: Status 404 returned error can't find the container with id cbcf448a5e97a880979e7df63e06f982d98829cc635a06e82eba6d977fd7cfb8
Oct 30 07:12:32.091391 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:32.091369    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a163b2-5d955f44dd-z4xbd"]
Oct 30 07:12:32.092790 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:32.092773    6224 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 07:12:32.510122 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:32.510089    6224 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 30 07:12:32.510477 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:32.510144    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl" podUID="9cb25921-0688-433d-90ec-e8ea626437b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 30 07:12:32.957253 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:32.957137    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a163b2-5d955f44dd-z4xbd" event={"ID":"5175e233-cb13-4706-a43f-f51f03927a87","Type":"ContainerStarted","Data":"d17cc38083e9d568bab7a393138767a69f2280a61d89e6153c3cfe223dc8f545"}
Oct 30 07:12:32.957253 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:32.957204    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a163b2-5d955f44dd-z4xbd" event={"ID":"5175e233-cb13-4706-a43f-f51f03927a87","Type":"ContainerStarted","Data":"cbcf448a5e97a880979e7df63e06f982d98829cc635a06e82eba6d977fd7cfb8"}
Oct 30 07:12:32.957947 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:32.957921    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a163b2-5d955f44dd-z4xbd"
Oct 30 07:12:33.509757 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:33.509719    6224 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 30 07:12:33.510055 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:33.509782    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl" podUID="9cb25921-0688-433d-90ec-e8ea626437b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 30 07:12:34.510174 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:34.510130    6224 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 30 07:12:34.510546 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:34.510188    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl" podUID="9cb25921-0688-433d-90ec-e8ea626437b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 30 07:12:35.510138 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:35.510106    6224 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 30 07:12:35.510347 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:35.510166    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl" podUID="9cb25921-0688-433d-90ec-e8ea626437b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 30 07:12:36.510609 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:36.510578    6224 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 30 07:12:36.511033 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:36.510626    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl" podUID="9cb25921-0688-433d-90ec-e8ea626437b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 30 07:12:37.514255 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:37.514220    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl"
Oct 30 07:12:37.514658 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:37.514464    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl"
Oct 30 07:12:37.515179 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:37.515162    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl"
Oct 30 07:12:37.600051 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:37.599990    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a163b2-5d955f44dd-z4xbd" podStartSLOduration=6.302874469 podStartE2EDuration="6.599975566s" totalImagesPullingTime="297.101097ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:12:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:12:33.021113683 +0000 UTC m=+911.630520655" watchObservedRunningTime="2026-10-30 07:12:37.599975566 +0000 UTC m=+916.209382536"
Oct 30 07:12:38.265327 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.265294    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lkxfq"]
Oct 30 07:12:38.269173 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.269150    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lkxfq"
Oct 30 07:12:38.282496 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.282467    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vzfkb"]
Oct 30 07:12:38.287145 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.287124    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vzfkb"
Oct 30 07:12:38.297779 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.297750    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lkxfq"]
Oct 30 07:12:38.311531 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.311506    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vzfkb"]
Oct 30 07:12:38.328549 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.328515    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0495c412-7a58-46b2-9ccc-c6abc6611c97-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-lkxfq\" (UID: \"0495c412-7a58-46b2-9ccc-c6abc6611c97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lkxfq"
Oct 30 07:12:38.328675 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.328559    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0495c412-7a58-46b2-9ccc-c6abc6611c97-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-lkxfq\" (UID: \"0495c412-7a58-46b2-9ccc-c6abc6611c97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lkxfq"
Oct 30 07:12:38.328675 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.328589    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0495c412-7a58-46b2-9ccc-c6abc6611c97-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-lkxfq\" (UID: \"0495c412-7a58-46b2-9ccc-c6abc6611c97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lkxfq"
Oct 30 07:12:38.328765 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.328699    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/233f4cde-e81e-4c82-8144-7bf56a77c876-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-vzfkb\" (UID: \"233f4cde-e81e-4c82-8144-7bf56a77c876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vzfkb"
Oct 30 07:12:38.328850 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.328792    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/233f4cde-e81e-4c82-8144-7bf56a77c876-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-vzfkb\" (UID: \"233f4cde-e81e-4c82-8144-7bf56a77c876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vzfkb"
Oct 30 07:12:38.328986 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.328963    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0495c412-7a58-46b2-9ccc-c6abc6611c97-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-lkxfq\" (UID: \"0495c412-7a58-46b2-9ccc-c6abc6611c97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lkxfq"
Oct 30 07:12:38.329034 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.329008    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgnhd\" (UniqueName: \"kubernetes.io/projected/233f4cde-e81e-4c82-8144-7bf56a77c876-kube-api-access-xgnhd\") pod \"maas-default-gateway-openshift-default-6dcf757899-vzfkb\" (UID: \"233f4cde-e81e-4c82-8144-7bf56a77c876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vzfkb"
Oct 30 07:12:38.329079 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.329048    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/233f4cde-e81e-4c82-8144-7bf56a77c876-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-vzfkb\" (UID: \"233f4cde-e81e-4c82-8144-7bf56a77c876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vzfkb"
Oct 30 07:12:38.329079 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.329073    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0495c412-7a58-46b2-9ccc-c6abc6611c97-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-lkxfq\" (UID: \"0495c412-7a58-46b2-9ccc-c6abc6611c97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lkxfq"
Oct 30 07:12:38.329196 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.329179    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tppff\" (UniqueName: \"kubernetes.io/projected/0495c412-7a58-46b2-9ccc-c6abc6611c97-kube-api-access-tppff\") pod \"maas-default-gateway-openshift-default-6dcf757899-lkxfq\" (UID: \"0495c412-7a58-46b2-9ccc-c6abc6611c97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lkxfq"
Oct 30 07:12:38.329293 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.329278    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/233f4cde-e81e-4c82-8144-7bf56a77c876-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-vzfkb\" (UID: \"233f4cde-e81e-4c82-8144-7bf56a77c876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vzfkb"
Oct 30 07:12:38.329331 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.329308    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/233f4cde-e81e-4c82-8144-7bf56a77c876-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-vzfkb\" (UID: \"233f4cde-e81e-4c82-8144-7bf56a77c876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vzfkb"
Oct 30 07:12:38.329371 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.329338    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0495c412-7a58-46b2-9ccc-c6abc6611c97-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-lkxfq\" (UID: \"0495c412-7a58-46b2-9ccc-c6abc6611c97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lkxfq"
Oct 30 07:12:38.329441 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.329420    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0495c412-7a58-46b2-9ccc-c6abc6611c97-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-lkxfq\" (UID: \"0495c412-7a58-46b2-9ccc-c6abc6611c97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lkxfq"
Oct 30 07:12:38.329531 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.329516    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/233f4cde-e81e-4c82-8144-7bf56a77c876-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-vzfkb\" (UID: \"233f4cde-e81e-4c82-8144-7bf56a77c876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vzfkb"
Oct 30 07:12:38.329579 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.329547    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0495c412-7a58-46b2-9ccc-c6abc6611c97-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-lkxfq\" (UID: \"0495c412-7a58-46b2-9ccc-c6abc6611c97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lkxfq"
Oct 30 07:12:38.329622 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.329612    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/233f4cde-e81e-4c82-8144-7bf56a77c876-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-vzfkb\" (UID: \"233f4cde-e81e-4c82-8144-7bf56a77c876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vzfkb"
Oct 30 07:12:38.329677 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.329659    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/233f4cde-e81e-4c82-8144-7bf56a77c876-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-vzfkb\" (UID: \"233f4cde-e81e-4c82-8144-7bf56a77c876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vzfkb"
Oct 30 07:12:38.430357 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.430320    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/233f4cde-e81e-4c82-8144-7bf56a77c876-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-vzfkb\" (UID: \"233f4cde-e81e-4c82-8144-7bf56a77c876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vzfkb"
Oct 30 07:12:38.430357 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.430358    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0495c412-7a58-46b2-9ccc-c6abc6611c97-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-lkxfq\" (UID: \"0495c412-7a58-46b2-9ccc-c6abc6611c97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lkxfq"
Oct 30 07:12:38.430598 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.430407    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tppff\" (UniqueName: \"kubernetes.io/projected/0495c412-7a58-46b2-9ccc-c6abc6611c97-kube-api-access-tppff\") pod \"maas-default-gateway-openshift-default-6dcf757899-lkxfq\" (UID: \"0495c412-7a58-46b2-9ccc-c6abc6611c97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lkxfq"
Oct 30 07:12:38.430598 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.430482    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/233f4cde-e81e-4c82-8144-7bf56a77c876-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-vzfkb\" (UID: \"233f4cde-e81e-4c82-8144-7bf56a77c876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vzfkb"
Oct 30 07:12:38.430598 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.430505    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/233f4cde-e81e-4c82-8144-7bf56a77c876-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-vzfkb\" (UID: \"233f4cde-e81e-4c82-8144-7bf56a77c876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vzfkb"
Oct 30 07:12:38.430598 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.430535    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0495c412-7a58-46b2-9ccc-c6abc6611c97-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-lkxfq\" (UID: \"0495c412-7a58-46b2-9ccc-c6abc6611c97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lkxfq"
Oct 30 07:12:38.430598 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.430561    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0495c412-7a58-46b2-9ccc-c6abc6611c97-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-lkxfq\" (UID: \"0495c412-7a58-46b2-9ccc-c6abc6611c97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lkxfq"
Oct 30 07:12:38.430882 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.430610    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/233f4cde-e81e-4c82-8144-7bf56a77c876-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-vzfkb\" (UID: \"233f4cde-e81e-4c82-8144-7bf56a77c876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vzfkb"
Oct 30 07:12:38.430882 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.430639    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0495c412-7a58-46b2-9ccc-c6abc6611c97-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-lkxfq\" (UID: \"0495c412-7a58-46b2-9ccc-c6abc6611c97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lkxfq"
Oct 30 07:12:38.430882 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.430697    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/233f4cde-e81e-4c82-8144-7bf56a77c876-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-vzfkb\" (UID: \"233f4cde-e81e-4c82-8144-7bf56a77c876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vzfkb"
Oct 30 07:12:38.430882 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.430731    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/233f4cde-e81e-4c82-8144-7bf56a77c876-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-vzfkb\" (UID: \"233f4cde-e81e-4c82-8144-7bf56a77c876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vzfkb"
Oct 30 07:12:38.430882 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.430771    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0495c412-7a58-46b2-9ccc-c6abc6611c97-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-lkxfq\" (UID: \"0495c412-7a58-46b2-9ccc-c6abc6611c97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lkxfq"
Oct 30 07:12:38.430882 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.430798    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0495c412-7a58-46b2-9ccc-c6abc6611c97-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-lkxfq\" (UID: \"0495c412-7a58-46b2-9ccc-c6abc6611c97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lkxfq"
Oct 30 07:12:38.430882 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.430830    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0495c412-7a58-46b2-9ccc-c6abc6611c97-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-lkxfq\" (UID: \"0495c412-7a58-46b2-9ccc-c6abc6611c97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lkxfq"
Oct 30 07:12:38.430882 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.430846    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0495c412-7a58-46b2-9ccc-c6abc6611c97-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-lkxfq\" (UID: \"0495c412-7a58-46b2-9ccc-c6abc6611c97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lkxfq"
Oct 30 07:12:38.431283 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.430902    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/233f4cde-e81e-4c82-8144-7bf56a77c876-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-vzfkb\" (UID: \"233f4cde-e81e-4c82-8144-7bf56a77c876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vzfkb"
Oct 30 07:12:38.431283 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.430911    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/233f4cde-e81e-4c82-8144-7bf56a77c876-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-vzfkb\" (UID: \"233f4cde-e81e-4c82-8144-7bf56a77c876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vzfkb"
Oct 30 07:12:38.431283 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.430924    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/233f4cde-e81e-4c82-8144-7bf56a77c876-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-vzfkb\" (UID: \"233f4cde-e81e-4c82-8144-7bf56a77c876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vzfkb"
Oct 30 07:12:38.431283 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.430900    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0495c412-7a58-46b2-9ccc-c6abc6611c97-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-lkxfq\" (UID: \"0495c412-7a58-46b2-9ccc-c6abc6611c97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lkxfq"
Oct 30 07:12:38.431283 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.431037    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/233f4cde-e81e-4c82-8144-7bf56a77c876-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-vzfkb\" (UID: \"233f4cde-e81e-4c82-8144-7bf56a77c876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vzfkb"
Oct 30 07:12:38.431283 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.431116    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0495c412-7a58-46b2-9ccc-c6abc6611c97-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-lkxfq\" (UID: \"0495c412-7a58-46b2-9ccc-c6abc6611c97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lkxfq"
Oct 30 07:12:38.431283 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.431152    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0495c412-7a58-46b2-9ccc-c6abc6611c97-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-lkxfq\" (UID: \"0495c412-7a58-46b2-9ccc-c6abc6611c97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lkxfq"
Oct 30 07:12:38.431283 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.431154    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgnhd\" (UniqueName: \"kubernetes.io/projected/233f4cde-e81e-4c82-8144-7bf56a77c876-kube-api-access-xgnhd\") pod \"maas-default-gateway-openshift-default-6dcf757899-vzfkb\" (UID: \"233f4cde-e81e-4c82-8144-7bf56a77c876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vzfkb"
Oct 30 07:12:38.431283 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.431151    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/233f4cde-e81e-4c82-8144-7bf56a77c876-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-vzfkb\" (UID: \"233f4cde-e81e-4c82-8144-7bf56a77c876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vzfkb"
Oct 30 07:12:38.431685 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.431511    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0495c412-7a58-46b2-9ccc-c6abc6611c97-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-lkxfq\" (UID: \"0495c412-7a58-46b2-9ccc-c6abc6611c97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lkxfq"
Oct 30 07:12:38.431685 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.431655    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0495c412-7a58-46b2-9ccc-c6abc6611c97-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-lkxfq\" (UID: \"0495c412-7a58-46b2-9ccc-c6abc6611c97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lkxfq"
Oct 30 07:12:38.431795 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.431701    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/233f4cde-e81e-4c82-8144-7bf56a77c876-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-vzfkb\" (UID: \"233f4cde-e81e-4c82-8144-7bf56a77c876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vzfkb"
Oct 30 07:12:38.431946 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.431928    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/233f4cde-e81e-4c82-8144-7bf56a77c876-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-vzfkb\" (UID: \"233f4cde-e81e-4c82-8144-7bf56a77c876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vzfkb"
Oct 30 07:12:38.433243 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.433220    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/233f4cde-e81e-4c82-8144-7bf56a77c876-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-vzfkb\" (UID: \"233f4cde-e81e-4c82-8144-7bf56a77c876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vzfkb"
Oct 30 07:12:38.433511 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.433492    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0495c412-7a58-46b2-9ccc-c6abc6611c97-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-lkxfq\" (UID: \"0495c412-7a58-46b2-9ccc-c6abc6611c97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lkxfq"
Oct 30 07:12:38.433643 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.433628    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/233f4cde-e81e-4c82-8144-7bf56a77c876-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-vzfkb\" (UID: \"233f4cde-e81e-4c82-8144-7bf56a77c876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vzfkb"
Oct 30 07:12:38.434038 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.434019    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0495c412-7a58-46b2-9ccc-c6abc6611c97-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-lkxfq\" (UID: \"0495c412-7a58-46b2-9ccc-c6abc6611c97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lkxfq"
Oct 30 07:12:38.456934 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.456906    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0495c412-7a58-46b2-9ccc-c6abc6611c97-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-lkxfq\" (UID: \"0495c412-7a58-46b2-9ccc-c6abc6611c97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lkxfq"
Oct 30 07:12:38.457074 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.457001    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tppff\" (UniqueName: \"kubernetes.io/projected/0495c412-7a58-46b2-9ccc-c6abc6611c97-kube-api-access-tppff\") pod \"maas-default-gateway-openshift-default-6dcf757899-lkxfq\" (UID: \"0495c412-7a58-46b2-9ccc-c6abc6611c97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lkxfq"
Oct 30 07:12:38.457253 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.457226    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/233f4cde-e81e-4c82-8144-7bf56a77c876-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-vzfkb\" (UID: \"233f4cde-e81e-4c82-8144-7bf56a77c876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vzfkb"
Oct 30 07:12:38.457498 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.457481    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgnhd\" (UniqueName: \"kubernetes.io/projected/233f4cde-e81e-4c82-8144-7bf56a77c876-kube-api-access-xgnhd\") pod \"maas-default-gateway-openshift-default-6dcf757899-vzfkb\" (UID: \"233f4cde-e81e-4c82-8144-7bf56a77c876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vzfkb"
Oct 30 07:12:38.586588 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.586550    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lkxfq"
Oct 30 07:12:38.609428 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.609400    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vzfkb"
Oct 30 07:12:38.902550 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:12:38.902523    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0495c412_7a58_46b2_9ccc_c6abc6611c97.slice/crio-dcece42a068a286cae899ea09473f1eda3f65e08829ea49b72f5898d3bc14abb WatchSource:0}: Error finding container dcece42a068a286cae899ea09473f1eda3f65e08829ea49b72f5898d3bc14abb: Status 404 returned error can't find the container with id dcece42a068a286cae899ea09473f1eda3f65e08829ea49b72f5898d3bc14abb
Oct 30 07:12:38.903515 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.903492    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lkxfq"]
Oct 30 07:12:38.905071 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.905039    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:12:38.905129 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.905103    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:12:38.905170 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.905131    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:12:38.939096 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:12:38.939070    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod233f4cde_e81e_4c82_8144_7bf56a77c876.slice/crio-59cd68cace3f580384f4b80fadd9549a7a4899d366b54004e7c7c1e373a5fda4 WatchSource:0}: Error finding container 59cd68cace3f580384f4b80fadd9549a7a4899d366b54004e7c7c1e373a5fda4: Status 404 returned error can't find the container with id 59cd68cace3f580384f4b80fadd9549a7a4899d366b54004e7c7c1e373a5fda4
Oct 30 07:12:38.940698 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.940676    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vzfkb"]
Oct 30 07:12:38.941554 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.941527    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:12:38.941648 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.941593    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:12:38.941648 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:38.941635    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:12:39.027234 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:39.026446    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vzfkb" event={"ID":"233f4cde-e81e-4c82-8144-7bf56a77c876","Type":"ContainerStarted","Data":"9bcacd625bef8e858de20e58316b6ff6856458affc8d1ef833e34dada225e9a6"}
Oct 30 07:12:39.027234 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:39.026509    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vzfkb" event={"ID":"233f4cde-e81e-4c82-8144-7bf56a77c876","Type":"ContainerStarted","Data":"59cd68cace3f580384f4b80fadd9549a7a4899d366b54004e7c7c1e373a5fda4"}
Oct 30 07:12:39.029085 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:39.028991    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lkxfq" event={"ID":"0495c412-7a58-46b2-9ccc-c6abc6611c97","Type":"ContainerStarted","Data":"bf9cdb7c8c3fa411d77858ef931ee15818c0e1eee3e798b15572302eabd9ae2a"}
Oct 30 07:12:39.029085 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:39.029046    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lkxfq" event={"ID":"0495c412-7a58-46b2-9ccc-c6abc6611c97","Type":"ContainerStarted","Data":"dcece42a068a286cae899ea09473f1eda3f65e08829ea49b72f5898d3bc14abb"}
Oct 30 07:12:39.077377 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:39.077324    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vzfkb" podStartSLOduration=1.077310421 podStartE2EDuration="1.077310421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:12:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:12:39.075595738 +0000 UTC m=+917.685002708" watchObservedRunningTime="2026-10-30 07:12:39.077310421 +0000 UTC m=+917.686717390"
Oct 30 07:12:39.128727 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:39.128663    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lkxfq" podStartSLOduration=1.128649203 podStartE2EDuration="1.128649203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:12:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:12:39.124612895 +0000 UTC m=+917.734019866" watchObservedRunningTime="2026-10-30 07:12:39.128649203 +0000 UTC m=+917.738056173"
Oct 30 07:12:39.588829 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:39.588246    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lkxfq"
Oct 30 07:12:39.596839 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:39.590554    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-lkxfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 07:12:39.596839 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:39.590615    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lkxfq" podUID="0495c412-7a58-46b2-9ccc-c6abc6611c97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 07:12:39.614065 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:39.614034    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vzfkb"
Oct 30 07:12:39.630603 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:39.614800    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-vzfkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 07:12:39.630603 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:39.614888    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vzfkb" podUID="233f4cde-e81e-4c82-8144-7bf56a77c876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 07:12:40.068828 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:40.068021    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a163b2-5d955f44dd-z4xbd"
Oct 30 07:12:40.594833 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:40.591583    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-lkxfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 07:12:40.594833 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:40.591637    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lkxfq" podUID="0495c412-7a58-46b2-9ccc-c6abc6611c97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 07:12:40.612834 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:40.612635    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-vzfkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 07:12:40.612834 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:40.612690    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vzfkb" podUID="233f4cde-e81e-4c82-8144-7bf56a77c876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 07:12:40.940876 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:40.932861    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-a6a14e07/e2e-lifecycle-model-a6a14e07-kserve-5fddf987df-g5t8r"]
Oct 30 07:12:40.990835 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:40.990373    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-a6a14e07/e2e-lifecycle-model-a6a14e07-kserve-5fddf987df-g5t8r"]
Oct 30 07:12:40.990835 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:40.990501    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-a6a14e07/e2e-lifecycle-model-a6a14e07-kserve-5fddf987df-g5t8r"
Oct 30 07:12:41.072961 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:41.069203    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-a6a14e07\"/\"default-dockercfg-fw6x7\""
Oct 30 07:12:41.072961 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:41.069413    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-a6a14e07\"/\"kube-root-ca.crt\""
Oct 30 07:12:41.101868 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:41.082982    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-a6a14e07\"/\"openshift-service-ca.crt\""
Oct 30 07:12:41.101868 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:41.083164    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-a6a14e07\"/\"e2e-lifecycle-model-a6a14e07-kserve-self-signed-certs\""
Oct 30 07:12:41.208888 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:41.208343    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf96p\" (UniqueName: \"kubernetes.io/projected/9e157c86-b1bc-4b22-9a7a-2c75044d65db-kube-api-access-qf96p\") pod \"e2e-lifecycle-model-a6a14e07-kserve-5fddf987df-g5t8r\" (UID: \"9e157c86-b1bc-4b22-9a7a-2c75044d65db\") " pod="ai-tenant-e2e-mt-a6a14e07/e2e-lifecycle-model-a6a14e07-kserve-5fddf987df-g5t8r"
Oct 30 07:12:41.208888 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:41.208395    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e157c86-b1bc-4b22-9a7a-2c75044d65db-dshm\") pod \"e2e-lifecycle-model-a6a14e07-kserve-5fddf987df-g5t8r\" (UID: \"9e157c86-b1bc-4b22-9a7a-2c75044d65db\") " pod="ai-tenant-e2e-mt-a6a14e07/e2e-lifecycle-model-a6a14e07-kserve-5fddf987df-g5t8r"
Oct 30 07:12:41.208888 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:41.208423    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e157c86-b1bc-4b22-9a7a-2c75044d65db-model-cache\") pod \"e2e-lifecycle-model-a6a14e07-kserve-5fddf987df-g5t8r\" (UID: \"9e157c86-b1bc-4b22-9a7a-2c75044d65db\") " pod="ai-tenant-e2e-mt-a6a14e07/e2e-lifecycle-model-a6a14e07-kserve-5fddf987df-g5t8r"
Oct 30 07:12:41.208888 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:41.208501    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e157c86-b1bc-4b22-9a7a-2c75044d65db-tmp-dir\") pod \"e2e-lifecycle-model-a6a14e07-kserve-5fddf987df-g5t8r\" (UID: \"9e157c86-b1bc-4b22-9a7a-2c75044d65db\") " pod="ai-tenant-e2e-mt-a6a14e07/e2e-lifecycle-model-a6a14e07-kserve-5fddf987df-g5t8r"
Oct 30 07:12:41.208888 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:41.208622    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e157c86-b1bc-4b22-9a7a-2c75044d65db-home\") pod \"e2e-lifecycle-model-a6a14e07-kserve-5fddf987df-g5t8r\" (UID: \"9e157c86-b1bc-4b22-9a7a-2c75044d65db\") " pod="ai-tenant-e2e-mt-a6a14e07/e2e-lifecycle-model-a6a14e07-kserve-5fddf987df-g5t8r"
Oct 30 07:12:41.208888 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:41.208665    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e157c86-b1bc-4b22-9a7a-2c75044d65db-tls-certs\") pod \"e2e-lifecycle-model-a6a14e07-kserve-5fddf987df-g5t8r\" (UID: \"9e157c86-b1bc-4b22-9a7a-2c75044d65db\") " pod="ai-tenant-e2e-mt-a6a14e07/e2e-lifecycle-model-a6a14e07-kserve-5fddf987df-g5t8r"
Oct 30 07:12:41.314834 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:41.313601    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e157c86-b1bc-4b22-9a7a-2c75044d65db-home\") pod \"e2e-lifecycle-model-a6a14e07-kserve-5fddf987df-g5t8r\" (UID: \"9e157c86-b1bc-4b22-9a7a-2c75044d65db\") " pod="ai-tenant-e2e-mt-a6a14e07/e2e-lifecycle-model-a6a14e07-kserve-5fddf987df-g5t8r"
Oct 30 07:12:41.314834 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:41.313664    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e157c86-b1bc-4b22-9a7a-2c75044d65db-tls-certs\") pod \"e2e-lifecycle-model-a6a14e07-kserve-5fddf987df-g5t8r\" (UID: \"9e157c86-b1bc-4b22-9a7a-2c75044d65db\") " pod="ai-tenant-e2e-mt-a6a14e07/e2e-lifecycle-model-a6a14e07-kserve-5fddf987df-g5t8r"
Oct 30 07:12:41.314834 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:41.313851    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf96p\" (UniqueName: \"kubernetes.io/projected/9e157c86-b1bc-4b22-9a7a-2c75044d65db-kube-api-access-qf96p\") pod \"e2e-lifecycle-model-a6a14e07-kserve-5fddf987df-g5t8r\" (UID: \"9e157c86-b1bc-4b22-9a7a-2c75044d65db\") " pod="ai-tenant-e2e-mt-a6a14e07/e2e-lifecycle-model-a6a14e07-kserve-5fddf987df-g5t8r"
Oct 30 07:12:41.314834 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:41.313896    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e157c86-b1bc-4b22-9a7a-2c75044d65db-dshm\") pod \"e2e-lifecycle-model-a6a14e07-kserve-5fddf987df-g5t8r\" (UID: \"9e157c86-b1bc-4b22-9a7a-2c75044d65db\") " pod="ai-tenant-e2e-mt-a6a14e07/e2e-lifecycle-model-a6a14e07-kserve-5fddf987df-g5t8r"
Oct 30 07:12:41.314834 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:41.313920    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e157c86-b1bc-4b22-9a7a-2c75044d65db-model-cache\") pod \"e2e-lifecycle-model-a6a14e07-kserve-5fddf987df-g5t8r\" (UID: \"9e157c86-b1bc-4b22-9a7a-2c75044d65db\") " pod="ai-tenant-e2e-mt-a6a14e07/e2e-lifecycle-model-a6a14e07-kserve-5fddf987df-g5t8r"
Oct 30 07:12:41.314834 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:41.314022    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e157c86-b1bc-4b22-9a7a-2c75044d65db-tmp-dir\") pod \"e2e-lifecycle-model-a6a14e07-kserve-5fddf987df-g5t8r\" (UID: \"9e157c86-b1bc-4b22-9a7a-2c75044d65db\") " pod="ai-tenant-e2e-mt-a6a14e07/e2e-lifecycle-model-a6a14e07-kserve-5fddf987df-g5t8r"
Oct 30 07:12:41.314834 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:41.314548    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e157c86-b1bc-4b22-9a7a-2c75044d65db-tmp-dir\") pod \"e2e-lifecycle-model-a6a14e07-kserve-5fddf987df-g5t8r\" (UID: \"9e157c86-b1bc-4b22-9a7a-2c75044d65db\") " pod="ai-tenant-e2e-mt-a6a14e07/e2e-lifecycle-model-a6a14e07-kserve-5fddf987df-g5t8r"
Oct 30 07:12:41.325885 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:41.315514    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e157c86-b1bc-4b22-9a7a-2c75044d65db-home\") pod \"e2e-lifecycle-model-a6a14e07-kserve-5fddf987df-g5t8r\" (UID: \"9e157c86-b1bc-4b22-9a7a-2c75044d65db\") " pod="ai-tenant-e2e-mt-a6a14e07/e2e-lifecycle-model-a6a14e07-kserve-5fddf987df-g5t8r"
Oct 30 07:12:41.329282 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:41.329249    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e157c86-b1bc-4b22-9a7a-2c75044d65db-model-cache\") pod \"e2e-lifecycle-model-a6a14e07-kserve-5fddf987df-g5t8r\" (UID: \"9e157c86-b1bc-4b22-9a7a-2c75044d65db\") " pod="ai-tenant-e2e-mt-a6a14e07/e2e-lifecycle-model-a6a14e07-kserve-5fddf987df-g5t8r"
Oct 30 07:12:41.391499 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:41.391466    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e157c86-b1bc-4b22-9a7a-2c75044d65db-tls-certs\") pod \"e2e-lifecycle-model-a6a14e07-kserve-5fddf987df-g5t8r\" (UID: \"9e157c86-b1bc-4b22-9a7a-2c75044d65db\") " pod="ai-tenant-e2e-mt-a6a14e07/e2e-lifecycle-model-a6a14e07-kserve-5fddf987df-g5t8r"
Oct 30 07:12:41.392396 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:41.392353    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e157c86-b1bc-4b22-9a7a-2c75044d65db-dshm\") pod \"e2e-lifecycle-model-a6a14e07-kserve-5fddf987df-g5t8r\" (UID: \"9e157c86-b1bc-4b22-9a7a-2c75044d65db\") " pod="ai-tenant-e2e-mt-a6a14e07/e2e-lifecycle-model-a6a14e07-kserve-5fddf987df-g5t8r"
Oct 30 07:12:41.400443 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:41.400379    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf96p\" (UniqueName: \"kubernetes.io/projected/9e157c86-b1bc-4b22-9a7a-2c75044d65db-kube-api-access-qf96p\") pod \"e2e-lifecycle-model-a6a14e07-kserve-5fddf987df-g5t8r\" (UID: \"9e157c86-b1bc-4b22-9a7a-2c75044d65db\") " pod="ai-tenant-e2e-mt-a6a14e07/e2e-lifecycle-model-a6a14e07-kserve-5fddf987df-g5t8r"
Oct 30 07:12:41.413291 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:41.413050    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-a6a14e07/e2e-lifecycle-model-a6a14e07-kserve-5fddf987df-g5t8r"
Oct 30 07:12:41.592194 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:41.592153    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-lkxfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 07:12:41.592469 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:41.592442    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lkxfq" podUID="0495c412-7a58-46b2-9ccc-c6abc6611c97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 07:12:41.613137 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:41.613101    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-vzfkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 07:12:41.613704 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:41.613675    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vzfkb" podUID="233f4cde-e81e-4c82-8144-7bf56a77c876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 07:12:41.808955 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:41.808912    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2bab4c/premium-w4-2bab4c-kserve-665887fc68-c6dgf" podUID="f9463476-fcf5-4d66-bfd2-b85ce4ed651f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:41.924833 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:41.920161    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-06b26e/premium-w0-06b26e-kserve-c9dbd9879-b4jlk" podUID="1230dcae-e1cd-460d-86fc-bd40c8427b5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:41.947830 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:41.946721    6224 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-2bab4c/premium-w4-2bab4c-kserve-665887fc68-c6dgf" podUID="f9463476-fcf5-4d66-bfd2-b85ce4ed651f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:12:42.588720 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:42.588151    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-lkxfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 07:12:42.588720 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:42.588208    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lkxfq" podUID="0495c412-7a58-46b2-9ccc-c6abc6611c97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 07:12:42.629099 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:42.628901    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-vzfkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 07:12:42.629099 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:42.628964    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vzfkb" podUID="233f4cde-e81e-4c82-8144-7bf56a77c876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 07:12:42.922854 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:12:42.917287    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e157c86_b1bc_4b22_9a7a_2c75044d65db.slice/crio-d5df3814f9d1f05da66107bca282ead2e4252b6e4a67997d5d493343697a0232 WatchSource:0}: Error finding container d5df3814f9d1f05da66107bca282ead2e4252b6e4a67997d5d493343697a0232: Status 404 returned error can't find the container with id d5df3814f9d1f05da66107bca282ead2e4252b6e4a67997d5d493343697a0232
Oct 30 07:12:42.922854 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:42.917626    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-a6a14e07/e2e-lifecycle-model-a6a14e07-kserve-5fddf987df-g5t8r"]
Oct 30 07:12:43.317572 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:43.317532    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-a6a14e07/e2e-lifecycle-model-a6a14e07-kserve-5fddf987df-g5t8r" event={"ID":"9e157c86-b1bc-4b22-9a7a-2c75044d65db","Type":"ContainerStarted","Data":"07513f38466ed270a704b1ebf33e315ef8b8aaecb33cdfc91efe5129b7d0ab41"}
Oct 30 07:12:43.318631 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:43.318606    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-a6a14e07/e2e-lifecycle-model-a6a14e07-kserve-5fddf987df-g5t8r" event={"ID":"9e157c86-b1bc-4b22-9a7a-2c75044d65db","Type":"ContainerStarted","Data":"d5df3814f9d1f05da66107bca282ead2e4252b6e4a67997d5d493343697a0232"}
Oct 30 07:12:43.397829 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:43.394654    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-a6a14e07/e2e-lifecycle-model-a6a14e07-kserve-5fddf987df-g5t8r" podStartSLOduration=3.394634219 podStartE2EDuration="3.394634219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:12:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:12:43.362786182 +0000 UTC m=+921.972193153" watchObservedRunningTime="2026-10-30 07:12:43.394634219 +0000 UTC m=+922.004041195"
Oct 30 07:12:43.416424 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:43.416077    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-a6a14e07/e2e-lifecycle-model-a6a14e07-kserve-5fddf987df-g5t8r"
Oct 30 07:12:43.610721 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:43.600491    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-lkxfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 07:12:43.610721 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:43.600650    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lkxfq" podUID="0495c412-7a58-46b2-9ccc-c6abc6611c97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 07:12:43.624290 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:43.614217    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-vzfkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 07:12:43.624290 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:43.614284    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vzfkb" podUID="233f4cde-e81e-4c82-8144-7bf56a77c876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 07:12:44.597834 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:44.589191    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-lkxfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 07:12:44.597834 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:44.589255    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lkxfq" podUID="0495c412-7a58-46b2-9ccc-c6abc6611c97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 07:12:44.610905 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:44.610153    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-vzfkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 07:12:44.610905 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:44.610209    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vzfkb" podUID="233f4cde-e81e-4c82-8144-7bf56a77c876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 07:12:45.589207 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:45.589168    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-lkxfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 07:12:45.589510 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:45.589482    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lkxfq" podUID="0495c412-7a58-46b2-9ccc-c6abc6611c97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 07:12:45.613201 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:45.613168    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-vzfkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 07:12:45.613769 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:45.613732    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vzfkb" podUID="233f4cde-e81e-4c82-8144-7bf56a77c876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 07:12:46.587017 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:46.586977    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-lkxfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 07:12:46.587164 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:46.587045    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lkxfq" podUID="0495c412-7a58-46b2-9ccc-c6abc6611c97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 07:12:46.609905 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:46.609874    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-vzfkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 07:12:46.610129 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:46.610094    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vzfkb" podUID="233f4cde-e81e-4c82-8144-7bf56a77c876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 07:12:47.587178 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:47.587143    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-lkxfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 07:12:47.587560 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:47.587207    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lkxfq" podUID="0495c412-7a58-46b2-9ccc-c6abc6611c97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 07:12:47.610277 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:47.610241    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-vzfkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 07:12:47.610467 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:47.610299    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vzfkb" podUID="233f4cde-e81e-4c82-8144-7bf56a77c876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 07:12:48.591563 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:48.591526    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lkxfq"
Oct 30 07:12:48.592091 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:48.591939    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lkxfq"
Oct 30 07:12:48.592752 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:48.592718    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lkxfq"
Oct 30 07:12:48.677562 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:48.677522    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vzfkb"
Oct 30 07:12:48.678069 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:48.678042    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vzfkb"
Oct 30 07:12:48.678563 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:48.678541    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vzfkb"
Oct 30 07:12:51.414527 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:51.414495    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-a6a14e07/e2e-lifecycle-model-a6a14e07-kserve-5fddf987df-g5t8r"
Oct 30 07:12:51.427882 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:51.427857    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-a6a14e07/e2e-lifecycle-model-a6a14e07-kserve-5fddf987df-g5t8r"
Oct 30 07:12:52.405091 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:12:52.405056    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-a6a14e07/e2e-lifecycle-model-a6a14e07-kserve-5fddf987df-g5t8r"
Oct 30 07:13:01.812225 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:01.812183    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:13:01.903059 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:13:01.903013    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:13:13.565484 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:13.565447    6224 generic.go:356] "Generic (PLEG): container finished" podID="14cba440-5936-4d19-bc3d-6d3aabc70f38" containerID="423d154d81c95109780fe710d18c7badf83807d6bd2dcba26247f232cf62c783" exitCode=0
Oct 30 07:13:13.565963 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:13.565502    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv" event={"ID":"14cba440-5936-4d19-bc3d-6d3aabc70f38","Type":"ContainerDied","Data":"423d154d81c95109780fe710d18c7badf83807d6bd2dcba26247f232cf62c783"}
Oct 30 07:13:13.565963 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:13.565933    6224 scope.go:122] "RemoveContainer" containerID="423d154d81c95109780fe710d18c7badf83807d6bd2dcba26247f232cf62c783"
Oct 30 07:13:13.567639 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:13.567597    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:13:13.567745 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:13.567730    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:13:13.567816 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:13.567791    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:13:14.596991 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:14.596954    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv" event={"ID":"14cba440-5936-4d19-bc3d-6d3aabc70f38","Type":"ContainerStarted","Data":"48c3751f6a33598a039451a7f7e5f74070244415d3cb7db5db04112b200c05f0"}
Oct 30 07:13:15.313898 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:15.313863    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv"
Oct 30 07:13:15.314093 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:15.313991    6224 patch_prober.go:29] interesting pod/e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 30 07:13:15.314093 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:15.314043    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv" podUID="14cba440-5936-4d19-bc3d-6d3aabc70f38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 30 07:13:16.313577 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:16.313541    6224 patch_prober.go:29] interesting pod/e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 30 07:13:16.313996 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:16.313593    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv" podUID="14cba440-5936-4d19-bc3d-6d3aabc70f38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 30 07:13:17.314854 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:17.314817    6224 patch_prober.go:29] interesting pod/e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 30 07:13:17.315423 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:17.315386    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv" podUID="14cba440-5936-4d19-bc3d-6d3aabc70f38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 30 07:13:18.315642 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:18.314900    6224 patch_prober.go:29] interesting pod/e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 30 07:13:18.315642 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:18.314955    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv" podUID="14cba440-5936-4d19-bc3d-6d3aabc70f38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 30 07:13:19.313932 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:19.313888    6224 patch_prober.go:29] interesting pod/e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 30 07:13:19.314144 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:19.313956    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv" podUID="14cba440-5936-4d19-bc3d-6d3aabc70f38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 30 07:13:19.993762 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:19.993724    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7c7884-5d44bb7f-f9sdh"]
Oct 30 07:13:19.998051 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:19.998027    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c7884-5d44bb7f-f9sdh"
Oct 30 07:13:20.014953 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:20.014926    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-7c7884-serving-cert\""
Oct 30 07:13:20.016819 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:20.016781    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7c7884-5d44bb7f-f9sdh"]
Oct 30 07:13:20.070447 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:20.070416    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq9zk\" (UniqueName: \"kubernetes.io/projected/7877e784-053a-44d6-b681-ba3bdcb4110b-kube-api-access-wq9zk\") pod \"maas-api-e2e-api-b-7c7884-5d44bb7f-f9sdh\" (UID: \"7877e784-053a-44d6-b681-ba3bdcb4110b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c7884-5d44bb7f-f9sdh"
Oct 30 07:13:20.070585 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:20.070452    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7877e784-053a-44d6-b681-ba3bdcb4110b-metrics-certs\") pod \"maas-api-e2e-api-b-7c7884-5d44bb7f-f9sdh\" (UID: \"7877e784-053a-44d6-b681-ba3bdcb4110b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c7884-5d44bb7f-f9sdh"
Oct 30 07:13:20.070585 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:20.070551    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7877e784-053a-44d6-b681-ba3bdcb4110b-maas-api-tls\") pod \"maas-api-e2e-api-b-7c7884-5d44bb7f-f9sdh\" (UID: \"7877e784-053a-44d6-b681-ba3bdcb4110b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c7884-5d44bb7f-f9sdh"
Oct 30 07:13:20.070662 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:20.070599    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7877e784-053a-44d6-b681-ba3bdcb4110b-openshift-service-ca\") pod \"maas-api-e2e-api-b-7c7884-5d44bb7f-f9sdh\" (UID: \"7877e784-053a-44d6-b681-ba3bdcb4110b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c7884-5d44bb7f-f9sdh"
Oct 30 07:13:20.172454 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:20.172423    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq9zk\" (UniqueName: \"kubernetes.io/projected/7877e784-053a-44d6-b681-ba3bdcb4110b-kube-api-access-wq9zk\") pod \"maas-api-e2e-api-b-7c7884-5d44bb7f-f9sdh\" (UID: \"7877e784-053a-44d6-b681-ba3bdcb4110b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c7884-5d44bb7f-f9sdh"
Oct 30 07:13:20.172454 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:20.172459    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7877e784-053a-44d6-b681-ba3bdcb4110b-metrics-certs\") pod \"maas-api-e2e-api-b-7c7884-5d44bb7f-f9sdh\" (UID: \"7877e784-053a-44d6-b681-ba3bdcb4110b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c7884-5d44bb7f-f9sdh"
Oct 30 07:13:20.172670 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:20.172503    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7877e784-053a-44d6-b681-ba3bdcb4110b-maas-api-tls\") pod \"maas-api-e2e-api-b-7c7884-5d44bb7f-f9sdh\" (UID: \"7877e784-053a-44d6-b681-ba3bdcb4110b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c7884-5d44bb7f-f9sdh"
Oct 30 07:13:20.172670 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:20.172533    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7877e784-053a-44d6-b681-ba3bdcb4110b-openshift-service-ca\") pod \"maas-api-e2e-api-b-7c7884-5d44bb7f-f9sdh\" (UID: \"7877e784-053a-44d6-b681-ba3bdcb4110b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c7884-5d44bb7f-f9sdh"
Oct 30 07:13:20.172783 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:13:20.172681    6224 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-7c7884-serving-cert: secret "maas-api-e2e-api-b-7c7884-serving-cert" not found
Oct 30 07:13:20.172783 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:13:20.172761    6224 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7877e784-053a-44d6-b681-ba3bdcb4110b-maas-api-tls podName:7877e784-053a-44d6-b681-ba3bdcb4110b nodeName:}" failed. No retries permitted until 2026-10-30 07:13:20.672739042 +0000 UTC m=+959.282146011 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7877e784-053a-44d6-b681-ba3bdcb4110b-maas-api-tls") pod "maas-api-e2e-api-b-7c7884-5d44bb7f-f9sdh" (UID: "7877e784-053a-44d6-b681-ba3bdcb4110b") : secret "maas-api-e2e-api-b-7c7884-serving-cert" not found
Oct 30 07:13:20.173348 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:20.173323    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7877e784-053a-44d6-b681-ba3bdcb4110b-openshift-service-ca\") pod \"maas-api-e2e-api-b-7c7884-5d44bb7f-f9sdh\" (UID: \"7877e784-053a-44d6-b681-ba3bdcb4110b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c7884-5d44bb7f-f9sdh"
Oct 30 07:13:20.175040 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:20.175013    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7877e784-053a-44d6-b681-ba3bdcb4110b-metrics-certs\") pod \"maas-api-e2e-api-b-7c7884-5d44bb7f-f9sdh\" (UID: \"7877e784-053a-44d6-b681-ba3bdcb4110b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c7884-5d44bb7f-f9sdh"
Oct 30 07:13:20.200051 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:20.200023    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq9zk\" (UniqueName: \"kubernetes.io/projected/7877e784-053a-44d6-b681-ba3bdcb4110b-kube-api-access-wq9zk\") pod \"maas-api-e2e-api-b-7c7884-5d44bb7f-f9sdh\" (UID: \"7877e784-053a-44d6-b681-ba3bdcb4110b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c7884-5d44bb7f-f9sdh"
Oct 30 07:13:20.320613 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:20.320586    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv"
Oct 30 07:13:20.321036 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:20.321008    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv"
Oct 30 07:13:20.321633 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:20.321616    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv"
Oct 30 07:13:20.679278 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:20.679200    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7877e784-053a-44d6-b681-ba3bdcb4110b-maas-api-tls\") pod \"maas-api-e2e-api-b-7c7884-5d44bb7f-f9sdh\" (UID: \"7877e784-053a-44d6-b681-ba3bdcb4110b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c7884-5d44bb7f-f9sdh"
Oct 30 07:13:20.681618 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:20.681591    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7877e784-053a-44d6-b681-ba3bdcb4110b-maas-api-tls\") pod \"maas-api-e2e-api-b-7c7884-5d44bb7f-f9sdh\" (UID: \"7877e784-053a-44d6-b681-ba3bdcb4110b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c7884-5d44bb7f-f9sdh"
Oct 30 07:13:20.911782 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:20.911739    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c7884-5d44bb7f-f9sdh"
Oct 30 07:13:21.281193 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:21.271392    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7c7884-5d44bb7f-f9sdh"]
Oct 30 07:13:21.660585 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:21.660547    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c7884-5d44bb7f-f9sdh" event={"ID":"7877e784-053a-44d6-b681-ba3bdcb4110b","Type":"ContainerStarted","Data":"9ccf87fedd97d76a21ec011cef044f7efe80cd30d945b1c5845614e93d172c67"}
Oct 30 07:13:22.569098 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:22.569068    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:13:22.668885 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:22.668845    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c7884-5d44bb7f-f9sdh" event={"ID":"7877e784-053a-44d6-b681-ba3bdcb4110b","Type":"ContainerStarted","Data":"3674ac7abf36baf1e29a4c0865759eb3096ac892d66119710ebd7a0c69ef9d4c"}
Oct 30 07:13:22.669055 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:22.668979    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c7884-5d44bb7f-f9sdh"
Oct 30 07:13:22.740316 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:22.740270    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c7884-5d44bb7f-f9sdh" podStartSLOduration=3.206314914 podStartE2EDuration="3.740252632s" totalImagesPullingTime="533.937718ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:13:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:13:22.73854475 +0000 UTC m=+961.347951713" watchObservedRunningTime="2026-10-30 07:13:22.740252632 +0000 UTC m=+961.349659604"
Oct 30 07:13:28.684503 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:28.684470    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c7884-5d44bb7f-f9sdh"
Oct 30 07:13:31.857500 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:13:31.857462    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:13:44.449889 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:44.449856    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc"]
Oct 30 07:13:44.454475 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:44.454454    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc"
Oct 30 07:13:44.477076 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:44.477024    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc"]
Oct 30 07:13:44.506232 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:44.506202    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79fa648e-4151-4998-b954-fc46d9841b97-istiod-ca-cert\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc\" (UID: \"79fa648e-4151-4998-b954-fc46d9841b97\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc"
Oct 30 07:13:44.506400 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:44.506245    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79fa648e-4151-4998-b954-fc46d9841b97-istio-podinfo\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc\" (UID: \"79fa648e-4151-4998-b954-fc46d9841b97\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc"
Oct 30 07:13:44.506400 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:44.506382    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79fa648e-4151-4998-b954-fc46d9841b97-istio-token\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc\" (UID: \"79fa648e-4151-4998-b954-fc46d9841b97\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc"
Oct 30 07:13:44.506536 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:44.506452    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79fa648e-4151-4998-b954-fc46d9841b97-istio-data\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc\" (UID: \"79fa648e-4151-4998-b954-fc46d9841b97\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc"
Oct 30 07:13:44.506670 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:44.506645    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79fa648e-4151-4998-b954-fc46d9841b97-istio-envoy\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc\" (UID: \"79fa648e-4151-4998-b954-fc46d9841b97\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc"
Oct 30 07:13:44.506709 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:44.506690    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79fa648e-4151-4998-b954-fc46d9841b97-credential-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc\" (UID: \"79fa648e-4151-4998-b954-fc46d9841b97\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc"
Oct 30 07:13:44.506749 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:44.506722    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9pvn\" (UniqueName: \"kubernetes.io/projected/79fa648e-4151-4998-b954-fc46d9841b97-kube-api-access-t9pvn\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc\" (UID: \"79fa648e-4151-4998-b954-fc46d9841b97\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc"
Oct 30 07:13:44.506798 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:44.506766    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79fa648e-4151-4998-b954-fc46d9841b97-workload-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc\" (UID: \"79fa648e-4151-4998-b954-fc46d9841b97\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc"
Oct 30 07:13:44.506932 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:44.506915    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79fa648e-4151-4998-b954-fc46d9841b97-workload-certs\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc\" (UID: \"79fa648e-4151-4998-b954-fc46d9841b97\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc"
Oct 30 07:13:44.608372 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:44.608340    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79fa648e-4151-4998-b954-fc46d9841b97-istio-token\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc\" (UID: \"79fa648e-4151-4998-b954-fc46d9841b97\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc"
Oct 30 07:13:44.608537 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:44.608392    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79fa648e-4151-4998-b954-fc46d9841b97-istio-data\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc\" (UID: \"79fa648e-4151-4998-b954-fc46d9841b97\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc"
Oct 30 07:13:44.608537 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:44.608480    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79fa648e-4151-4998-b954-fc46d9841b97-istio-envoy\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc\" (UID: \"79fa648e-4151-4998-b954-fc46d9841b97\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc"
Oct 30 07:13:44.608537 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:44.608504    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79fa648e-4151-4998-b954-fc46d9841b97-credential-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc\" (UID: \"79fa648e-4151-4998-b954-fc46d9841b97\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc"
Oct 30 07:13:44.608537 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:44.608524    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9pvn\" (UniqueName: \"kubernetes.io/projected/79fa648e-4151-4998-b954-fc46d9841b97-kube-api-access-t9pvn\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc\" (UID: \"79fa648e-4151-4998-b954-fc46d9841b97\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc"
Oct 30 07:13:44.608739 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:44.608555    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79fa648e-4151-4998-b954-fc46d9841b97-workload-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc\" (UID: \"79fa648e-4151-4998-b954-fc46d9841b97\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc"
Oct 30 07:13:44.608739 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:44.608573    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79fa648e-4151-4998-b954-fc46d9841b97-workload-certs\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc\" (UID: \"79fa648e-4151-4998-b954-fc46d9841b97\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc"
Oct 30 07:13:44.608739 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:44.608628    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79fa648e-4151-4998-b954-fc46d9841b97-istiod-ca-cert\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc\" (UID: \"79fa648e-4151-4998-b954-fc46d9841b97\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc"
Oct 30 07:13:44.608739 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:44.608665    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79fa648e-4151-4998-b954-fc46d9841b97-istio-podinfo\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc\" (UID: \"79fa648e-4151-4998-b954-fc46d9841b97\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc"
Oct 30 07:13:44.608990 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:44.608966    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79fa648e-4151-4998-b954-fc46d9841b97-istio-data\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc\" (UID: \"79fa648e-4151-4998-b954-fc46d9841b97\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc"
Oct 30 07:13:44.609108 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:44.609086    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79fa648e-4151-4998-b954-fc46d9841b97-workload-certs\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc\" (UID: \"79fa648e-4151-4998-b954-fc46d9841b97\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc"
Oct 30 07:13:44.609165 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:44.609144    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79fa648e-4151-4998-b954-fc46d9841b97-workload-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc\" (UID: \"79fa648e-4151-4998-b954-fc46d9841b97\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc"
Oct 30 07:13:44.609204 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:44.609176    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79fa648e-4151-4998-b954-fc46d9841b97-credential-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc\" (UID: \"79fa648e-4151-4998-b954-fc46d9841b97\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc"
Oct 30 07:13:44.609492 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:44.609470    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79fa648e-4151-4998-b954-fc46d9841b97-istiod-ca-cert\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc\" (UID: \"79fa648e-4151-4998-b954-fc46d9841b97\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc"
Oct 30 07:13:44.611003 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:44.610981    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79fa648e-4151-4998-b954-fc46d9841b97-istio-envoy\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc\" (UID: \"79fa648e-4151-4998-b954-fc46d9841b97\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc"
Oct 30 07:13:44.611255 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:44.611235    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79fa648e-4151-4998-b954-fc46d9841b97-istio-podinfo\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc\" (UID: \"79fa648e-4151-4998-b954-fc46d9841b97\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc"
Oct 30 07:13:44.633595 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:44.633568    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9pvn\" (UniqueName: \"kubernetes.io/projected/79fa648e-4151-4998-b954-fc46d9841b97-kube-api-access-t9pvn\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc\" (UID: \"79fa648e-4151-4998-b954-fc46d9841b97\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc"
Oct 30 07:13:44.633745 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:44.633697    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79fa648e-4151-4998-b954-fc46d9841b97-istio-token\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc\" (UID: \"79fa648e-4151-4998-b954-fc46d9841b97\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc"
Oct 30 07:13:44.767649 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:44.767565    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc"
Oct 30 07:13:45.091148 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:13:45.091115    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79fa648e_4151_4998_b954_fc46d9841b97.slice/crio-7b7c8fa38bcf14d6573412161b5985d292fe3b810a4a92cb7b27d4a4a93c7bbb WatchSource:0}: Error finding container 7b7c8fa38bcf14d6573412161b5985d292fe3b810a4a92cb7b27d4a4a93c7bbb: Status 404 returned error can't find the container with id 7b7c8fa38bcf14d6573412161b5985d292fe3b810a4a92cb7b27d4a4a93c7bbb
Oct 30 07:13:45.092993 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:45.092939    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc"]
Oct 30 07:13:45.093790 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:45.093758    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:13:45.093867 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:45.093852    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:13:45.093904 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:45.093887    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:13:45.845487 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:45.840145    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc" event={"ID":"79fa648e-4151-4998-b954-fc46d9841b97","Type":"ContainerStarted","Data":"7c571c755f2c14a1b556679c7c9b250883d3784dd73c8aa9d214ab917cf63b71"}
Oct 30 07:13:45.845487 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:45.840218    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc" event={"ID":"79fa648e-4151-4998-b954-fc46d9841b97","Type":"ContainerStarted","Data":"7b7c8fa38bcf14d6573412161b5985d292fe3b810a4a92cb7b27d4a4a93c7bbb"}
Oct 30 07:13:45.920725 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:45.920673    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc" podStartSLOduration=1.920658066 podStartE2EDuration="1.920658066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:13:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:13:45.9047282 +0000 UTC m=+984.514135173" watchObservedRunningTime="2026-10-30 07:13:45.920658066 +0000 UTC m=+984.530065035"
Oct 30 07:13:46.783831 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:46.772144    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc"
Oct 30 07:13:46.787956 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:46.787920    6224 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 30 07:13:46.788097 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:46.787976    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc" podUID="79fa648e-4151-4998-b954-fc46d9841b97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 30 07:13:47.768554 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:47.768521    6224 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 30 07:13:47.769209 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:47.769178    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc" podUID="79fa648e-4151-4998-b954-fc46d9841b97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 30 07:13:48.772228 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:48.772188    6224 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 30 07:13:48.772870 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:48.772834    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc" podUID="79fa648e-4151-4998-b954-fc46d9841b97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 30 07:13:49.787832 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:49.781849    6224 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 30 07:13:49.787832 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:49.781911    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc" podUID="79fa648e-4151-4998-b954-fc46d9841b97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 30 07:13:50.288656 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:50.288617    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:13:50.425136 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:50.425105    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7430-032ba0"]
Oct 30 07:13:50.429133 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:50.429110    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7430-032ba0"
Oct 30 07:13:50.447481 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:50.447453    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5q8j4\""
Oct 30 07:13:50.450409 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:50.450384    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7430-032ba0"]
Oct 30 07:13:50.524257 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:50.524220    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf2pw\" (UniqueName: \"kubernetes.io/projected/cedda12d-5f44-48eb-87ef-06404d7ee935-kube-api-access-nf2pw\") pod \"test-curl-7430-032ba0\" (UID: \"cedda12d-5f44-48eb-87ef-06404d7ee935\") " pod="opendatahub/test-curl-7430-032ba0"
Oct 30 07:13:50.625548 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:50.625461    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf2pw\" (UniqueName: \"kubernetes.io/projected/cedda12d-5f44-48eb-87ef-06404d7ee935-kube-api-access-nf2pw\") pod \"test-curl-7430-032ba0\" (UID: \"cedda12d-5f44-48eb-87ef-06404d7ee935\") " pod="opendatahub/test-curl-7430-032ba0"
Oct 30 07:13:50.650630 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:50.650605    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf2pw\" (UniqueName: \"kubernetes.io/projected/cedda12d-5f44-48eb-87ef-06404d7ee935-kube-api-access-nf2pw\") pod \"test-curl-7430-032ba0\" (UID: \"cedda12d-5f44-48eb-87ef-06404d7ee935\") " pod="opendatahub/test-curl-7430-032ba0"
Oct 30 07:13:50.747921 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:50.747890    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7430-032ba0"
Oct 30 07:13:50.788996 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:50.788962    6224 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 30 07:13:50.789597 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:50.789539    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc" podUID="79fa648e-4151-4998-b954-fc46d9841b97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 30 07:13:51.092524 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:51.092498    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7430-032ba0"]
Oct 30 07:13:51.094466 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:13:51.094440    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcedda12d_5f44_48eb_87ef_06404d7ee935.slice/crio-2231187047c1b6eff36cd8dc114aec18f70135c91beb22cdd673373abfb579cf WatchSource:0}: Error finding container 2231187047c1b6eff36cd8dc114aec18f70135c91beb22cdd673373abfb579cf: Status 404 returned error can't find the container with id 2231187047c1b6eff36cd8dc114aec18f70135c91beb22cdd673373abfb579cf
Oct 30 07:13:51.767939 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:51.767897    6224 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 30 07:13:51.768460 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:51.768421    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc" podUID="79fa648e-4151-4998-b954-fc46d9841b97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 30 07:13:51.907020 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:51.906943    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7430-032ba0" event={"ID":"cedda12d-5f44-48eb-87ef-06404d7ee935","Type":"ContainerStarted","Data":"2231187047c1b6eff36cd8dc114aec18f70135c91beb22cdd673373abfb579cf"}
Oct 30 07:13:53.301890 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:53.292165    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc"
Oct 30 07:13:53.301890 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:53.293627    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc"
Oct 30 07:13:53.326854 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:53.310205    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc"
Oct 30 07:13:54.217259 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:54.216365    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb6be4/premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t" podUID="7dd5133c-480f-4fa0-9dd1-27662004aae9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:13:56.139444 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:56.139355    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7430-032ba0" event={"ID":"cedda12d-5f44-48eb-87ef-06404d7ee935","Type":"ContainerStarted","Data":"54fbed134ed606b82a898dbaa9ebe81558691bc0f426001764e6c1f9dde04da1"}
Oct 30 07:13:56.185978 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:56.185931    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7430-032ba0" podStartSLOduration=1.392622859 podStartE2EDuration="6.185918311s" totalImagesPullingTime="4.793295452s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:13:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:13:56.184997081 +0000 UTC m=+994.794404054" watchObservedRunningTime="2026-10-30 07:13:56.185918311 +0000 UTC m=+994.795325280"
Oct 30 07:13:57.172236 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:57.172202    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7430-032ba0"]
Oct 30 07:13:57.177092 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:57.177065    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7430-032ba0"]
Oct 30 07:13:58.156364 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:13:58.156327    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7430-032ba0" podUID="cedda12d-5f44-48eb-87ef-06404d7ee935" containerName="test-curl-7430-032ba0" containerID="cri-o://54fbed134ed606b82a898dbaa9ebe81558691bc0f426001764e6c1f9dde04da1" gracePeriod=2
Oct 30 07:14:00.463330 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:00.463300    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7430-032ba0"
Oct 30 07:14:00.567785 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:00.567750    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cedda12d-5f44-48eb-87ef-06404d7ee935-kube-api-access-nf2pw\" (UniqueName: \"kubernetes.io/projected/cedda12d-5f44-48eb-87ef-06404d7ee935-kube-api-access-nf2pw\") pod \"cedda12d-5f44-48eb-87ef-06404d7ee935\" (UID: \"cedda12d-5f44-48eb-87ef-06404d7ee935\") "
Oct 30 07:14:00.569827 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:00.569792    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cedda12d-5f44-48eb-87ef-06404d7ee935-kube-api-access-nf2pw" pod "cedda12d-5f44-48eb-87ef-06404d7ee935" (UID: "cedda12d-5f44-48eb-87ef-06404d7ee935"). InnerVolumeSpecName "kube-api-access-nf2pw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:14:00.669289 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:00.669261    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nf2pw\" (UniqueName: \"kubernetes.io/projected/cedda12d-5f44-48eb-87ef-06404d7ee935-kube-api-access-nf2pw\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:14:01.182202 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:01.182152    6224 generic.go:356] "Generic (PLEG): container finished" podID="cedda12d-5f44-48eb-87ef-06404d7ee935" containerID="54fbed134ed606b82a898dbaa9ebe81558691bc0f426001764e6c1f9dde04da1" exitCode=137
Oct 30 07:14:01.182202 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:01.182198    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7430-032ba0"
Oct 30 07:14:01.182430 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:01.182238    6224 scope.go:122] "RemoveContainer" containerID="54fbed134ed606b82a898dbaa9ebe81558691bc0f426001764e6c1f9dde04da1"
Oct 30 07:14:01.192136 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:01.192118    6224 scope.go:122] "RemoveContainer" containerID="54fbed134ed606b82a898dbaa9ebe81558691bc0f426001764e6c1f9dde04da1"
Oct 30 07:14:01.192394 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:14:01.192376    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54fbed134ed606b82a898dbaa9ebe81558691bc0f426001764e6c1f9dde04da1\": container with ID starting with 54fbed134ed606b82a898dbaa9ebe81558691bc0f426001764e6c1f9dde04da1 not found: ID does not exist" containerID="54fbed134ed606b82a898dbaa9ebe81558691bc0f426001764e6c1f9dde04da1"
Oct 30 07:14:01.192448 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:01.192403    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54fbed134ed606b82a898dbaa9ebe81558691bc0f426001764e6c1f9dde04da1"} err="failed to get container status \"54fbed134ed606b82a898dbaa9ebe81558691bc0f426001764e6c1f9dde04da1\": rpc error: code = NotFound desc = could not find container \"54fbed134ed606b82a898dbaa9ebe81558691bc0f426001764e6c1f9dde04da1\": container with ID starting with 54fbed134ed606b82a898dbaa9ebe81558691bc0f426001764e6c1f9dde04da1 not found: ID does not exist"
Oct 30 07:14:01.206057 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:14:01.206030    6224 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7430-032ba0\" is forbidden: User \"system:node:ip-10-0-128-125.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-125.ec2.internal' and this object" podUID="cedda12d-5f44-48eb-87ef-06404d7ee935" pod="opendatahub/test-curl-7430-032ba0"
Oct 30 07:14:01.504514 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:01.504434    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cedda12d-5f44-48eb-87ef-06404d7ee935" path="/var/lib/kubelet/pods/cedda12d-5f44-48eb-87ef-06404d7ee935/volumes"
Oct 30 07:14:01.514897 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:14:01.514866    6224 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7430-032ba0\" is forbidden: User \"system:node:ip-10-0-128-125.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-125.ec2.internal' and this object" podUID="cedda12d-5f44-48eb-87ef-06404d7ee935" pod="opendatahub/test-curl-7430-032ba0"
Oct 30 07:14:01.554979 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:01.554947    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm"]
Oct 30 07:14:01.555792 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:01.555762    6224 cpu_manager.go:414] "containerMap: removing container" podUID="cedda12d-5f44-48eb-87ef-06404d7ee935" containerName="test-curl-7430-032ba0"
Oct 30 07:14:01.555792 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:01.555787    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cedda12d-5f44-48eb-87ef-06404d7ee935" containerName="test-curl-7430-032ba0"
Oct 30 07:14:01.556002 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:01.555989    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="cedda12d-5f44-48eb-87ef-06404d7ee935" containerName="test-curl-7430-032ba0"
Oct 30 07:14:01.561284 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:01.561260    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm"
Oct 30 07:14:01.580603 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:01.580577    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-a273fd-openshift-default-dockercfg-5bvzf\""
Oct 30 07:14:01.618860 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:01.610491    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm"]
Oct 30 07:14:01.690710 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:01.690669    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16b1f8ac-8fe8-4dd6-a843-2470652b69ab-istio-data\") pod \"e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm\" (UID: \"16b1f8ac-8fe8-4dd6-a843-2470652b69ab\") " pod="openshift-ingress/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm"
Oct 30 07:14:01.691532 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:01.691504    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16b1f8ac-8fe8-4dd6-a843-2470652b69ab-istio-podinfo\") pod \"e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm\" (UID: \"16b1f8ac-8fe8-4dd6-a843-2470652b69ab\") " pod="openshift-ingress/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm"
Oct 30 07:14:01.691849 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:01.691826    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnsbh\" (UniqueName: \"kubernetes.io/projected/16b1f8ac-8fe8-4dd6-a843-2470652b69ab-kube-api-access-rnsbh\") pod \"e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm\" (UID: \"16b1f8ac-8fe8-4dd6-a843-2470652b69ab\") " pod="openshift-ingress/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm"
Oct 30 07:14:01.692062 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:01.692045    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16b1f8ac-8fe8-4dd6-a843-2470652b69ab-workload-certs\") pod \"e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm\" (UID: \"16b1f8ac-8fe8-4dd6-a843-2470652b69ab\") " pod="openshift-ingress/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm"
Oct 30 07:14:01.692214 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:01.692198    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b1f8ac-8fe8-4dd6-a843-2470652b69ab-credential-socket\") pod \"e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm\" (UID: \"16b1f8ac-8fe8-4dd6-a843-2470652b69ab\") " pod="openshift-ingress/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm"
Oct 30 07:14:01.692348 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:01.692334    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b1f8ac-8fe8-4dd6-a843-2470652b69ab-workload-socket\") pod \"e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm\" (UID: \"16b1f8ac-8fe8-4dd6-a843-2470652b69ab\") " pod="openshift-ingress/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm"
Oct 30 07:14:01.692444 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:01.692430    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16b1f8ac-8fe8-4dd6-a843-2470652b69ab-istio-token\") pod \"e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm\" (UID: \"16b1f8ac-8fe8-4dd6-a843-2470652b69ab\") " pod="openshift-ingress/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm"
Oct 30 07:14:01.692554 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:01.692540    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16b1f8ac-8fe8-4dd6-a843-2470652b69ab-istiod-ca-cert\") pod \"e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm\" (UID: \"16b1f8ac-8fe8-4dd6-a843-2470652b69ab\") " pod="openshift-ingress/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm"
Oct 30 07:14:01.708150 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:01.708116    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16b1f8ac-8fe8-4dd6-a843-2470652b69ab-istio-envoy\") pod \"e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm\" (UID: \"16b1f8ac-8fe8-4dd6-a843-2470652b69ab\") " pod="openshift-ingress/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm"
Oct 30 07:14:01.813257 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:01.813213    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16b1f8ac-8fe8-4dd6-a843-2470652b69ab-istio-podinfo\") pod \"e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm\" (UID: \"16b1f8ac-8fe8-4dd6-a843-2470652b69ab\") " pod="openshift-ingress/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm"
Oct 30 07:14:01.813631 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:01.813598    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnsbh\" (UniqueName: \"kubernetes.io/projected/16b1f8ac-8fe8-4dd6-a843-2470652b69ab-kube-api-access-rnsbh\") pod \"e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm\" (UID: \"16b1f8ac-8fe8-4dd6-a843-2470652b69ab\") " pod="openshift-ingress/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm"
Oct 30 07:14:01.813983 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:01.813959    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16b1f8ac-8fe8-4dd6-a843-2470652b69ab-workload-certs\") pod \"e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm\" (UID: \"16b1f8ac-8fe8-4dd6-a843-2470652b69ab\") " pod="openshift-ingress/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm"
Oct 30 07:14:01.814155 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:01.814139    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b1f8ac-8fe8-4dd6-a843-2470652b69ab-credential-socket\") pod \"e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm\" (UID: \"16b1f8ac-8fe8-4dd6-a843-2470652b69ab\") " pod="openshift-ingress/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm"
Oct 30 07:14:01.814326 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:01.814297    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b1f8ac-8fe8-4dd6-a843-2470652b69ab-workload-socket\") pod \"e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm\" (UID: \"16b1f8ac-8fe8-4dd6-a843-2470652b69ab\") " pod="openshift-ingress/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm"
Oct 30 07:14:01.814451 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:01.814436    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16b1f8ac-8fe8-4dd6-a843-2470652b69ab-istio-token\") pod \"e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm\" (UID: \"16b1f8ac-8fe8-4dd6-a843-2470652b69ab\") " pod="openshift-ingress/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm"
Oct 30 07:14:01.814577 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:01.814560    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16b1f8ac-8fe8-4dd6-a843-2470652b69ab-istiod-ca-cert\") pod \"e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm\" (UID: \"16b1f8ac-8fe8-4dd6-a843-2470652b69ab\") " pod="openshift-ingress/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm"
Oct 30 07:14:01.814708 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:01.814695    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16b1f8ac-8fe8-4dd6-a843-2470652b69ab-istio-envoy\") pod \"e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm\" (UID: \"16b1f8ac-8fe8-4dd6-a843-2470652b69ab\") " pod="openshift-ingress/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm"
Oct 30 07:14:01.815083 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:01.815058    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16b1f8ac-8fe8-4dd6-a843-2470652b69ab-workload-certs\") pod \"e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm\" (UID: \"16b1f8ac-8fe8-4dd6-a843-2470652b69ab\") " pod="openshift-ingress/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm"
Oct 30 07:14:01.815296 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:01.815277    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b1f8ac-8fe8-4dd6-a843-2470652b69ab-credential-socket\") pod \"e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm\" (UID: \"16b1f8ac-8fe8-4dd6-a843-2470652b69ab\") " pod="openshift-ingress/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm"
Oct 30 07:14:01.815527 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:01.815504    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16b1f8ac-8fe8-4dd6-a843-2470652b69ab-istio-data\") pod \"e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm\" (UID: \"16b1f8ac-8fe8-4dd6-a843-2470652b69ab\") " pod="openshift-ingress/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm"
Oct 30 07:14:01.815894 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:01.814855    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b1f8ac-8fe8-4dd6-a843-2470652b69ab-workload-socket\") pod \"e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm\" (UID: \"16b1f8ac-8fe8-4dd6-a843-2470652b69ab\") " pod="openshift-ingress/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm"
Oct 30 07:14:01.817009 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:01.816985    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16b1f8ac-8fe8-4dd6-a843-2470652b69ab-istio-data\") pod \"e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm\" (UID: \"16b1f8ac-8fe8-4dd6-a843-2470652b69ab\") " pod="openshift-ingress/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm"
Oct 30 07:14:01.817418 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:01.817394    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16b1f8ac-8fe8-4dd6-a843-2470652b69ab-istiod-ca-cert\") pod \"e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm\" (UID: \"16b1f8ac-8fe8-4dd6-a843-2470652b69ab\") " pod="openshift-ingress/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm"
Oct 30 07:14:01.819672 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:01.819618    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16b1f8ac-8fe8-4dd6-a843-2470652b69ab-istio-envoy\") pod \"e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm\" (UID: \"16b1f8ac-8fe8-4dd6-a843-2470652b69ab\") " pod="openshift-ingress/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm"
Oct 30 07:14:01.821494 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:01.821469    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16b1f8ac-8fe8-4dd6-a843-2470652b69ab-istio-podinfo\") pod \"e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm\" (UID: \"16b1f8ac-8fe8-4dd6-a843-2470652b69ab\") " pod="openshift-ingress/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm"
Oct 30 07:14:01.843182 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:01.843124    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16b1f8ac-8fe8-4dd6-a843-2470652b69ab-istio-token\") pod \"e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm\" (UID: \"16b1f8ac-8fe8-4dd6-a843-2470652b69ab\") " pod="openshift-ingress/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm"
Oct 30 07:14:01.843854 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:01.843786    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnsbh\" (UniqueName: \"kubernetes.io/projected/16b1f8ac-8fe8-4dd6-a843-2470652b69ab-kube-api-access-rnsbh\") pod \"e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm\" (UID: \"16b1f8ac-8fe8-4dd6-a843-2470652b69ab\") " pod="openshift-ingress/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm"
Oct 30 07:14:01.875990 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:01.875956    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm"
Oct 30 07:14:01.879211 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:14:01.879176    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:14:02.234873 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:02.234841    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm"]
Oct 30 07:14:02.235342 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:14:02.235315    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16b1f8ac_8fe8_4dd6_a843_2470652b69ab.slice/crio-965fff3da67538dd43684796f63aa8ac1ccaa1c9a9947c33c22598fe98739a1e WatchSource:0}: Error finding container 965fff3da67538dd43684796f63aa8ac1ccaa1c9a9947c33c22598fe98739a1e: Status 404 returned error can't find the container with id 965fff3da67538dd43684796f63aa8ac1ccaa1c9a9947c33c22598fe98739a1e
Oct 30 07:14:02.237690 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:02.237657    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:14:02.237774 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:02.237733    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:14:02.237879 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:02.237781    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:14:03.074461 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:03.074372    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c0e3de-c4f94dbf9-p82jw"]
Oct 30 07:14:03.078678 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:03.078655    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c0e3de-c4f94dbf9-p82jw"
Oct 30 07:14:03.100052 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:03.100027    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-c0e3de\""
Oct 30 07:14:03.113699 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:03.113667    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-c0e3de-dockercfg-ptzvj\""
Oct 30 07:14:03.119359 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:03.119335    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c0e3de-c4f94dbf9-p82jw"]
Oct 30 07:14:03.137042 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:03.137006    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c34be83-969f-4fef-8954-c014d0d2e5cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c0e3de-c4f94dbf9-p82jw\" (UID: \"2c34be83-969f-4fef-8954-c014d0d2e5cc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c0e3de-c4f94dbf9-p82jw"
Oct 30 07:14:03.137209 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:03.137059    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svpqj\" (UniqueName: \"kubernetes.io/projected/2c34be83-969f-4fef-8954-c014d0d2e5cc-kube-api-access-svpqj\") pod \"payload-pre-processing-e2e-api-a-c0e3de-c4f94dbf9-p82jw\" (UID: \"2c34be83-969f-4fef-8954-c014d0d2e5cc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c0e3de-c4f94dbf9-p82jw"
Oct 30 07:14:03.182889 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:03.180374    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c0e3de-59fcb65d7d-68h76"]
Oct 30 07:14:03.187416 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:03.187385    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c0e3de-59fcb65d7d-68h76"
Oct 30 07:14:03.217853 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:03.207606    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c0e3de-59fcb65d7d-68h76"]
Oct 30 07:14:03.247838 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:03.238633    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svpqj\" (UniqueName: \"kubernetes.io/projected/2c34be83-969f-4fef-8954-c014d0d2e5cc-kube-api-access-svpqj\") pod \"payload-pre-processing-e2e-api-a-c0e3de-c4f94dbf9-p82jw\" (UID: \"2c34be83-969f-4fef-8954-c014d0d2e5cc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c0e3de-c4f94dbf9-p82jw"
Oct 30 07:14:03.247838 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:03.238758    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsq9z\" (UniqueName: \"kubernetes.io/projected/71cfd33e-bb00-4995-a800-628145442737-kube-api-access-hsq9z\") pod \"payload-processing-e2e-api-a-c0e3de-59fcb65d7d-68h76\" (UID: \"71cfd33e-bb00-4995-a800-628145442737\") " pod="openshift-ingress/payload-processing-e2e-api-a-c0e3de-59fcb65d7d-68h76"
Oct 30 07:14:03.247838 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:03.239023    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71cfd33e-bb00-4995-a800-628145442737-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c0e3de-59fcb65d7d-68h76\" (UID: \"71cfd33e-bb00-4995-a800-628145442737\") " pod="openshift-ingress/payload-processing-e2e-api-a-c0e3de-59fcb65d7d-68h76"
Oct 30 07:14:03.247838 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:03.239092    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c34be83-969f-4fef-8954-c014d0d2e5cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c0e3de-c4f94dbf9-p82jw\" (UID: \"2c34be83-969f-4fef-8954-c014d0d2e5cc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c0e3de-c4f94dbf9-p82jw"
Oct 30 07:14:03.247838 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:03.240085    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c34be83-969f-4fef-8954-c014d0d2e5cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c0e3de-c4f94dbf9-p82jw\" (UID: \"2c34be83-969f-4fef-8954-c014d0d2e5cc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c0e3de-c4f94dbf9-p82jw"
Oct 30 07:14:03.247838 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:03.247212    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm" event={"ID":"16b1f8ac-8fe8-4dd6-a843-2470652b69ab","Type":"ContainerStarted","Data":"1167e354b39eceac6c07062ada570d558bb29a3790e412303c9ad6b7fb0ded5c"}
Oct 30 07:14:03.247838 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:03.247272    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm" event={"ID":"16b1f8ac-8fe8-4dd6-a843-2470652b69ab","Type":"ContainerStarted","Data":"965fff3da67538dd43684796f63aa8ac1ccaa1c9a9947c33c22598fe98739a1e"}
Oct 30 07:14:03.275968 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:03.270070    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svpqj\" (UniqueName: \"kubernetes.io/projected/2c34be83-969f-4fef-8954-c014d0d2e5cc-kube-api-access-svpqj\") pod \"payload-pre-processing-e2e-api-a-c0e3de-c4f94dbf9-p82jw\" (UID: \"2c34be83-969f-4fef-8954-c014d0d2e5cc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c0e3de-c4f94dbf9-p82jw"
Oct 30 07:14:03.340143 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:03.340067    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsq9z\" (UniqueName: \"kubernetes.io/projected/71cfd33e-bb00-4995-a800-628145442737-kube-api-access-hsq9z\") pod \"payload-processing-e2e-api-a-c0e3de-59fcb65d7d-68h76\" (UID: \"71cfd33e-bb00-4995-a800-628145442737\") " pod="openshift-ingress/payload-processing-e2e-api-a-c0e3de-59fcb65d7d-68h76"
Oct 30 07:14:03.340304 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:03.340229    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71cfd33e-bb00-4995-a800-628145442737-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c0e3de-59fcb65d7d-68h76\" (UID: \"71cfd33e-bb00-4995-a800-628145442737\") " pod="openshift-ingress/payload-processing-e2e-api-a-c0e3de-59fcb65d7d-68h76"
Oct 30 07:14:03.340934 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:03.340910    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71cfd33e-bb00-4995-a800-628145442737-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c0e3de-59fcb65d7d-68h76\" (UID: \"71cfd33e-bb00-4995-a800-628145442737\") " pod="openshift-ingress/payload-processing-e2e-api-a-c0e3de-59fcb65d7d-68h76"
Oct 30 07:14:03.365323 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:03.365301    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsq9z\" (UniqueName: \"kubernetes.io/projected/71cfd33e-bb00-4995-a800-628145442737-kube-api-access-hsq9z\") pod \"payload-processing-e2e-api-a-c0e3de-59fcb65d7d-68h76\" (UID: \"71cfd33e-bb00-4995-a800-628145442737\") " pod="openshift-ingress/payload-processing-e2e-api-a-c0e3de-59fcb65d7d-68h76"
Oct 30 07:14:03.393332 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:03.393306    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c0e3de-c4f94dbf9-p82jw"
Oct 30 07:14:03.576845 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:03.576781    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c0e3de-59fcb65d7d-68h76"
Oct 30 07:14:03.744546 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:03.744475    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm" podStartSLOduration=2.744452376 podStartE2EDuration="2.744452376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:14:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:14:03.326167106 +0000 UTC m=+1001.935574080" watchObservedRunningTime="2026-10-30 07:14:03.744452376 +0000 UTC m=+1002.353859347"
Oct 30 07:14:03.747320 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:03.747288    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c0e3de-c4f94dbf9-p82jw"]
Oct 30 07:14:03.877000 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:03.876923    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm"
Oct 30 07:14:03.883394 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:03.883371    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm"
Oct 30 07:14:03.895222 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:14:03.895196    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71cfd33e_bb00_4995_a800_628145442737.slice/crio-7d180e36921f8bd1aa786fc5fe408c0441e3785b1071b2681d504325ab82a109 WatchSource:0}: Error finding container 7d180e36921f8bd1aa786fc5fe408c0441e3785b1071b2681d504325ab82a109: Status 404 returned error can't find the container with id 7d180e36921f8bd1aa786fc5fe408c0441e3785b1071b2681d504325ab82a109
Oct 30 07:14:03.896228 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:03.896205    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c0e3de-59fcb65d7d-68h76"]
Oct 30 07:14:04.255979 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:04.255871    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c0e3de-c4f94dbf9-p82jw" event={"ID":"2c34be83-969f-4fef-8954-c014d0d2e5cc","Type":"ContainerStarted","Data":"b87aac45e21c8e18b2e83d3d89931562053533e30ae1cceca9aa50cfa14975f3"}
Oct 30 07:14:04.256671 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:04.256644    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c0e3de-c4f94dbf9-p82jw" event={"ID":"2c34be83-969f-4fef-8954-c014d0d2e5cc","Type":"ContainerStarted","Data":"50cb6f1dde98fc0c40631dd833ed9322ac77d939bce16ae4a8d65fc23b68e9cd"}
Oct 30 07:14:04.256861 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:04.256844    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c0e3de-c4f94dbf9-p82jw"
Oct 30 07:14:04.258611 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:04.258471    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c0e3de-59fcb65d7d-68h76" event={"ID":"71cfd33e-bb00-4995-a800-628145442737","Type":"ContainerStarted","Data":"7443c480d6e893fbae14777cbbc817add86ed8a73826b5726b88040956ec0064"}
Oct 30 07:14:04.258611 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:04.258588    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c0e3de-59fcb65d7d-68h76" event={"ID":"71cfd33e-bb00-4995-a800-628145442737","Type":"ContainerStarted","Data":"7d180e36921f8bd1aa786fc5fe408c0441e3785b1071b2681d504325ab82a109"}
Oct 30 07:14:04.259225 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:04.259023    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-c0e3de-59fcb65d7d-68h76"
Oct 30 07:14:04.259225 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:04.259090    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm"
Oct 30 07:14:04.260339 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:04.260245    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm"
Oct 30 07:14:04.302848 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:04.302789    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c0e3de-c4f94dbf9-p82jw" podStartSLOduration=1.302777104 podStartE2EDuration="1.302777104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:14:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:14:04.299857747 +0000 UTC m=+1002.909264716" watchObservedRunningTime="2026-10-30 07:14:04.302777104 +0000 UTC m=+1002.912184074"
Oct 30 07:14:04.354217 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:04.354170    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-c0e3de-59fcb65d7d-68h76" podStartSLOduration=1.3541565709999999 podStartE2EDuration="1.354156571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:14:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:14:04.351221138 +0000 UTC m=+1002.960628110" watchObservedRunningTime="2026-10-30 07:14:04.354156571 +0000 UTC m=+1002.963563558"
Oct 30 07:14:15.280831 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:15.275602    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-c0e3de-59fcb65d7d-68h76"
Oct 30 07:14:16.277070 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:16.277033    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c0e3de-c4f94dbf9-p82jw"
Oct 30 07:14:31.857347 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:14:31.857312    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:14:40.085445 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:40.085410    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:14:43.655140 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:43.655106    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a163b2-5d955f44dd-z4xbd"]
Oct 30 07:14:43.655584 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:43.655369    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a163b2-5d955f44dd-z4xbd" podUID="5175e233-cb13-4706-a43f-f51f03927a87" containerName="maas-api" containerID="cri-o://d17cc38083e9d568bab7a393138767a69f2280a61d89e6153c3cfe223dc8f545" gracePeriod=30
Oct 30 07:14:44.247289 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:44.247259    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a163b2-5d955f44dd-z4xbd"
Oct 30 07:14:44.358916 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:44.351421    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5175e233-cb13-4706-a43f-f51f03927a87-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5175e233-cb13-4706-a43f-f51f03927a87-maas-api-tls\") pod \"5175e233-cb13-4706-a43f-f51f03927a87\" (UID: \"5175e233-cb13-4706-a43f-f51f03927a87\") "
Oct 30 07:14:44.358916 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:44.351604    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5175e233-cb13-4706-a43f-f51f03927a87-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5175e233-cb13-4706-a43f-f51f03927a87-metrics-certs\") pod \"5175e233-cb13-4706-a43f-f51f03927a87\" (UID: \"5175e233-cb13-4706-a43f-f51f03927a87\") "
Oct 30 07:14:44.358916 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:44.351886    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5175e233-cb13-4706-a43f-f51f03927a87-kube-api-access-7km6v\" (UniqueName: \"kubernetes.io/projected/5175e233-cb13-4706-a43f-f51f03927a87-kube-api-access-7km6v\") pod \"5175e233-cb13-4706-a43f-f51f03927a87\" (UID: \"5175e233-cb13-4706-a43f-f51f03927a87\") "
Oct 30 07:14:44.358916 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:44.351921    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5175e233-cb13-4706-a43f-f51f03927a87-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5175e233-cb13-4706-a43f-f51f03927a87-openshift-service-ca\") pod \"5175e233-cb13-4706-a43f-f51f03927a87\" (UID: \"5175e233-cb13-4706-a43f-f51f03927a87\") "
Oct 30 07:14:44.358916 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:44.352942    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5175e233-cb13-4706-a43f-f51f03927a87-openshift-service-ca" pod "5175e233-cb13-4706-a43f-f51f03927a87" (UID: "5175e233-cb13-4706-a43f-f51f03927a87"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:14:44.358916 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:44.355648    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5175e233-cb13-4706-a43f-f51f03927a87-maas-api-tls" pod "5175e233-cb13-4706-a43f-f51f03927a87" (UID: "5175e233-cb13-4706-a43f-f51f03927a87"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:14:44.359744 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:44.359720    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5175e233-cb13-4706-a43f-f51f03927a87-kube-api-access-7km6v" pod "5175e233-cb13-4706-a43f-f51f03927a87" (UID: "5175e233-cb13-4706-a43f-f51f03927a87"). InnerVolumeSpecName "kube-api-access-7km6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:14:44.361382 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:44.361356    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5175e233-cb13-4706-a43f-f51f03927a87-metrics-certs" pod "5175e233-cb13-4706-a43f-f51f03927a87" (UID: "5175e233-cb13-4706-a43f-f51f03927a87"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:14:44.454254 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:44.454166    6224 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5175e233-cb13-4706-a43f-f51f03927a87-metrics-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:14:44.454254 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:44.454206    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7km6v\" (UniqueName: \"kubernetes.io/projected/5175e233-cb13-4706-a43f-f51f03927a87-kube-api-access-7km6v\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:14:44.454254 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:44.454221    6224 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5175e233-cb13-4706-a43f-f51f03927a87-openshift-service-ca\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:14:44.454254 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:44.454235    6224 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5175e233-cb13-4706-a43f-f51f03927a87-maas-api-tls\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:14:44.610972 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:44.610932    6224 generic.go:356] "Generic (PLEG): container finished" podID="5175e233-cb13-4706-a43f-f51f03927a87" containerID="d17cc38083e9d568bab7a393138767a69f2280a61d89e6153c3cfe223dc8f545" exitCode=0
Oct 30 07:14:44.611180 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:44.610993    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a163b2-5d955f44dd-z4xbd"
Oct 30 07:14:44.611180 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:44.611025    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a163b2-5d955f44dd-z4xbd" event={"ID":"5175e233-cb13-4706-a43f-f51f03927a87","Type":"ContainerDied","Data":"d17cc38083e9d568bab7a393138767a69f2280a61d89e6153c3cfe223dc8f545"}
Oct 30 07:14:44.611180 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:44.611093    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a163b2-5d955f44dd-z4xbd" event={"ID":"5175e233-cb13-4706-a43f-f51f03927a87","Type":"ContainerDied","Data":"cbcf448a5e97a880979e7df63e06f982d98829cc635a06e82eba6d977fd7cfb8"}
Oct 30 07:14:44.611180 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:44.611125    6224 scope.go:122] "RemoveContainer" containerID="d17cc38083e9d568bab7a393138767a69f2280a61d89e6153c3cfe223dc8f545"
Oct 30 07:14:44.623802 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:44.623779    6224 scope.go:122] "RemoveContainer" containerID="d17cc38083e9d568bab7a393138767a69f2280a61d89e6153c3cfe223dc8f545"
Oct 30 07:14:44.624105 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:14:44.624081    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d17cc38083e9d568bab7a393138767a69f2280a61d89e6153c3cfe223dc8f545\": container with ID starting with d17cc38083e9d568bab7a393138767a69f2280a61d89e6153c3cfe223dc8f545 not found: ID does not exist" containerID="d17cc38083e9d568bab7a393138767a69f2280a61d89e6153c3cfe223dc8f545"
Oct 30 07:14:44.624188 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:44.624114    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d17cc38083e9d568bab7a393138767a69f2280a61d89e6153c3cfe223dc8f545"} err="failed to get container status \"d17cc38083e9d568bab7a393138767a69f2280a61d89e6153c3cfe223dc8f545\": rpc error: code = NotFound desc = could not find container \"d17cc38083e9d568bab7a393138767a69f2280a61d89e6153c3cfe223dc8f545\": container with ID starting with d17cc38083e9d568bab7a393138767a69f2280a61d89e6153c3cfe223dc8f545 not found: ID does not exist"
Oct 30 07:14:44.704494 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:44.704396    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a163b2-5d955f44dd-z4xbd"]
Oct 30 07:14:44.707621 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:44.707595    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a163b2-5d955f44dd-z4xbd"]
Oct 30 07:14:45.503698 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:45.503663    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5175e233-cb13-4706-a43f-f51f03927a87" path="/var/lib/kubelet/pods/5175e233-cb13-4706-a43f-f51f03927a87/volumes"
Oct 30 07:14:50.300951 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:50.300904    6224 patch_prober.go:29] interesting pod/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 07:14:50.301449 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:50.300988    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm" podUID="16b1f8ac-8fe8-4dd6-a843-2470652b69ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:14:50.799005 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:50.798970    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a6a14e07-bdc5d9c5f-j5wkj"]
Oct 30 07:14:50.799527 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:50.799514    6224 cpu_manager.go:414] "containerMap: removing container" podUID="5175e233-cb13-4706-a43f-f51f03927a87" containerName="maas-api"
Oct 30 07:14:50.799571 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:50.799528    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5175e233-cb13-4706-a43f-f51f03927a87" containerName="maas-api"
Oct 30 07:14:50.799648 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:50.799637    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="5175e233-cb13-4706-a43f-f51f03927a87" containerName="maas-api"
Oct 30 07:14:50.804348 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:50.804328    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a6a14e07-bdc5d9c5f-j5wkj"
Oct 30 07:14:50.823841 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:50.822832    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-a6a14e07\""
Oct 30 07:14:50.839001 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:50.838977    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a6a14e07-bdc5d9c5f-j5wkj"]
Oct 30 07:14:50.874532 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:50.874496    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/485a6676-1591-4e3c-b856-5bf04f8aef3b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a6a14e07-bdc5d9c5f-j5wkj\" (UID: \"485a6676-1591-4e3c-b856-5bf04f8aef3b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a6a14e07-bdc5d9c5f-j5wkj"
Oct 30 07:14:50.874677 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:50.874558    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drm9s\" (UniqueName: \"kubernetes.io/projected/485a6676-1591-4e3c-b856-5bf04f8aef3b-kube-api-access-drm9s\") pod \"payload-pre-processing-e2e-mt-a6a14e07-bdc5d9c5f-j5wkj\" (UID: \"485a6676-1591-4e3c-b856-5bf04f8aef3b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a6a14e07-bdc5d9c5f-j5wkj"
Oct 30 07:14:50.876922 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:50.876889    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-a6a14e07-dockercfg-b4ff6\""
Oct 30 07:14:50.989282 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:50.989234    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/485a6676-1591-4e3c-b856-5bf04f8aef3b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a6a14e07-bdc5d9c5f-j5wkj\" (UID: \"485a6676-1591-4e3c-b856-5bf04f8aef3b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a6a14e07-bdc5d9c5f-j5wkj"
Oct 30 07:14:50.992521 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:50.992479    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drm9s\" (UniqueName: \"kubernetes.io/projected/485a6676-1591-4e3c-b856-5bf04f8aef3b-kube-api-access-drm9s\") pod \"payload-pre-processing-e2e-mt-a6a14e07-bdc5d9c5f-j5wkj\" (UID: \"485a6676-1591-4e3c-b856-5bf04f8aef3b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a6a14e07-bdc5d9c5f-j5wkj"
Oct 30 07:14:50.993136 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:50.992386    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/485a6676-1591-4e3c-b856-5bf04f8aef3b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a6a14e07-bdc5d9c5f-j5wkj\" (UID: \"485a6676-1591-4e3c-b856-5bf04f8aef3b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a6a14e07-bdc5d9c5f-j5wkj"
Oct 30 07:14:51.053219 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:51.053137    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drm9s\" (UniqueName: \"kubernetes.io/projected/485a6676-1591-4e3c-b856-5bf04f8aef3b-kube-api-access-drm9s\") pod \"payload-pre-processing-e2e-mt-a6a14e07-bdc5d9c5f-j5wkj\" (UID: \"485a6676-1591-4e3c-b856-5bf04f8aef3b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a6a14e07-bdc5d9c5f-j5wkj"
Oct 30 07:14:51.119397 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:51.119358    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a6a14e07-bdc5d9c5f-j5wkj"
Oct 30 07:14:51.437978 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:14:51.437934    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod485a6676_1591_4e3c_b856_5bf04f8aef3b.slice/crio-2165bf000ee353a701385ded9d507ad0d3deb050d549965c16f04d11810281a6 WatchSource:0}: Error finding container 2165bf000ee353a701385ded9d507ad0d3deb050d549965c16f04d11810281a6: Status 404 returned error can't find the container with id 2165bf000ee353a701385ded9d507ad0d3deb050d549965c16f04d11810281a6
Oct 30 07:14:51.444842 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:51.441306    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a6a14e07-bdc5d9c5f-j5wkj"]
Oct 30 07:14:51.670762 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:51.670676    6224 generic.go:356] "Generic (PLEG): container finished" podID="16b1f8ac-8fe8-4dd6-a843-2470652b69ab" containerID="1167e354b39eceac6c07062ada570d558bb29a3790e412303c9ad6b7fb0ded5c" exitCode=0
Oct 30 07:14:51.670762 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:51.670748    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm" event={"ID":"16b1f8ac-8fe8-4dd6-a843-2470652b69ab","Type":"ContainerDied","Data":"1167e354b39eceac6c07062ada570d558bb29a3790e412303c9ad6b7fb0ded5c"}
Oct 30 07:14:51.671217 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:51.671200    6224 scope.go:122] "RemoveContainer" containerID="1167e354b39eceac6c07062ada570d558bb29a3790e412303c9ad6b7fb0ded5c"
Oct 30 07:14:51.672593 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:51.672566    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a6a14e07-bdc5d9c5f-j5wkj" event={"ID":"485a6676-1591-4e3c-b856-5bf04f8aef3b","Type":"ContainerStarted","Data":"80d58d6265926564828b11afa18523729a2defd492a7b0de1927595aa3a4163d"}
Oct 30 07:14:51.672697 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:51.672616    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a6a14e07-bdc5d9c5f-j5wkj" event={"ID":"485a6676-1591-4e3c-b856-5bf04f8aef3b","Type":"ContainerStarted","Data":"2165bf000ee353a701385ded9d507ad0d3deb050d549965c16f04d11810281a6"}
Oct 30 07:14:51.672697 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:51.672632    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:14:51.672697 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:51.672647    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-a6a14e07-bdc5d9c5f-j5wkj"
Oct 30 07:14:51.672697 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:51.672692    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:14:51.672865 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:51.672726    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:14:51.787837 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:51.772367    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-a6a14e07-bdc5d9c5f-j5wkj" podStartSLOduration=1.772346278 podStartE2EDuration="1.772346278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:14:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:14:51.768563872 +0000 UTC m=+1050.377970868" watchObservedRunningTime="2026-10-30 07:14:51.772346278 +0000 UTC m=+1050.381753249"
Oct 30 07:14:52.615493 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:52.612947    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv"]
Oct 30 07:14:52.615493 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:52.613211    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv" podUID="14cba440-5936-4d19-bc3d-6d3aabc70f38" containerName="istio-proxy" containerID="cri-o://48c3751f6a33598a039451a7f7e5f74070244415d3cb7db5db04112b200c05f0" gracePeriod=30
Oct 30 07:14:52.697232 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:52.697196    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm" event={"ID":"16b1f8ac-8fe8-4dd6-a843-2470652b69ab","Type":"ContainerStarted","Data":"a4635e456f5d90f57ef4b1b2bb0357d5d4ed8bed6f88bbb580fb26500e5598d0"}
Oct 30 07:14:52.887953 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:52.876366    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm"
Oct 30 07:14:52.887953 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:52.876475    6224 patch_prober.go:29] interesting pod/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 30 07:14:52.887953 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:52.876519    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm" podUID="16b1f8ac-8fe8-4dd6-a843-2470652b69ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 30 07:14:53.884828 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:53.884144    6224 patch_prober.go:29] interesting pod/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 30 07:14:53.884828 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:53.884204    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm" podUID="16b1f8ac-8fe8-4dd6-a843-2470652b69ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 30 07:14:54.590131 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:54.590091    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4e42c8-64d8f55c96-6wxtx"]
Oct 30 07:14:54.594290 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:54.594268    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4e42c8-64d8f55c96-6wxtx"
Oct 30 07:14:54.598189 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:54.598166    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4e42c8-64d8f55c96-6wxtx"]
Oct 30 07:14:54.628071 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:54.628042    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-4e42c8\""
Oct 30 07:14:54.628071 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:54.628047    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-4e42c8-dockercfg-2mg42\""
Oct 30 07:14:54.631624 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:54.631599    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4e42c8-658945d447-wkthc"]
Oct 30 07:14:54.635410 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:54.635393    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4e42c8-658945d447-wkthc"
Oct 30 07:14:54.649611 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:54.649588    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4e42c8-658945d447-wkthc"]
Oct 30 07:14:54.662713 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:54.662683    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8beda270-8cb9-4211-889b-1e8a42cf3891-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4e42c8-64d8f55c96-6wxtx\" (UID: \"8beda270-8cb9-4211-889b-1e8a42cf3891\") " pod="openshift-ingress/payload-processing-e2e-api-b-4e42c8-64d8f55c96-6wxtx"
Oct 30 07:14:54.662835 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:54.662792    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqnnv\" (UniqueName: \"kubernetes.io/projected/8beda270-8cb9-4211-889b-1e8a42cf3891-kube-api-access-wqnnv\") pod \"payload-processing-e2e-api-b-4e42c8-64d8f55c96-6wxtx\" (UID: \"8beda270-8cb9-4211-889b-1e8a42cf3891\") " pod="openshift-ingress/payload-processing-e2e-api-b-4e42c8-64d8f55c96-6wxtx"
Oct 30 07:14:54.773832 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:54.765709    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk5zg\" (UniqueName: \"kubernetes.io/projected/dcc1ddda-8a2c-40cb-b3fa-82b7e931e7b7-kube-api-access-tk5zg\") pod \"payload-pre-processing-e2e-api-b-4e42c8-658945d447-wkthc\" (UID: \"dcc1ddda-8a2c-40cb-b3fa-82b7e931e7b7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4e42c8-658945d447-wkthc"
Oct 30 07:14:54.773832 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:54.765854    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8beda270-8cb9-4211-889b-1e8a42cf3891-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4e42c8-64d8f55c96-6wxtx\" (UID: \"8beda270-8cb9-4211-889b-1e8a42cf3891\") " pod="openshift-ingress/payload-processing-e2e-api-b-4e42c8-64d8f55c96-6wxtx"
Oct 30 07:14:54.773832 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:54.765929    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqnnv\" (UniqueName: \"kubernetes.io/projected/8beda270-8cb9-4211-889b-1e8a42cf3891-kube-api-access-wqnnv\") pod \"payload-processing-e2e-api-b-4e42c8-64d8f55c96-6wxtx\" (UID: \"8beda270-8cb9-4211-889b-1e8a42cf3891\") " pod="openshift-ingress/payload-processing-e2e-api-b-4e42c8-64d8f55c96-6wxtx"
Oct 30 07:14:54.773832 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:54.766018    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcc1ddda-8a2c-40cb-b3fa-82b7e931e7b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4e42c8-658945d447-wkthc\" (UID: \"dcc1ddda-8a2c-40cb-b3fa-82b7e931e7b7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4e42c8-658945d447-wkthc"
Oct 30 07:14:54.800297 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:54.800262    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8beda270-8cb9-4211-889b-1e8a42cf3891-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4e42c8-64d8f55c96-6wxtx\" (UID: \"8beda270-8cb9-4211-889b-1e8a42cf3891\") " pod="openshift-ingress/payload-processing-e2e-api-b-4e42c8-64d8f55c96-6wxtx"
Oct 30 07:14:54.842979 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:54.839544    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqnnv\" (UniqueName: \"kubernetes.io/projected/8beda270-8cb9-4211-889b-1e8a42cf3891-kube-api-access-wqnnv\") pod \"payload-processing-e2e-api-b-4e42c8-64d8f55c96-6wxtx\" (UID: \"8beda270-8cb9-4211-889b-1e8a42cf3891\") " pod="openshift-ingress/payload-processing-e2e-api-b-4e42c8-64d8f55c96-6wxtx"
Oct 30 07:14:54.872836 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:54.869864    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcc1ddda-8a2c-40cb-b3fa-82b7e931e7b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4e42c8-658945d447-wkthc\" (UID: \"dcc1ddda-8a2c-40cb-b3fa-82b7e931e7b7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4e42c8-658945d447-wkthc"
Oct 30 07:14:54.872836 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:54.870071    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk5zg\" (UniqueName: \"kubernetes.io/projected/dcc1ddda-8a2c-40cb-b3fa-82b7e931e7b7-kube-api-access-tk5zg\") pod \"payload-pre-processing-e2e-api-b-4e42c8-658945d447-wkthc\" (UID: \"dcc1ddda-8a2c-40cb-b3fa-82b7e931e7b7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4e42c8-658945d447-wkthc"
Oct 30 07:14:54.872836 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:54.871328    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcc1ddda-8a2c-40cb-b3fa-82b7e931e7b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4e42c8-658945d447-wkthc\" (UID: \"dcc1ddda-8a2c-40cb-b3fa-82b7e931e7b7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4e42c8-658945d447-wkthc"
Oct 30 07:14:54.883842 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:54.879352    6224 patch_prober.go:29] interesting pod/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 30 07:14:54.883842 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:54.879405    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm" podUID="16b1f8ac-8fe8-4dd6-a843-2470652b69ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 30 07:14:54.922171 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:54.903041    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk5zg\" (UniqueName: \"kubernetes.io/projected/dcc1ddda-8a2c-40cb-b3fa-82b7e931e7b7-kube-api-access-tk5zg\") pod \"payload-pre-processing-e2e-api-b-4e42c8-658945d447-wkthc\" (UID: \"dcc1ddda-8a2c-40cb-b3fa-82b7e931e7b7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4e42c8-658945d447-wkthc"
Oct 30 07:14:54.922171 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:54.914039    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4e42c8-64d8f55c96-6wxtx"
Oct 30 07:14:54.954265 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:54.953854    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4e42c8-658945d447-wkthc"
Oct 30 07:14:55.304016 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:14:55.303988    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8beda270_8cb9_4211_889b_1e8a42cf3891.slice/crio-9684516c85b76b144b1c0912ba5007f6532f0f75a2c5a34c08ee1c1689c47912 WatchSource:0}: Error finding container 9684516c85b76b144b1c0912ba5007f6532f0f75a2c5a34c08ee1c1689c47912: Status 404 returned error can't find the container with id 9684516c85b76b144b1c0912ba5007f6532f0f75a2c5a34c08ee1c1689c47912
Oct 30 07:14:55.305345 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:55.305317    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4e42c8-64d8f55c96-6wxtx"]
Oct 30 07:14:55.340743 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:55.340718    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4e42c8-658945d447-wkthc"]
Oct 30 07:14:55.341129 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:14:55.341108    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcc1ddda_8a2c_40cb_b3fa_82b7e931e7b7.slice/crio-646f90edf58d2300b6149046e346451e1e30e6231d30636be625d232be9c4118 WatchSource:0}: Error finding container 646f90edf58d2300b6149046e346451e1e30e6231d30636be625d232be9c4118: Status 404 returned error can't find the container with id 646f90edf58d2300b6149046e346451e1e30e6231d30636be625d232be9c4118
Oct 30 07:14:55.725549 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:55.725448    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4e42c8-64d8f55c96-6wxtx" event={"ID":"8beda270-8cb9-4211-889b-1e8a42cf3891","Type":"ContainerStarted","Data":"e470451d24cb0d55996889804f23dcc2ace9bb8b7b5bd2c58760657d9057d00b"}
Oct 30 07:14:55.725549 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:55.725503    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4e42c8-64d8f55c96-6wxtx" event={"ID":"8beda270-8cb9-4211-889b-1e8a42cf3891","Type":"ContainerStarted","Data":"9684516c85b76b144b1c0912ba5007f6532f0f75a2c5a34c08ee1c1689c47912"}
Oct 30 07:14:55.725777 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:55.725634    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-4e42c8-64d8f55c96-6wxtx"
Oct 30 07:14:55.727159 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:55.727134    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4e42c8-658945d447-wkthc" event={"ID":"dcc1ddda-8a2c-40cb-b3fa-82b7e931e7b7","Type":"ContainerStarted","Data":"6f7d1dc929970778cc5bb8f83314729f79ca623523a914db44a8e688bd19efd4"}
Oct 30 07:14:55.727274 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:55.727171    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4e42c8-658945d447-wkthc" event={"ID":"dcc1ddda-8a2c-40cb-b3fa-82b7e931e7b7","Type":"ContainerStarted","Data":"646f90edf58d2300b6149046e346451e1e30e6231d30636be625d232be9c4118"}
Oct 30 07:14:55.727274 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:55.727225    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4e42c8-658945d447-wkthc"
Oct 30 07:14:55.780840 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:55.780476    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-4e42c8-64d8f55c96-6wxtx" podStartSLOduration=1.780456079 podStartE2EDuration="1.780456079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:14:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:14:55.779198072 +0000 UTC m=+1054.388605044" watchObservedRunningTime="2026-10-30 07:14:55.780456079 +0000 UTC m=+1054.389863050"
Oct 30 07:14:55.824919 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:55.824869    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4e42c8-658945d447-wkthc" podStartSLOduration=1.824790242 podStartE2EDuration="1.824790242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:14:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:14:55.822322877 +0000 UTC m=+1054.431729849" watchObservedRunningTime="2026-10-30 07:14:55.824790242 +0000 UTC m=+1054.434197239"
Oct 30 07:14:55.883832 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:55.879213    6224 patch_prober.go:29] interesting pod/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 30 07:14:55.883832 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:55.879276    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm" podUID="16b1f8ac-8fe8-4dd6-a843-2470652b69ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 30 07:14:56.877080 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:56.877027    6224 patch_prober.go:29] interesting pod/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 30 07:14:56.877870 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:56.877827    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm" podUID="16b1f8ac-8fe8-4dd6-a843-2470652b69ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 30 07:14:57.748529 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:57.748471    6224 generic.go:356] "Generic (PLEG): container finished" podID="14cba440-5936-4d19-bc3d-6d3aabc70f38" containerID="48c3751f6a33598a039451a7f7e5f74070244415d3cb7db5db04112b200c05f0" exitCode=0
Oct 30 07:14:57.748678 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:57.748538    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv" event={"ID":"14cba440-5936-4d19-bc3d-6d3aabc70f38","Type":"ContainerDied","Data":"48c3751f6a33598a039451a7f7e5f74070244415d3cb7db5db04112b200c05f0"}
Oct 30 07:14:57.748678 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:57.748591    6224 scope.go:122] "RemoveContainer" containerID="423d154d81c95109780fe710d18c7badf83807d6bd2dcba26247f232cf62c783"
Oct 30 07:14:57.877168 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:57.877133    6224 patch_prober.go:29] interesting pod/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 30 07:14:57.877511 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:57.877222    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm" podUID="16b1f8ac-8fe8-4dd6-a843-2470652b69ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 30 07:14:57.953144 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:57.953121    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv"
Oct 30 07:14:58.047041 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:58.046993    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14cba440-5936-4d19-bc3d-6d3aabc70f38-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14cba440-5936-4d19-bc3d-6d3aabc70f38-istio-envoy\") pod \"14cba440-5936-4d19-bc3d-6d3aabc70f38\" (UID: \"14cba440-5936-4d19-bc3d-6d3aabc70f38\") "
Oct 30 07:14:58.047232 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:58.047116    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14cba440-5936-4d19-bc3d-6d3aabc70f38-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14cba440-5936-4d19-bc3d-6d3aabc70f38-istiod-ca-cert\") pod \"14cba440-5936-4d19-bc3d-6d3aabc70f38\" (UID: \"14cba440-5936-4d19-bc3d-6d3aabc70f38\") "
Oct 30 07:14:58.047232 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:58.047205    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14cba440-5936-4d19-bc3d-6d3aabc70f38-istio-token\" (UniqueName: \"kubernetes.io/projected/14cba440-5936-4d19-bc3d-6d3aabc70f38-istio-token\") pod \"14cba440-5936-4d19-bc3d-6d3aabc70f38\" (UID: \"14cba440-5936-4d19-bc3d-6d3aabc70f38\") "
Oct 30 07:14:58.047361 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:58.047303    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14cba440-5936-4d19-bc3d-6d3aabc70f38-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14cba440-5936-4d19-bc3d-6d3aabc70f38-istio-data\") pod \"14cba440-5936-4d19-bc3d-6d3aabc70f38\" (UID: \"14cba440-5936-4d19-bc3d-6d3aabc70f38\") "
Oct 30 07:14:58.047427 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:58.047397    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/14cba440-5936-4d19-bc3d-6d3aabc70f38-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14cba440-5936-4d19-bc3d-6d3aabc70f38-istio-podinfo\") pod \"14cba440-5936-4d19-bc3d-6d3aabc70f38\" (UID: \"14cba440-5936-4d19-bc3d-6d3aabc70f38\") "
Oct 30 07:14:58.047502 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:58.047480    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14cba440-5936-4d19-bc3d-6d3aabc70f38-kube-api-access-8hfr7\" (UniqueName: \"kubernetes.io/projected/14cba440-5936-4d19-bc3d-6d3aabc70f38-kube-api-access-8hfr7\") pod \"14cba440-5936-4d19-bc3d-6d3aabc70f38\" (UID: \"14cba440-5936-4d19-bc3d-6d3aabc70f38\") "
Oct 30 07:14:58.047564 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:58.047533    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14cba440-5936-4d19-bc3d-6d3aabc70f38-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14cba440-5936-4d19-bc3d-6d3aabc70f38-credential-socket\") pod \"14cba440-5936-4d19-bc3d-6d3aabc70f38\" (UID: \"14cba440-5936-4d19-bc3d-6d3aabc70f38\") "
Oct 30 07:14:58.047621 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:58.047576    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14cba440-5936-4d19-bc3d-6d3aabc70f38-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14cba440-5936-4d19-bc3d-6d3aabc70f38-workload-socket\") pod \"14cba440-5936-4d19-bc3d-6d3aabc70f38\" (UID: \"14cba440-5936-4d19-bc3d-6d3aabc70f38\") "
Oct 30 07:14:58.047621 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:58.047608    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14cba440-5936-4d19-bc3d-6d3aabc70f38-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14cba440-5936-4d19-bc3d-6d3aabc70f38-workload-certs\") pod \"14cba440-5936-4d19-bc3d-6d3aabc70f38\" (UID: \"14cba440-5936-4d19-bc3d-6d3aabc70f38\") "
Oct 30 07:14:58.048381 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:58.048298    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14cba440-5936-4d19-bc3d-6d3aabc70f38-credential-socket" pod "14cba440-5936-4d19-bc3d-6d3aabc70f38" (UID: "14cba440-5936-4d19-bc3d-6d3aabc70f38"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:14:58.048520 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:58.048452    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14cba440-5936-4d19-bc3d-6d3aabc70f38-workload-certs" pod "14cba440-5936-4d19-bc3d-6d3aabc70f38" (UID: "14cba440-5936-4d19-bc3d-6d3aabc70f38"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:14:58.048689 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:58.048659    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14cba440-5936-4d19-bc3d-6d3aabc70f38-workload-socket" pod "14cba440-5936-4d19-bc3d-6d3aabc70f38" (UID: "14cba440-5936-4d19-bc3d-6d3aabc70f38"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:14:58.048801 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:58.048769    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14cba440-5936-4d19-bc3d-6d3aabc70f38-istiod-ca-cert" pod "14cba440-5936-4d19-bc3d-6d3aabc70f38" (UID: "14cba440-5936-4d19-bc3d-6d3aabc70f38"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:14:58.049561 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:58.049532    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14cba440-5936-4d19-bc3d-6d3aabc70f38-istio-data" pod "14cba440-5936-4d19-bc3d-6d3aabc70f38" (UID: "14cba440-5936-4d19-bc3d-6d3aabc70f38"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:14:58.052870 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:58.051524    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14cba440-5936-4d19-bc3d-6d3aabc70f38-istio-token" pod "14cba440-5936-4d19-bc3d-6d3aabc70f38" (UID: "14cba440-5936-4d19-bc3d-6d3aabc70f38"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:14:58.064197 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:58.064172    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14cba440-5936-4d19-bc3d-6d3aabc70f38-istio-envoy" pod "14cba440-5936-4d19-bc3d-6d3aabc70f38" (UID: "14cba440-5936-4d19-bc3d-6d3aabc70f38"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:14:58.064299 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:58.064255    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14cba440-5936-4d19-bc3d-6d3aabc70f38-kube-api-access-8hfr7" pod "14cba440-5936-4d19-bc3d-6d3aabc70f38" (UID: "14cba440-5936-4d19-bc3d-6d3aabc70f38"). InnerVolumeSpecName "kube-api-access-8hfr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:14:58.064527 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:58.064501    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/14cba440-5936-4d19-bc3d-6d3aabc70f38-istio-podinfo" pod "14cba440-5936-4d19-bc3d-6d3aabc70f38" (UID: "14cba440-5936-4d19-bc3d-6d3aabc70f38"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:14:58.149711 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:58.149676    6224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14cba440-5936-4d19-bc3d-6d3aabc70f38-istio-token\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:14:58.149711 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:58.149704    6224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14cba440-5936-4d19-bc3d-6d3aabc70f38-istio-data\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:14:58.149711 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:58.149714    6224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14cba440-5936-4d19-bc3d-6d3aabc70f38-istio-podinfo\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:14:58.150056 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:58.149723    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hfr7\" (UniqueName: \"kubernetes.io/projected/14cba440-5936-4d19-bc3d-6d3aabc70f38-kube-api-access-8hfr7\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:14:58.150056 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:58.149733    6224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14cba440-5936-4d19-bc3d-6d3aabc70f38-credential-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:14:58.150056 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:58.149742    6224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14cba440-5936-4d19-bc3d-6d3aabc70f38-workload-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:14:58.150056 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:58.149752    6224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14cba440-5936-4d19-bc3d-6d3aabc70f38-workload-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:14:58.150056 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:58.149760    6224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14cba440-5936-4d19-bc3d-6d3aabc70f38-istio-envoy\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:14:58.150056 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:58.149769    6224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14cba440-5936-4d19-bc3d-6d3aabc70f38-istiod-ca-cert\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:14:58.756235 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:58.756205    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv"
Oct 30 07:14:58.756235 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:58.756220    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv" event={"ID":"14cba440-5936-4d19-bc3d-6d3aabc70f38","Type":"ContainerDied","Data":"30aa2ef05fa7c0f1727cf9d3f0082f4afe5aa4aa684f274fd65ecfadbc888c3b"}
Oct 30 07:14:58.756471 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:58.756272    6224 scope.go:122] "RemoveContainer" containerID="48c3751f6a33598a039451a7f7e5f74070244415d3cb7db5db04112b200c05f0"
Oct 30 07:14:58.835224 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:58.830403    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv"]
Oct 30 07:14:58.845540 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:58.845509    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-a163b2-openshift-default-7547cc8bdc-bxssv"]
Oct 30 07:14:58.881037 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:58.881009    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm"
Oct 30 07:14:58.881417 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:58.881368    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm"
Oct 30 07:14:58.882019 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:58.881997    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm"
Oct 30 07:14:59.504323 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:14:59.504292    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14cba440-5936-4d19-bc3d-6d3aabc70f38" path="/var/lib/kubelet/pods/14cba440-5936-4d19-bc3d-6d3aabc70f38/volumes"
Oct 30 07:15:00.164254 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:00.164225    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889075-jqz4k"]
Oct 30 07:15:00.164777 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:00.164764    6224 cpu_manager.go:414] "containerMap: removing container" podUID="14cba440-5936-4d19-bc3d-6d3aabc70f38" containerName="istio-proxy"
Oct 30 07:15:00.164846 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:00.164778    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14cba440-5936-4d19-bc3d-6d3aabc70f38" containerName="istio-proxy"
Oct 30 07:15:00.164846 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:00.164792    6224 cpu_manager.go:414] "containerMap: removing container" podUID="14cba440-5936-4d19-bc3d-6d3aabc70f38" containerName="istio-proxy"
Oct 30 07:15:00.164846 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:00.164798    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14cba440-5936-4d19-bc3d-6d3aabc70f38" containerName="istio-proxy"
Oct 30 07:15:00.164980 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:00.164921    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="14cba440-5936-4d19-bc3d-6d3aabc70f38" containerName="istio-proxy"
Oct 30 07:15:00.169789 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:00.169771    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889075-jqz4k"
Oct 30 07:15:00.185370 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:00.185348    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3bda71-29889075-qthmg"]
Oct 30 07:15:00.186065 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:00.186043    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="14cba440-5936-4d19-bc3d-6d3aabc70f38" containerName="istio-proxy"
Oct 30 07:15:00.190255 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:00.190234    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3bda71-29889075-qthmg"
Oct 30 07:15:00.202106 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:00.202088    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889075-jqz4k"]
Oct 30 07:15:00.211035 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:00.209486    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889075-5zpzs"]
Oct 30 07:15:00.229945 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:00.229921    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889075-5zpzs"
Oct 30 07:15:00.234351 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:00.234324    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f357db-29889075-r7jb7"]
Oct 30 07:15:00.246550 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:00.246525    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f357db-29889075-r7jb7"
Oct 30 07:15:00.248287 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:00.248258    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3bda71-29889075-qthmg"]
Oct 30 07:15:00.252398 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:00.252376    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f357db-29889075-r7jb7"]
Oct 30 07:15:00.255406 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:00.255383    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889075-5zpzs"]
Oct 30 07:15:00.275453 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:00.275427    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/272f437e-01c1-4ce3-b2a6-3ec21c50aab1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889075-jqz4k\" (UID: \"272f437e-01c1-4ce3-b2a6-3ec21c50aab1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889075-jqz4k"
Oct 30 07:15:00.275550 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:00.275502    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rm8z\" (UniqueName: \"kubernetes.io/projected/272f437e-01c1-4ce3-b2a6-3ec21c50aab1-kube-api-access-9rm8z\") pod \"maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889075-jqz4k\" (UID: \"272f437e-01c1-4ce3-b2a6-3ec21c50aab1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889075-jqz4k"
Oct 30 07:15:00.281339 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:00.281316    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:15:00.376541 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:00.376513    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc136a4a-c414-48a8-9759-39010f5d76e1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-f357db-29889075-r7jb7\" (UID: \"bc136a4a-c414-48a8-9759-39010f5d76e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f357db-29889075-r7jb7"
Oct 30 07:15:00.376678 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:00.376551    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zncc\" (UniqueName: \"kubernetes.io/projected/a7d79c98-6c60-4d59-b9ce-a289adc98992-kube-api-access-6zncc\") pod \"maas-api-key-cleanup-e2e-api-a-3bda71-29889075-qthmg\" (UID: \"a7d79c98-6c60-4d59-b9ce-a289adc98992\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3bda71-29889075-qthmg"
Oct 30 07:15:00.376719 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:00.376708    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32a34b72-254d-4660-8c61-edf07e947407-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889075-5zpzs\" (UID: \"32a34b72-254d-4660-8c61-edf07e947407\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889075-5zpzs"
Oct 30 07:15:00.376771 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:00.376755    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7d79c98-6c60-4d59-b9ce-a289adc98992-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-3bda71-29889075-qthmg\" (UID: \"a7d79c98-6c60-4d59-b9ce-a289adc98992\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3bda71-29889075-qthmg"
Oct 30 07:15:00.376915 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:00.376892    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/272f437e-01c1-4ce3-b2a6-3ec21c50aab1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889075-jqz4k\" (UID: \"272f437e-01c1-4ce3-b2a6-3ec21c50aab1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889075-jqz4k"
Oct 30 07:15:00.376963 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:00.376955    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2l8w\" (UniqueName: \"kubernetes.io/projected/bc136a4a-c414-48a8-9759-39010f5d76e1-kube-api-access-n2l8w\") pod \"maas-api-key-cleanup-e2e-api-b-f357db-29889075-r7jb7\" (UID: \"bc136a4a-c414-48a8-9759-39010f5d76e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f357db-29889075-r7jb7"
Oct 30 07:15:00.377004 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:00.376976    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rm8z\" (UniqueName: \"kubernetes.io/projected/272f437e-01c1-4ce3-b2a6-3ec21c50aab1-kube-api-access-9rm8z\") pod \"maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889075-jqz4k\" (UID: \"272f437e-01c1-4ce3-b2a6-3ec21c50aab1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889075-jqz4k"
Oct 30 07:15:00.377068 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:00.377022    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zfdt\" (UniqueName: \"kubernetes.io/projected/32a34b72-254d-4660-8c61-edf07e947407-kube-api-access-2zfdt\") pod \"maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889075-5zpzs\" (UID: \"32a34b72-254d-4660-8c61-edf07e947407\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889075-5zpzs"
Oct 30 07:15:00.377654 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:00.377631    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/272f437e-01c1-4ce3-b2a6-3ec21c50aab1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889075-jqz4k\" (UID: \"272f437e-01c1-4ce3-b2a6-3ec21c50aab1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889075-jqz4k"
Oct 30 07:15:00.402187 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:00.402163    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rm8z\" (UniqueName: \"kubernetes.io/projected/272f437e-01c1-4ce3-b2a6-3ec21c50aab1-kube-api-access-9rm8z\") pod \"maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889075-jqz4k\" (UID: \"272f437e-01c1-4ce3-b2a6-3ec21c50aab1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889075-jqz4k"
Oct 30 07:15:00.478100 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:00.478040    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32a34b72-254d-4660-8c61-edf07e947407-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889075-5zpzs\" (UID: \"32a34b72-254d-4660-8c61-edf07e947407\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889075-5zpzs"
Oct 30 07:15:00.478218 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:00.478101    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7d79c98-6c60-4d59-b9ce-a289adc98992-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-3bda71-29889075-qthmg\" (UID: \"a7d79c98-6c60-4d59-b9ce-a289adc98992\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3bda71-29889075-qthmg"
Oct 30 07:15:00.478218 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:00.478158    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2l8w\" (UniqueName: \"kubernetes.io/projected/bc136a4a-c414-48a8-9759-39010f5d76e1-kube-api-access-n2l8w\") pod \"maas-api-key-cleanup-e2e-api-b-f357db-29889075-r7jb7\" (UID: \"bc136a4a-c414-48a8-9759-39010f5d76e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f357db-29889075-r7jb7"
Oct 30 07:15:00.478333 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:00.478214    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zfdt\" (UniqueName: \"kubernetes.io/projected/32a34b72-254d-4660-8c61-edf07e947407-kube-api-access-2zfdt\") pod \"maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889075-5zpzs\" (UID: \"32a34b72-254d-4660-8c61-edf07e947407\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889075-5zpzs"
Oct 30 07:15:00.478333 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:00.478269    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc136a4a-c414-48a8-9759-39010f5d76e1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-f357db-29889075-r7jb7\" (UID: \"bc136a4a-c414-48a8-9759-39010f5d76e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f357db-29889075-r7jb7"
Oct 30 07:15:00.478333 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:00.478298    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zncc\" (UniqueName: \"kubernetes.io/projected/a7d79c98-6c60-4d59-b9ce-a289adc98992-kube-api-access-6zncc\") pod \"maas-api-key-cleanup-e2e-api-a-3bda71-29889075-qthmg\" (UID: \"a7d79c98-6c60-4d59-b9ce-a289adc98992\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3bda71-29889075-qthmg"
Oct 30 07:15:00.478721 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:00.478696    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32a34b72-254d-4660-8c61-edf07e947407-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889075-5zpzs\" (UID: \"32a34b72-254d-4660-8c61-edf07e947407\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889075-5zpzs"
Oct 30 07:15:00.478866 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:00.478847    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7d79c98-6c60-4d59-b9ce-a289adc98992-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-3bda71-29889075-qthmg\" (UID: \"a7d79c98-6c60-4d59-b9ce-a289adc98992\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3bda71-29889075-qthmg"
Oct 30 07:15:00.479056 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:00.479038    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc136a4a-c414-48a8-9759-39010f5d76e1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-f357db-29889075-r7jb7\" (UID: \"bc136a4a-c414-48a8-9759-39010f5d76e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f357db-29889075-r7jb7"
Oct 30 07:15:00.484173 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:00.484153    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889075-jqz4k"
Oct 30 07:15:00.503594 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:00.503573    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zncc\" (UniqueName: \"kubernetes.io/projected/a7d79c98-6c60-4d59-b9ce-a289adc98992-kube-api-access-6zncc\") pod \"maas-api-key-cleanup-e2e-api-a-3bda71-29889075-qthmg\" (UID: \"a7d79c98-6c60-4d59-b9ce-a289adc98992\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3bda71-29889075-qthmg"
Oct 30 07:15:00.503830 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:00.503797    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2l8w\" (UniqueName: \"kubernetes.io/projected/bc136a4a-c414-48a8-9759-39010f5d76e1-kube-api-access-n2l8w\") pod \"maas-api-key-cleanup-e2e-api-b-f357db-29889075-r7jb7\" (UID: \"bc136a4a-c414-48a8-9759-39010f5d76e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f357db-29889075-r7jb7"
Oct 30 07:15:00.503915 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:00.503854    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zfdt\" (UniqueName: \"kubernetes.io/projected/32a34b72-254d-4660-8c61-edf07e947407-kube-api-access-2zfdt\") pod \"maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889075-5zpzs\" (UID: \"32a34b72-254d-4660-8c61-edf07e947407\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889075-5zpzs"
Oct 30 07:15:00.548274 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:00.548248    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3bda71-29889075-qthmg"
Oct 30 07:15:00.562401 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:00.562373    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889075-5zpzs"
Oct 30 07:15:00.570839 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:00.569668    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f357db-29889075-r7jb7"
Oct 30 07:15:00.785312 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:00.785285    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889075-jqz4k"]
Oct 30 07:15:00.785463 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:15:00.785430    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod272f437e_01c1_4ce3_b2a6_3ec21c50aab1.slice/crio-580d5652c9a602c8b63110af3dd9ca4579b39e70f43542c609a1712c6e1515f8 WatchSource:0}: Error finding container 580d5652c9a602c8b63110af3dd9ca4579b39e70f43542c609a1712c6e1515f8: Status 404 returned error can't find the container with id 580d5652c9a602c8b63110af3dd9ca4579b39e70f43542c609a1712c6e1515f8
Oct 30 07:15:00.890995 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:15:00.890965    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7d79c98_6c60_4d59_b9ce_a289adc98992.slice/crio-bb60f486b2aecda303c5194883c7ac4502d660756117fb39267c98818c519e06 WatchSource:0}: Error finding container bb60f486b2aecda303c5194883c7ac4502d660756117fb39267c98818c519e06: Status 404 returned error can't find the container with id bb60f486b2aecda303c5194883c7ac4502d660756117fb39267c98818c519e06
Oct 30 07:15:00.892782 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:00.892720    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3bda71-29889075-qthmg"]
Oct 30 07:15:00.896852 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:00.896802    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a6a14e07-84d574ddfc-7whtt"]
Oct 30 07:15:00.898767 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:15:00.898741    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc136a4a_c414_48a8_9759_39010f5d76e1.slice/crio-a0d97717e6e57f55709eaab83aa9ec06b9c3235ab33628dd301d1ad621f71cad WatchSource:0}: Error finding container a0d97717e6e57f55709eaab83aa9ec06b9c3235ab33628dd301d1ad621f71cad: Status 404 returned error can't find the container with id a0d97717e6e57f55709eaab83aa9ec06b9c3235ab33628dd301d1ad621f71cad
Oct 30 07:15:00.902370 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:15:00.902351    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32a34b72_254d_4660_8c61_edf07e947407.slice/crio-870ad11af8b02b1022990a4d2f848345f6c952aa60e3b88bd8b6a0b4c54ffb79 WatchSource:0}: Error finding container 870ad11af8b02b1022990a4d2f848345f6c952aa60e3b88bd8b6a0b4c54ffb79: Status 404 returned error can't find the container with id 870ad11af8b02b1022990a4d2f848345f6c952aa60e3b88bd8b6a0b4c54ffb79
Oct 30 07:15:00.902966 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:00.902946    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f357db-29889075-r7jb7"]
Oct 30 07:15:00.903064 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:00.903036    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6a14e07-84d574ddfc-7whtt"
Oct 30 07:15:00.906684 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:00.906661    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889075-5zpzs"]
Oct 30 07:15:00.920047 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:00.920028    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-a6a14e07-serving-cert\""
Oct 30 07:15:00.930563 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:00.930540    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a6a14e07-84d574ddfc-7whtt"]
Oct 30 07:15:01.086303 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:01.086228    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee-openshift-service-ca\") pod \"maas-api-e2e-mt-a6a14e07-84d574ddfc-7whtt\" (UID: \"fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6a14e07-84d574ddfc-7whtt"
Oct 30 07:15:01.086303 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:01.086287    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdb47\" (UniqueName: \"kubernetes.io/projected/fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee-kube-api-access-fdb47\") pod \"maas-api-e2e-mt-a6a14e07-84d574ddfc-7whtt\" (UID: \"fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6a14e07-84d574ddfc-7whtt"
Oct 30 07:15:01.086492 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:01.086398    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee-maas-api-tls\") pod \"maas-api-e2e-mt-a6a14e07-84d574ddfc-7whtt\" (UID: \"fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6a14e07-84d574ddfc-7whtt"
Oct 30 07:15:01.086557 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:01.086542    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee-metrics-certs\") pod \"maas-api-e2e-mt-a6a14e07-84d574ddfc-7whtt\" (UID: \"fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6a14e07-84d574ddfc-7whtt"
Oct 30 07:15:01.188292 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:01.188254    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee-maas-api-tls\") pod \"maas-api-e2e-mt-a6a14e07-84d574ddfc-7whtt\" (UID: \"fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6a14e07-84d574ddfc-7whtt"
Oct 30 07:15:01.188764 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:15:01.188428    6224 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-a6a14e07-serving-cert: secret "maas-api-e2e-mt-a6a14e07-serving-cert" not found
Oct 30 07:15:01.188764 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:01.188441    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee-metrics-certs\") pod \"maas-api-e2e-mt-a6a14e07-84d574ddfc-7whtt\" (UID: \"fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6a14e07-84d574ddfc-7whtt"
Oct 30 07:15:01.188764 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:15:01.188509    6224 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee-maas-api-tls podName:fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee nodeName:}" failed. No retries permitted until 2026-10-30 07:15:01.688487051 +0000 UTC m=+1060.297894021 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee-maas-api-tls") pod "maas-api-e2e-mt-a6a14e07-84d574ddfc-7whtt" (UID: "fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee") : secret "maas-api-e2e-mt-a6a14e07-serving-cert" not found
Oct 30 07:15:01.188764 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:01.188684    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee-openshift-service-ca\") pod \"maas-api-e2e-mt-a6a14e07-84d574ddfc-7whtt\" (UID: \"fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6a14e07-84d574ddfc-7whtt"
Oct 30 07:15:01.188764 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:01.188753    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdb47\" (UniqueName: \"kubernetes.io/projected/fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee-kube-api-access-fdb47\") pod \"maas-api-e2e-mt-a6a14e07-84d574ddfc-7whtt\" (UID: \"fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6a14e07-84d574ddfc-7whtt"
Oct 30 07:15:01.189431 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:01.189411    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee-openshift-service-ca\") pod \"maas-api-e2e-mt-a6a14e07-84d574ddfc-7whtt\" (UID: \"fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6a14e07-84d574ddfc-7whtt"
Oct 30 07:15:01.191592 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:01.191573    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee-metrics-certs\") pod \"maas-api-e2e-mt-a6a14e07-84d574ddfc-7whtt\" (UID: \"fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6a14e07-84d574ddfc-7whtt"
Oct 30 07:15:01.214007 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:01.213978    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdb47\" (UniqueName: \"kubernetes.io/projected/fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee-kube-api-access-fdb47\") pod \"maas-api-e2e-mt-a6a14e07-84d574ddfc-7whtt\" (UID: \"fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6a14e07-84d574ddfc-7whtt"
Oct 30 07:15:01.696477 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:01.696441    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee-maas-api-tls\") pod \"maas-api-e2e-mt-a6a14e07-84d574ddfc-7whtt\" (UID: \"fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6a14e07-84d574ddfc-7whtt"
Oct 30 07:15:01.698961 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:01.698927    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee-maas-api-tls\") pod \"maas-api-e2e-mt-a6a14e07-84d574ddfc-7whtt\" (UID: \"fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6a14e07-84d574ddfc-7whtt"
Oct 30 07:15:01.780954 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:01.780870    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889075-5zpzs" event={"ID":"32a34b72-254d-4660-8c61-edf07e947407","Type":"ContainerStarted","Data":"870ad11af8b02b1022990a4d2f848345f6c952aa60e3b88bd8b6a0b4c54ffb79"}
Oct 30 07:15:01.782728 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:01.782696    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f357db-29889075-r7jb7" event={"ID":"bc136a4a-c414-48a8-9759-39010f5d76e1","Type":"ContainerStarted","Data":"a0d97717e6e57f55709eaab83aa9ec06b9c3235ab33628dd301d1ad621f71cad"}
Oct 30 07:15:01.784538 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:01.784512    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889075-jqz4k" event={"ID":"272f437e-01c1-4ce3-b2a6-3ec21c50aab1","Type":"ContainerStarted","Data":"580d5652c9a602c8b63110af3dd9ca4579b39e70f43542c609a1712c6e1515f8"}
Oct 30 07:15:01.786121 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:01.786076    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3bda71-29889075-qthmg" event={"ID":"a7d79c98-6c60-4d59-b9ce-a289adc98992","Type":"ContainerStarted","Data":"bb60f486b2aecda303c5194883c7ac4502d660756117fb39267c98818c519e06"}
Oct 30 07:15:01.817984 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:01.817949    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6a14e07-84d574ddfc-7whtt"
Oct 30 07:15:01.894441 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:15:01.894369    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:15:02.192168 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:02.189114    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a6a14e07-84d574ddfc-7whtt"]
Oct 30 07:15:02.192168 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:15:02.189768    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd5b0ba9_f329_49c2_80a2_f1ea3f5a09ee.slice/crio-96e48e00565dcc1491d3d2f5d53f4ceb4c67639beea0934a23c701f49daf3e1c WatchSource:0}: Error finding container 96e48e00565dcc1491d3d2f5d53f4ceb4c67639beea0934a23c701f49daf3e1c: Status 404 returned error can't find the container with id 96e48e00565dcc1491d3d2f5d53f4ceb4c67639beea0934a23c701f49daf3e1c
Oct 30 07:15:02.701239 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:02.701205    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-a6a14e07-bdc5d9c5f-j5wkj"
Oct 30 07:15:02.804866 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:02.804787    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6a14e07-84d574ddfc-7whtt" event={"ID":"fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee","Type":"ContainerStarted","Data":"5ceac104f1b97c77de0c5d629ce7c019120202e7180b027da4ff166dce515f57"}
Oct 30 07:15:02.805122 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:02.804891    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6a14e07-84d574ddfc-7whtt" event={"ID":"fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee","Type":"ContainerStarted","Data":"96e48e00565dcc1491d3d2f5d53f4ceb4c67639beea0934a23c701f49daf3e1c"}
Oct 30 07:15:02.805122 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:02.805102    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6a14e07-84d574ddfc-7whtt"
Oct 30 07:15:02.856160 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:02.853491    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6a14e07-84d574ddfc-7whtt" podStartSLOduration=2.471973733 podStartE2EDuration="2.85347178s" totalImagesPullingTime="381.498047ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:15:02.851195931 +0000 UTC m=+1061.460602929" watchObservedRunningTime="2026-10-30 07:15:02.85347178 +0000 UTC m=+1061.462878751"
Oct 30 07:15:05.859493 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:05.859456    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3bda71-29889075-qthmg" event={"ID":"a7d79c98-6c60-4d59-b9ce-a289adc98992","Type":"ContainerStarted","Data":"9335d931d641b212fcd5e523b6c19251904f2009d2ae6c1314116685242f0979"}
Oct 30 07:15:05.904532 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:05.904483    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3bda71-29889075-qthmg" podStartSLOduration=1.5725809769999999 podStartE2EDuration="5.90446731s" totalImagesPullingTime="4.331886333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:15:05.903338379 +0000 UTC m=+1064.512745349" watchObservedRunningTime="2026-10-30 07:15:05.90446731 +0000 UTC m=+1064.513874279"
Oct 30 07:15:06.736793 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:06.736765    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-4e42c8-64d8f55c96-6wxtx"
Oct 30 07:15:06.736971 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:06.736860    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4e42c8-658945d447-wkthc"
Oct 30 07:15:06.868093 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:06.868005    6224 generic.go:356] "Generic (PLEG): container finished" podID="bc136a4a-c414-48a8-9759-39010f5d76e1" containerID="58c4f0107f7486b23392c867415415d477076a50a5e2c31bae991d8a2f1358b0" exitCode=60
Oct 30 07:15:06.868536 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:06.868081    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f357db-29889075-r7jb7" event={"ID":"bc136a4a-c414-48a8-9759-39010f5d76e1","Type":"ContainerDied","Data":"58c4f0107f7486b23392c867415415d477076a50a5e2c31bae991d8a2f1358b0"}
Oct 30 07:15:06.868536 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:06.868475    6224 scope.go:122] "RemoveContainer" containerID="58c4f0107f7486b23392c867415415d477076a50a5e2c31bae991d8a2f1358b0"
Oct 30 07:15:06.869961 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:06.869941    6224 generic.go:356] "Generic (PLEG): container finished" podID="272f437e-01c1-4ce3-b2a6-3ec21c50aab1" containerID="0bb23ab295dcf4cf029353491b2c020e4aadd94609c363a87d6ae1375e5511b6" exitCode=60
Oct 30 07:15:06.870090 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:06.870027    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889075-jqz4k" event={"ID":"272f437e-01c1-4ce3-b2a6-3ec21c50aab1","Type":"ContainerDied","Data":"0bb23ab295dcf4cf029353491b2c020e4aadd94609c363a87d6ae1375e5511b6"}
Oct 30 07:15:06.870310 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:06.870289    6224 scope.go:122] "RemoveContainer" containerID="0bb23ab295dcf4cf029353491b2c020e4aadd94609c363a87d6ae1375e5511b6"
Oct 30 07:15:06.871927 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:06.871899    6224 generic.go:356] "Generic (PLEG): container finished" podID="a7d79c98-6c60-4d59-b9ce-a289adc98992" containerID="9335d931d641b212fcd5e523b6c19251904f2009d2ae6c1314116685242f0979" exitCode=60
Oct 30 07:15:06.872015 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:06.871966    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3bda71-29889075-qthmg" event={"ID":"a7d79c98-6c60-4d59-b9ce-a289adc98992","Type":"ContainerDied","Data":"9335d931d641b212fcd5e523b6c19251904f2009d2ae6c1314116685242f0979"}
Oct 30 07:15:06.872231 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:06.872214    6224 scope.go:122] "RemoveContainer" containerID="9335d931d641b212fcd5e523b6c19251904f2009d2ae6c1314116685242f0979"
Oct 30 07:15:06.873926 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:06.873899    6224 generic.go:356] "Generic (PLEG): container finished" podID="32a34b72-254d-4660-8c61-edf07e947407" containerID="5ed3da087ba820fe39a2663f424253636c63d3626b6da2deb9f85e5b207bdbcc" exitCode=60
Oct 30 07:15:06.874034 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:06.873941    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889075-5zpzs" event={"ID":"32a34b72-254d-4660-8c61-edf07e947407","Type":"ContainerDied","Data":"5ed3da087ba820fe39a2663f424253636c63d3626b6da2deb9f85e5b207bdbcc"}
Oct 30 07:15:06.874204 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:06.874175    6224 scope.go:122] "RemoveContainer" containerID="5ed3da087ba820fe39a2663f424253636c63d3626b6da2deb9f85e5b207bdbcc"
Oct 30 07:15:07.884058 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:07.884030    6224 generic.go:356] "Generic (PLEG): container finished" podID="bc136a4a-c414-48a8-9759-39010f5d76e1" containerID="b73457faa1eae37ea0207b73ffcd6e3636b33ec52c0916f40f93488c7a276c19" exitCode=60
Oct 30 07:15:07.884448 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:07.884069    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f357db-29889075-r7jb7" event={"ID":"bc136a4a-c414-48a8-9759-39010f5d76e1","Type":"ContainerDied","Data":"b73457faa1eae37ea0207b73ffcd6e3636b33ec52c0916f40f93488c7a276c19"}
Oct 30 07:15:07.884448 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:07.884133    6224 scope.go:122] "RemoveContainer" containerID="58c4f0107f7486b23392c867415415d477076a50a5e2c31bae991d8a2f1358b0"
Oct 30 07:15:07.884448 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:07.884434    6224 scope.go:122] "RemoveContainer" containerID="b73457faa1eae37ea0207b73ffcd6e3636b33ec52c0916f40f93488c7a276c19"
Oct 30 07:15:07.884816 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:15:07.884776    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-f357db-29889075-r7jb7_odh-ai-gateway-infra(bc136a4a-c414-48a8-9759-39010f5d76e1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f357db-29889075-r7jb7" podUID="bc136a4a-c414-48a8-9759-39010f5d76e1"
Oct 30 07:15:07.886245 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:07.886225    6224 generic.go:356] "Generic (PLEG): container finished" podID="272f437e-01c1-4ce3-b2a6-3ec21c50aab1" containerID="865809e270ec7271a5c5c5c688df0748d9a65633edd8163dc517b6e3533d64f4" exitCode=60
Oct 30 07:15:07.886363 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:07.886315    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889075-jqz4k" event={"ID":"272f437e-01c1-4ce3-b2a6-3ec21c50aab1","Type":"ContainerDied","Data":"865809e270ec7271a5c5c5c688df0748d9a65633edd8163dc517b6e3533d64f4"}
Oct 30 07:15:07.886573 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:07.886554    6224 scope.go:122] "RemoveContainer" containerID="865809e270ec7271a5c5c5c688df0748d9a65633edd8163dc517b6e3533d64f4"
Oct 30 07:15:07.886943 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:15:07.886924    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889075-jqz4k_odh-ai-gateway-infra(272f437e-01c1-4ce3-b2a6-3ec21c50aab1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889075-jqz4k" podUID="272f437e-01c1-4ce3-b2a6-3ec21c50aab1"
Oct 30 07:15:07.888376 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:07.888348    6224 generic.go:356] "Generic (PLEG): container finished" podID="a7d79c98-6c60-4d59-b9ce-a289adc98992" containerID="31e6bf497e7c0c08e46962b82423716be1f04e3bf86c0eb1d4b4153ebbfa66a0" exitCode=60
Oct 30 07:15:07.888496 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:07.888480    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3bda71-29889075-qthmg" event={"ID":"a7d79c98-6c60-4d59-b9ce-a289adc98992","Type":"ContainerDied","Data":"31e6bf497e7c0c08e46962b82423716be1f04e3bf86c0eb1d4b4153ebbfa66a0"}
Oct 30 07:15:07.888749 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:07.888735    6224 scope.go:122] "RemoveContainer" containerID="31e6bf497e7c0c08e46962b82423716be1f04e3bf86c0eb1d4b4153ebbfa66a0"
Oct 30 07:15:07.888967 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:15:07.888949    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-3bda71-29889075-qthmg_odh-ai-gateway-infra(a7d79c98-6c60-4d59-b9ce-a289adc98992)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3bda71-29889075-qthmg" podUID="a7d79c98-6c60-4d59-b9ce-a289adc98992"
Oct 30 07:15:07.891176 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:07.891158    6224 generic.go:356] "Generic (PLEG): container finished" podID="32a34b72-254d-4660-8c61-edf07e947407" containerID="1143e4423565aba8dd8c1717fbc321835f5358c879e6b35ebebdcb0df7aacfe3" exitCode=60
Oct 30 07:15:07.891254 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:07.891190    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889075-5zpzs" event={"ID":"32a34b72-254d-4660-8c61-edf07e947407","Type":"ContainerDied","Data":"1143e4423565aba8dd8c1717fbc321835f5358c879e6b35ebebdcb0df7aacfe3"}
Oct 30 07:15:07.891404 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:07.891389    6224 scope.go:122] "RemoveContainer" containerID="1143e4423565aba8dd8c1717fbc321835f5358c879e6b35ebebdcb0df7aacfe3"
Oct 30 07:15:07.891585 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:15:07.891568    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889075-5zpzs_odh-ai-gateway-infra(32a34b72-254d-4660-8c61-edf07e947407)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889075-5zpzs" podUID="32a34b72-254d-4660-8c61-edf07e947407"
Oct 30 07:15:07.994032 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:07.994006    6224 scope.go:122] "RemoveContainer" containerID="0bb23ab295dcf4cf029353491b2c020e4aadd94609c363a87d6ae1375e5511b6"
Oct 30 07:15:08.008459 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:08.008436    6224 scope.go:122] "RemoveContainer" containerID="9335d931d641b212fcd5e523b6c19251904f2009d2ae6c1314116685242f0979"
Oct 30 07:15:08.023799 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:08.023775    6224 scope.go:122] "RemoveContainer" containerID="5ed3da087ba820fe39a2663f424253636c63d3626b6da2deb9f85e5b207bdbcc"
Oct 30 07:15:08.822918 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:08.822889    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6a14e07-84d574ddfc-7whtt"
Oct 30 07:15:08.904909 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:08.904874    6224 scope.go:122] "RemoveContainer" containerID="31e6bf497e7c0c08e46962b82423716be1f04e3bf86c0eb1d4b4153ebbfa66a0"
Oct 30 07:15:08.905401 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:15:08.905247    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-3bda71-29889075-qthmg_odh-ai-gateway-infra(a7d79c98-6c60-4d59-b9ce-a289adc98992)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3bda71-29889075-qthmg" podUID="a7d79c98-6c60-4d59-b9ce-a289adc98992"
Oct 30 07:15:08.907013 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:08.906987    6224 scope.go:122] "RemoveContainer" containerID="1143e4423565aba8dd8c1717fbc321835f5358c879e6b35ebebdcb0df7aacfe3"
Oct 30 07:15:08.907387 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:15:08.907349    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889075-5zpzs_odh-ai-gateway-infra(32a34b72-254d-4660-8c61-edf07e947407)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889075-5zpzs" podUID="32a34b72-254d-4660-8c61-edf07e947407"
Oct 30 07:15:08.908926 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:08.908906    6224 scope.go:122] "RemoveContainer" containerID="b73457faa1eae37ea0207b73ffcd6e3636b33ec52c0916f40f93488c7a276c19"
Oct 30 07:15:08.909240 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:15:08.909213    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-f357db-29889075-r7jb7_odh-ai-gateway-infra(bc136a4a-c414-48a8-9759-39010f5d76e1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f357db-29889075-r7jb7" podUID="bc136a4a-c414-48a8-9759-39010f5d76e1"
Oct 30 07:15:08.910715 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:08.910694    6224 scope.go:122] "RemoveContainer" containerID="865809e270ec7271a5c5c5c688df0748d9a65633edd8163dc517b6e3533d64f4"
Oct 30 07:15:08.911044 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:15:08.911006    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889075-jqz4k_odh-ai-gateway-infra(272f437e-01c1-4ce3-b2a6-3ec21c50aab1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889075-jqz4k" podUID="272f437e-01c1-4ce3-b2a6-3ec21c50aab1"
Oct 30 07:15:09.264467 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:09.264377    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3bda71-29889075-qthmg"]
Oct 30 07:15:10.103899 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:10.103874    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3bda71-29889075-qthmg"
Oct 30 07:15:10.198581 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:10.198499    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7d79c98-6c60-4d59-b9ce-a289adc98992-kube-api-access-6zncc\" (UniqueName: \"kubernetes.io/projected/a7d79c98-6c60-4d59-b9ce-a289adc98992-kube-api-access-6zncc\") pod \"a7d79c98-6c60-4d59-b9ce-a289adc98992\" (UID: \"a7d79c98-6c60-4d59-b9ce-a289adc98992\") "
Oct 30 07:15:10.198714 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:10.198660    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7d79c98-6c60-4d59-b9ce-a289adc98992-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7d79c98-6c60-4d59-b9ce-a289adc98992-maas-api-service-ca\") pod \"a7d79c98-6c60-4d59-b9ce-a289adc98992\" (UID: \"a7d79c98-6c60-4d59-b9ce-a289adc98992\") "
Oct 30 07:15:10.199062 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:10.199037    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7d79c98-6c60-4d59-b9ce-a289adc98992-maas-api-service-ca" pod "a7d79c98-6c60-4d59-b9ce-a289adc98992" (UID: "a7d79c98-6c60-4d59-b9ce-a289adc98992"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:15:10.200629 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:10.200607    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7d79c98-6c60-4d59-b9ce-a289adc98992-kube-api-access-6zncc" pod "a7d79c98-6c60-4d59-b9ce-a289adc98992" (UID: "a7d79c98-6c60-4d59-b9ce-a289adc98992"). InnerVolumeSpecName "kube-api-access-6zncc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:15:10.300942 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:10.300907    6224 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7d79c98-6c60-4d59-b9ce-a289adc98992-maas-api-service-ca\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:15:10.300942 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:10.300937    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zncc\" (UniqueName: \"kubernetes.io/projected/a7d79c98-6c60-4d59-b9ce-a289adc98992-kube-api-access-6zncc\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:15:10.938914 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:10.938878    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3bda71-29889075-qthmg" event={"ID":"a7d79c98-6c60-4d59-b9ce-a289adc98992","Type":"ContainerDied","Data":"bb60f486b2aecda303c5194883c7ac4502d660756117fb39267c98818c519e06"}
Oct 30 07:15:10.939078 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:10.938921    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3bda71-29889075-qthmg"
Oct 30 07:15:10.939078 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:10.938940    6224 scope.go:122] "RemoveContainer" containerID="31e6bf497e7c0c08e46962b82423716be1f04e3bf86c0eb1d4b4153ebbfa66a0"
Oct 30 07:15:11.000646 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:11.000615    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3bda71-29889075-qthmg"]
Oct 30 07:15:11.006647 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:11.006623    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3bda71-29889075-qthmg"]
Oct 30 07:15:11.511561 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:11.511522    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7d79c98-6c60-4d59-b9ce-a289adc98992" path="/var/lib/kubelet/pods/a7d79c98-6c60-4d59-b9ce-a289adc98992/volumes"
Oct 30 07:15:18.850098 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:18.850060    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dpzbb"]
Oct 30 07:15:18.850924 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:18.850903    6224 cpu_manager.go:414] "containerMap: removing container" podUID="a7d79c98-6c60-4d59-b9ce-a289adc98992" containerName="cleanup"
Oct 30 07:15:18.850924 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:18.850924    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7d79c98-6c60-4d59-b9ce-a289adc98992" containerName="cleanup"
Oct 30 07:15:18.851088 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:18.850953    6224 cpu_manager.go:414] "containerMap: removing container" podUID="a7d79c98-6c60-4d59-b9ce-a289adc98992" containerName="cleanup"
Oct 30 07:15:18.851088 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:18.850960    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7d79c98-6c60-4d59-b9ce-a289adc98992" containerName="cleanup"
Oct 30 07:15:18.851195 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:18.851181    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7d79c98-6c60-4d59-b9ce-a289adc98992" containerName="cleanup"
Oct 30 07:15:18.851249 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:18.851203    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7d79c98-6c60-4d59-b9ce-a289adc98992" containerName="cleanup"
Oct 30 07:15:18.856048 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:18.856027    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dpzbb"
Oct 30 07:15:18.880716 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:18.880693    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dpzbb"]
Oct 30 07:15:18.889897 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:18.889877    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 07:15:18.890018 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:18.889902    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 07:15:18.890018 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:18.889911    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 07:15:18.890018 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:18.889995    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-xjg2r\""
Oct 30 07:15:18.914852 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:18.914764    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bf2e54e1-8437-47eb-bfa7-04bde103ebf8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dpzbb\" (UID: \"bf2e54e1-8437-47eb-bfa7-04bde103ebf8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dpzbb"
Oct 30 07:15:18.914852 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:18.914820    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bf2e54e1-8437-47eb-bfa7-04bde103ebf8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dpzbb\" (UID: \"bf2e54e1-8437-47eb-bfa7-04bde103ebf8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dpzbb"
Oct 30 07:15:18.915230 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:18.914896    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmtpp\" (UniqueName: \"kubernetes.io/projected/bf2e54e1-8437-47eb-bfa7-04bde103ebf8-kube-api-access-kmtpp\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dpzbb\" (UID: \"bf2e54e1-8437-47eb-bfa7-04bde103ebf8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dpzbb"
Oct 30 07:15:18.916298 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:18.914938    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf2e54e1-8437-47eb-bfa7-04bde103ebf8-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dpzbb\" (UID: \"bf2e54e1-8437-47eb-bfa7-04bde103ebf8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dpzbb"
Oct 30 07:15:18.916616 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:18.916592    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bf2e54e1-8437-47eb-bfa7-04bde103ebf8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dpzbb\" (UID: \"bf2e54e1-8437-47eb-bfa7-04bde103ebf8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dpzbb"
Oct 30 07:15:18.916792 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:18.916772    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf2e54e1-8437-47eb-bfa7-04bde103ebf8-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dpzbb\" (UID: \"bf2e54e1-8437-47eb-bfa7-04bde103ebf8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dpzbb"
Oct 30 07:15:18.916917 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:18.916845    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bf2e54e1-8437-47eb-bfa7-04bde103ebf8-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dpzbb\" (UID: \"bf2e54e1-8437-47eb-bfa7-04bde103ebf8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dpzbb"
Oct 30 07:15:19.019067 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:19.019029    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmtpp\" (UniqueName: \"kubernetes.io/projected/bf2e54e1-8437-47eb-bfa7-04bde103ebf8-kube-api-access-kmtpp\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dpzbb\" (UID: \"bf2e54e1-8437-47eb-bfa7-04bde103ebf8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dpzbb"
Oct 30 07:15:19.019275 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:19.019094    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf2e54e1-8437-47eb-bfa7-04bde103ebf8-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dpzbb\" (UID: \"bf2e54e1-8437-47eb-bfa7-04bde103ebf8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dpzbb"
Oct 30 07:15:19.019275 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:19.019123    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bf2e54e1-8437-47eb-bfa7-04bde103ebf8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dpzbb\" (UID: \"bf2e54e1-8437-47eb-bfa7-04bde103ebf8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dpzbb"
Oct 30 07:15:19.019275 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:19.019198    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf2e54e1-8437-47eb-bfa7-04bde103ebf8-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dpzbb\" (UID: \"bf2e54e1-8437-47eb-bfa7-04bde103ebf8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dpzbb"
Oct 30 07:15:19.019275 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:19.019250    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bf2e54e1-8437-47eb-bfa7-04bde103ebf8-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dpzbb\" (UID: \"bf2e54e1-8437-47eb-bfa7-04bde103ebf8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dpzbb"
Oct 30 07:15:19.019487 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:19.019464    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bf2e54e1-8437-47eb-bfa7-04bde103ebf8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dpzbb\" (UID: \"bf2e54e1-8437-47eb-bfa7-04bde103ebf8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dpzbb"
Oct 30 07:15:19.019758 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:19.019590    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bf2e54e1-8437-47eb-bfa7-04bde103ebf8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dpzbb\" (UID: \"bf2e54e1-8437-47eb-bfa7-04bde103ebf8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dpzbb"
Oct 30 07:15:19.020355 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:19.020330    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bf2e54e1-8437-47eb-bfa7-04bde103ebf8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dpzbb\" (UID: \"bf2e54e1-8437-47eb-bfa7-04bde103ebf8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dpzbb"
Oct 30 07:15:19.022127 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:19.022105    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bf2e54e1-8437-47eb-bfa7-04bde103ebf8-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dpzbb\" (UID: \"bf2e54e1-8437-47eb-bfa7-04bde103ebf8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dpzbb"
Oct 30 07:15:19.022254 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:19.022159    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf2e54e1-8437-47eb-bfa7-04bde103ebf8-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dpzbb\" (UID: \"bf2e54e1-8437-47eb-bfa7-04bde103ebf8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dpzbb"
Oct 30 07:15:19.022254 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:19.022228    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bf2e54e1-8437-47eb-bfa7-04bde103ebf8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dpzbb\" (UID: \"bf2e54e1-8437-47eb-bfa7-04bde103ebf8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dpzbb"
Oct 30 07:15:19.022738 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:19.022715    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bf2e54e1-8437-47eb-bfa7-04bde103ebf8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dpzbb\" (UID: \"bf2e54e1-8437-47eb-bfa7-04bde103ebf8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dpzbb"
Oct 30 07:15:19.046892 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:19.046862    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmtpp\" (UniqueName: \"kubernetes.io/projected/bf2e54e1-8437-47eb-bfa7-04bde103ebf8-kube-api-access-kmtpp\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dpzbb\" (UID: \"bf2e54e1-8437-47eb-bfa7-04bde103ebf8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dpzbb"
Oct 30 07:15:19.047350 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:19.047326    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf2e54e1-8437-47eb-bfa7-04bde103ebf8-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dpzbb\" (UID: \"bf2e54e1-8437-47eb-bfa7-04bde103ebf8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dpzbb"
Oct 30 07:15:19.169240 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:19.169155    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dpzbb"
Oct 30 07:15:19.488887 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:15:19.488852    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf2e54e1_8437_47eb_bfa7_04bde103ebf8.slice/crio-9e1303123eaee352393fcb5f69aa7c345a593beb874f5bf0b11961ecb32949ca WatchSource:0}: Error finding container 9e1303123eaee352393fcb5f69aa7c345a593beb874f5bf0b11961ecb32949ca: Status 404 returned error can't find the container with id 9e1303123eaee352393fcb5f69aa7c345a593beb874f5bf0b11961ecb32949ca
Oct 30 07:15:19.490498 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:19.490472    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dpzbb"]
Oct 30 07:15:19.497097 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:19.497073    6224 scope.go:122] "RemoveContainer" containerID="b73457faa1eae37ea0207b73ffcd6e3636b33ec52c0916f40f93488c7a276c19"
Oct 30 07:15:19.497217 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:19.497198    6224 scope.go:122] "RemoveContainer" containerID="865809e270ec7271a5c5c5c688df0748d9a65633edd8163dc517b6e3533d64f4"
Oct 30 07:15:20.005925 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:20.005885    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dpzbb" event={"ID":"bf2e54e1-8437-47eb-bfa7-04bde103ebf8","Type":"ContainerStarted","Data":"9e1303123eaee352393fcb5f69aa7c345a593beb874f5bf0b11961ecb32949ca"}
Oct 30 07:15:20.007823 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:20.007783    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f357db-29889075-r7jb7" event={"ID":"bc136a4a-c414-48a8-9759-39010f5d76e1","Type":"ContainerStarted","Data":"497eedfb909f8edcd780f7137c9405874291f4c8b843dc1d2e7f53f27efefe0a"}
Oct 30 07:15:20.009521 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:20.009499    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889075-jqz4k" event={"ID":"272f437e-01c1-4ce3-b2a6-3ec21c50aab1","Type":"ContainerStarted","Data":"d64429002e163a33c15ee1e99aefe0f4c074809b9edfed3e0417bc531f332af4"}
Oct 30 07:15:20.052027 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:20.051987    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f357db-29889075-r7jb7" podStartSLOduration=15.333738608 podStartE2EDuration="20.051975091s" totalImagesPullingTime="4.718236483s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:15:20.050719888 +0000 UTC m=+1078.660126869" watchObservedRunningTime="2026-10-30 07:15:20.051975091 +0000 UTC m=+1078.661382061"
Oct 30 07:15:20.095001 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:20.094958    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889075-jqz4k" podStartSLOduration=15.262330782 podStartE2EDuration="20.094941811s" totalImagesPullingTime="4.832611029s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:15:20.093348671 +0000 UTC m=+1078.702755641" watchObservedRunningTime="2026-10-30 07:15:20.094941811 +0000 UTC m=+1078.704348789"
Oct 30 07:15:21.021230 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:21.021197    6224 generic.go:356] "Generic (PLEG): container finished" podID="bc136a4a-c414-48a8-9759-39010f5d76e1" containerID="497eedfb909f8edcd780f7137c9405874291f4c8b843dc1d2e7f53f27efefe0a" exitCode=60
Oct 30 07:15:21.021711 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:21.021254    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f357db-29889075-r7jb7" event={"ID":"bc136a4a-c414-48a8-9759-39010f5d76e1","Type":"ContainerDied","Data":"497eedfb909f8edcd780f7137c9405874291f4c8b843dc1d2e7f53f27efefe0a"}
Oct 30 07:15:21.021711 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:21.021327    6224 scope.go:122] "RemoveContainer" containerID="b73457faa1eae37ea0207b73ffcd6e3636b33ec52c0916f40f93488c7a276c19"
Oct 30 07:15:21.021874 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:21.021722    6224 scope.go:122] "RemoveContainer" containerID="497eedfb909f8edcd780f7137c9405874291f4c8b843dc1d2e7f53f27efefe0a"
Oct 30 07:15:21.022196 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:15:21.022133    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-f357db-29889075-r7jb7_odh-ai-gateway-infra(bc136a4a-c414-48a8-9759-39010f5d76e1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f357db-29889075-r7jb7" podUID="bc136a4a-c414-48a8-9759-39010f5d76e1"
Oct 30 07:15:21.023996 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:21.023971    6224 generic.go:356] "Generic (PLEG): container finished" podID="272f437e-01c1-4ce3-b2a6-3ec21c50aab1" containerID="d64429002e163a33c15ee1e99aefe0f4c074809b9edfed3e0417bc531f332af4" exitCode=60
Oct 30 07:15:21.024101 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:21.024053    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889075-jqz4k" event={"ID":"272f437e-01c1-4ce3-b2a6-3ec21c50aab1","Type":"ContainerDied","Data":"d64429002e163a33c15ee1e99aefe0f4c074809b9edfed3e0417bc531f332af4"}
Oct 30 07:15:21.024433 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:21.024415    6224 scope.go:122] "RemoveContainer" containerID="d64429002e163a33c15ee1e99aefe0f4c074809b9edfed3e0417bc531f332af4"
Oct 30 07:15:21.024765 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:15:21.024742    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889075-jqz4k_odh-ai-gateway-infra(272f437e-01c1-4ce3-b2a6-3ec21c50aab1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889075-jqz4k" podUID="272f437e-01c1-4ce3-b2a6-3ec21c50aab1"
Oct 30 07:15:21.039505 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:21.039484    6224 scope.go:122] "RemoveContainer" containerID="865809e270ec7271a5c5c5c688df0748d9a65633edd8163dc517b6e3533d64f4"
Oct 30 07:15:21.062374 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:21.062323    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f357db-29889075-r7jb7"]
Oct 30 07:15:21.109290 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:21.109259    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889075-jqz4k"]
Oct 30 07:15:21.240924 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:21.240878    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:15:22.317277 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:22.317250    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f357db-29889075-r7jb7"
Oct 30 07:15:22.443431 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:22.443399    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889075-jqz4k"
Oct 30 07:15:22.448388 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:22.448364    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc136a4a-c414-48a8-9759-39010f5d76e1-kube-api-access-n2l8w\" (UniqueName: \"kubernetes.io/projected/bc136a4a-c414-48a8-9759-39010f5d76e1-kube-api-access-n2l8w\") pod \"bc136a4a-c414-48a8-9759-39010f5d76e1\" (UID: \"bc136a4a-c414-48a8-9759-39010f5d76e1\") "
Oct 30 07:15:22.448534 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:22.448519    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc136a4a-c414-48a8-9759-39010f5d76e1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc136a4a-c414-48a8-9759-39010f5d76e1-maas-api-service-ca\") pod \"bc136a4a-c414-48a8-9759-39010f5d76e1\" (UID: \"bc136a4a-c414-48a8-9759-39010f5d76e1\") "
Oct 30 07:15:22.448967 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:22.448931    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc136a4a-c414-48a8-9759-39010f5d76e1-maas-api-service-ca" pod "bc136a4a-c414-48a8-9759-39010f5d76e1" (UID: "bc136a4a-c414-48a8-9759-39010f5d76e1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:15:22.449255 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:22.449233    6224 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc136a4a-c414-48a8-9759-39010f5d76e1-maas-api-service-ca\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:15:22.450676 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:22.450648    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc136a4a-c414-48a8-9759-39010f5d76e1-kube-api-access-n2l8w" pod "bc136a4a-c414-48a8-9759-39010f5d76e1" (UID: "bc136a4a-c414-48a8-9759-39010f5d76e1"). InnerVolumeSpecName "kube-api-access-n2l8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:15:22.496713 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:22.496680    6224 scope.go:122] "RemoveContainer" containerID="1143e4423565aba8dd8c1717fbc321835f5358c879e6b35ebebdcb0df7aacfe3"
Oct 30 07:15:22.542767 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:22.542708    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:15:22.542866 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:22.542783    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:15:22.550790 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:22.550765    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/272f437e-01c1-4ce3-b2a6-3ec21c50aab1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/272f437e-01c1-4ce3-b2a6-3ec21c50aab1-maas-api-service-ca\") pod \"272f437e-01c1-4ce3-b2a6-3ec21c50aab1\" (UID: \"272f437e-01c1-4ce3-b2a6-3ec21c50aab1\") "
Oct 30 07:15:22.550960 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:22.550943    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/272f437e-01c1-4ce3-b2a6-3ec21c50aab1-kube-api-access-9rm8z\" (UniqueName: \"kubernetes.io/projected/272f437e-01c1-4ce3-b2a6-3ec21c50aab1-kube-api-access-9rm8z\") pod \"272f437e-01c1-4ce3-b2a6-3ec21c50aab1\" (UID: \"272f437e-01c1-4ce3-b2a6-3ec21c50aab1\") "
Oct 30 07:15:22.551168 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:22.551144    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/272f437e-01c1-4ce3-b2a6-3ec21c50aab1-maas-api-service-ca" pod "272f437e-01c1-4ce3-b2a6-3ec21c50aab1" (UID: "272f437e-01c1-4ce3-b2a6-3ec21c50aab1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:15:22.551466 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:22.551443    6224 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/272f437e-01c1-4ce3-b2a6-3ec21c50aab1-maas-api-service-ca\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:15:22.551573 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:22.551471    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2l8w\" (UniqueName: \"kubernetes.io/projected/bc136a4a-c414-48a8-9759-39010f5d76e1-kube-api-access-n2l8w\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:15:22.553309 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:22.553276    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/272f437e-01c1-4ce3-b2a6-3ec21c50aab1-kube-api-access-9rm8z" pod "272f437e-01c1-4ce3-b2a6-3ec21c50aab1" (UID: "272f437e-01c1-4ce3-b2a6-3ec21c50aab1"). InnerVolumeSpecName "kube-api-access-9rm8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:15:22.652795 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:22.652758    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rm8z\" (UniqueName: \"kubernetes.io/projected/272f437e-01c1-4ce3-b2a6-3ec21c50aab1-kube-api-access-9rm8z\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:15:23.052615 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:23.052567    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dpzbb" event={"ID":"bf2e54e1-8437-47eb-bfa7-04bde103ebf8","Type":"ContainerStarted","Data":"725955b0bb733b6239ae04fcbed2ac3bce21ec0884d929e0ee2321c890d4eebb"}
Oct 30 07:15:23.052841 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:23.052721    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dpzbb"
Oct 30 07:15:23.056315 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:23.056286    6224 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-dpzbb container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 07:15:23.056471 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:23.056336    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dpzbb" podUID="bf2e54e1-8437-47eb-bfa7-04bde103ebf8" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 07:15:23.060849 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:23.058362    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889075-5zpzs" event={"ID":"32a34b72-254d-4660-8c61-edf07e947407","Type":"ContainerStarted","Data":"6e319271d297460db4192fc65c94c9191f65a7b217e81c580503fab636300189"}
Oct 30 07:15:23.062994 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:23.062967    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f357db-29889075-r7jb7"
Oct 30 07:15:23.063543 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:23.063518    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f357db-29889075-r7jb7" event={"ID":"bc136a4a-c414-48a8-9759-39010f5d76e1","Type":"ContainerDied","Data":"a0d97717e6e57f55709eaab83aa9ec06b9c3235ab33628dd301d1ad621f71cad"}
Oct 30 07:15:23.063721 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:23.063706    6224 scope.go:122] "RemoveContainer" containerID="497eedfb909f8edcd780f7137c9405874291f4c8b843dc1d2e7f53f27efefe0a"
Oct 30 07:15:23.069020 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:23.068996    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889075-jqz4k" event={"ID":"272f437e-01c1-4ce3-b2a6-3ec21c50aab1","Type":"ContainerDied","Data":"580d5652c9a602c8b63110af3dd9ca4579b39e70f43542c609a1712c6e1515f8"}
Oct 30 07:15:23.069165 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:23.069045    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889075-jqz4k"
Oct 30 07:15:23.094014 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:23.093958    6224 scope.go:122] "RemoveContainer" containerID="d64429002e163a33c15ee1e99aefe0f4c074809b9edfed3e0417bc531f332af4"
Oct 30 07:15:23.107721 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:23.105976    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dpzbb" podStartSLOduration=2.056126496 podStartE2EDuration="5.105956231s" totalImagesPullingTime="3.049829735s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:15:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:15:23.104257996 +0000 UTC m=+1081.713664968" watchObservedRunningTime="2026-10-30 07:15:23.105956231 +0000 UTC m=+1081.715363202"
Oct 30 07:15:23.148985 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:23.148930    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889075-5zpzs" podStartSLOduration=18.417379268 podStartE2EDuration="23.148911158s" totalImagesPullingTime="4.73153189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:15:23.147187905 +0000 UTC m=+1081.756594877" watchObservedRunningTime="2026-10-30 07:15:23.148911158 +0000 UTC m=+1081.758318129"
Oct 30 07:15:23.193611 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:23.193576    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889075-jqz4k"]
Oct 30 07:15:23.198858 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:23.198830    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889075-jqz4k"]
Oct 30 07:15:23.236854 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:23.236824    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f357db-29889075-r7jb7"]
Oct 30 07:15:23.241523 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:23.241500    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f357db-29889075-r7jb7"]
Oct 30 07:15:23.505021 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:23.504990    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="272f437e-01c1-4ce3-b2a6-3ec21c50aab1" path="/var/lib/kubelet/pods/272f437e-01c1-4ce3-b2a6-3ec21c50aab1/volumes"
Oct 30 07:15:23.505514 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:23.505477    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc136a4a-c414-48a8-9759-39010f5d76e1" path="/var/lib/kubelet/pods/bc136a4a-c414-48a8-9759-39010f5d76e1/volumes"
Oct 30 07:15:24.078157 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:24.078125    6224 generic.go:356] "Generic (PLEG): container finished" podID="32a34b72-254d-4660-8c61-edf07e947407" containerID="6e319271d297460db4192fc65c94c9191f65a7b217e81c580503fab636300189" exitCode=60
Oct 30 07:15:24.078365 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:24.078199    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889075-5zpzs" event={"ID":"32a34b72-254d-4660-8c61-edf07e947407","Type":"ContainerDied","Data":"6e319271d297460db4192fc65c94c9191f65a7b217e81c580503fab636300189"}
Oct 30 07:15:24.078365 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:24.078259    6224 scope.go:122] "RemoveContainer" containerID="1143e4423565aba8dd8c1717fbc321835f5358c879e6b35ebebdcb0df7aacfe3"
Oct 30 07:15:24.078673 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:24.078657    6224 scope.go:122] "RemoveContainer" containerID="6e319271d297460db4192fc65c94c9191f65a7b217e81c580503fab636300189"
Oct 30 07:15:24.079062 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:15:24.079038    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889075-5zpzs_odh-ai-gateway-infra(32a34b72-254d-4660-8c61-edf07e947407)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889075-5zpzs" podUID="32a34b72-254d-4660-8c61-edf07e947407"
Oct 30 07:15:24.081160 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:24.081139    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dpzbb"
Oct 30 07:15:24.160395 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:24.160363    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889075-5zpzs"]
Oct 30 07:15:25.248934 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:25.248910    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889075-5zpzs"
Oct 30 07:15:25.309364 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:25.309329    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32a34b72-254d-4660-8c61-edf07e947407-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32a34b72-254d-4660-8c61-edf07e947407-maas-api-service-ca\") pod \"32a34b72-254d-4660-8c61-edf07e947407\" (UID: \"32a34b72-254d-4660-8c61-edf07e947407\") "
Oct 30 07:15:25.309555 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:25.309384    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32a34b72-254d-4660-8c61-edf07e947407-kube-api-access-2zfdt\" (UniqueName: \"kubernetes.io/projected/32a34b72-254d-4660-8c61-edf07e947407-kube-api-access-2zfdt\") pod \"32a34b72-254d-4660-8c61-edf07e947407\" (UID: \"32a34b72-254d-4660-8c61-edf07e947407\") "
Oct 30 07:15:25.310188 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:25.310153    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32a34b72-254d-4660-8c61-edf07e947407-maas-api-service-ca" pod "32a34b72-254d-4660-8c61-edf07e947407" (UID: "32a34b72-254d-4660-8c61-edf07e947407"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:15:25.312373 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:25.312344    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32a34b72-254d-4660-8c61-edf07e947407-kube-api-access-2zfdt" pod "32a34b72-254d-4660-8c61-edf07e947407" (UID: "32a34b72-254d-4660-8c61-edf07e947407"). InnerVolumeSpecName "kube-api-access-2zfdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:15:25.411604 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:25.411526    6224 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32a34b72-254d-4660-8c61-edf07e947407-maas-api-service-ca\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:15:25.411604 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:25.411557    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zfdt\" (UniqueName: \"kubernetes.io/projected/32a34b72-254d-4660-8c61-edf07e947407-kube-api-access-2zfdt\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:15:26.096472 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:26.096441    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889075-5zpzs"
Oct 30 07:15:26.096649 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:26.096482    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889075-5zpzs" event={"ID":"32a34b72-254d-4660-8c61-edf07e947407","Type":"ContainerDied","Data":"870ad11af8b02b1022990a4d2f848345f6c952aa60e3b88bd8b6a0b4c54ffb79"}
Oct 30 07:15:26.096649 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:26.096546    6224 scope.go:122] "RemoveContainer" containerID="6e319271d297460db4192fc65c94c9191f65a7b217e81c580503fab636300189"
Oct 30 07:15:26.154257 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:26.154228    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889075-5zpzs"]
Oct 30 07:15:26.167270 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:26.167234    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889075-5zpzs"]
Oct 30 07:15:27.503945 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:27.503910    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32a34b72-254d-4660-8c61-edf07e947407" path="/var/lib/kubelet/pods/32a34b72-254d-4660-8c61-edf07e947407/volumes"
Oct 30 07:15:31.856431 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:15:31.856395    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:15:42.910446 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:42.910365    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7c7884-5d44bb7f-f9sdh"]
Oct 30 07:15:42.910891 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:42.910577    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c7884-5d44bb7f-f9sdh" podUID="7877e784-053a-44d6-b681-ba3bdcb4110b" containerName="maas-api" containerID="cri-o://3674ac7abf36baf1e29a4c0865759eb3096ac892d66119710ebd7a0c69ef9d4c" gracePeriod=30
Oct 30 07:15:43.306835 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:43.305992    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c7884-5d44bb7f-f9sdh"
Oct 30 07:15:43.313836 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:43.307475    6224 generic.go:356] "Generic (PLEG): container finished" podID="7877e784-053a-44d6-b681-ba3bdcb4110b" containerID="3674ac7abf36baf1e29a4c0865759eb3096ac892d66119710ebd7a0c69ef9d4c" exitCode=0
Oct 30 07:15:43.313836 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:43.307577    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c7884-5d44bb7f-f9sdh" event={"ID":"7877e784-053a-44d6-b681-ba3bdcb4110b","Type":"ContainerDied","Data":"3674ac7abf36baf1e29a4c0865759eb3096ac892d66119710ebd7a0c69ef9d4c"}
Oct 30 07:15:43.313836 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:43.307622    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c7884-5d44bb7f-f9sdh" event={"ID":"7877e784-053a-44d6-b681-ba3bdcb4110b","Type":"ContainerDied","Data":"9ccf87fedd97d76a21ec011cef044f7efe80cd30d945b1c5845614e93d172c67"}
Oct 30 07:15:43.313836 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:43.307652    6224 scope.go:122] "RemoveContainer" containerID="3674ac7abf36baf1e29a4c0865759eb3096ac892d66119710ebd7a0c69ef9d4c"
Oct 30 07:15:43.328554 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:43.327256    6224 scope.go:122] "RemoveContainer" containerID="3674ac7abf36baf1e29a4c0865759eb3096ac892d66119710ebd7a0c69ef9d4c"
Oct 30 07:15:43.328554 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:15:43.328437    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3674ac7abf36baf1e29a4c0865759eb3096ac892d66119710ebd7a0c69ef9d4c\": container with ID starting with 3674ac7abf36baf1e29a4c0865759eb3096ac892d66119710ebd7a0c69ef9d4c not found: ID does not exist" containerID="3674ac7abf36baf1e29a4c0865759eb3096ac892d66119710ebd7a0c69ef9d4c"
Oct 30 07:15:43.328554 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:43.328476    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3674ac7abf36baf1e29a4c0865759eb3096ac892d66119710ebd7a0c69ef9d4c"} err="failed to get container status \"3674ac7abf36baf1e29a4c0865759eb3096ac892d66119710ebd7a0c69ef9d4c\": rpc error: code = NotFound desc = could not find container \"3674ac7abf36baf1e29a4c0865759eb3096ac892d66119710ebd7a0c69ef9d4c\": container with ID starting with 3674ac7abf36baf1e29a4c0865759eb3096ac892d66119710ebd7a0c69ef9d4c not found: ID does not exist"
Oct 30 07:15:43.487368 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:43.487287    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7877e784-053a-44d6-b681-ba3bdcb4110b-kube-api-access-wq9zk\" (UniqueName: \"kubernetes.io/projected/7877e784-053a-44d6-b681-ba3bdcb4110b-kube-api-access-wq9zk\") pod \"7877e784-053a-44d6-b681-ba3bdcb4110b\" (UID: \"7877e784-053a-44d6-b681-ba3bdcb4110b\") "
Oct 30 07:15:43.487544 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:43.487430    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7877e784-053a-44d6-b681-ba3bdcb4110b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7877e784-053a-44d6-b681-ba3bdcb4110b-openshift-service-ca\") pod \"7877e784-053a-44d6-b681-ba3bdcb4110b\" (UID: \"7877e784-053a-44d6-b681-ba3bdcb4110b\") "
Oct 30 07:15:43.487544 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:43.487514    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7877e784-053a-44d6-b681-ba3bdcb4110b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7877e784-053a-44d6-b681-ba3bdcb4110b-maas-api-tls\") pod \"7877e784-053a-44d6-b681-ba3bdcb4110b\" (UID: \"7877e784-053a-44d6-b681-ba3bdcb4110b\") "
Oct 30 07:15:43.487688 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:43.487560    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7877e784-053a-44d6-b681-ba3bdcb4110b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7877e784-053a-44d6-b681-ba3bdcb4110b-metrics-certs\") pod \"7877e784-053a-44d6-b681-ba3bdcb4110b\" (UID: \"7877e784-053a-44d6-b681-ba3bdcb4110b\") "
Oct 30 07:15:43.487932 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:43.487888    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7877e784-053a-44d6-b681-ba3bdcb4110b-openshift-service-ca" pod "7877e784-053a-44d6-b681-ba3bdcb4110b" (UID: "7877e784-053a-44d6-b681-ba3bdcb4110b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:15:43.488269 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:43.488247    6224 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7877e784-053a-44d6-b681-ba3bdcb4110b-openshift-service-ca\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:15:43.490253 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:43.490226    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7877e784-053a-44d6-b681-ba3bdcb4110b-maas-api-tls" pod "7877e784-053a-44d6-b681-ba3bdcb4110b" (UID: "7877e784-053a-44d6-b681-ba3bdcb4110b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:15:43.490343 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:43.490288    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7877e784-053a-44d6-b681-ba3bdcb4110b-kube-api-access-wq9zk" pod "7877e784-053a-44d6-b681-ba3bdcb4110b" (UID: "7877e784-053a-44d6-b681-ba3bdcb4110b"). InnerVolumeSpecName "kube-api-access-wq9zk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:15:43.490343 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:43.490290    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7877e784-053a-44d6-b681-ba3bdcb4110b-metrics-certs" pod "7877e784-053a-44d6-b681-ba3bdcb4110b" (UID: "7877e784-053a-44d6-b681-ba3bdcb4110b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:15:43.590162 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:43.590116    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wq9zk\" (UniqueName: \"kubernetes.io/projected/7877e784-053a-44d6-b681-ba3bdcb4110b-kube-api-access-wq9zk\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:15:43.590162 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:43.590160    6224 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7877e784-053a-44d6-b681-ba3bdcb4110b-maas-api-tls\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:15:43.590389 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:43.590173    6224 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7877e784-053a-44d6-b681-ba3bdcb4110b-metrics-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:15:44.316131 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:44.316102    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c7884-5d44bb7f-f9sdh"
Oct 30 07:15:44.375152 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:44.375117    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7c7884-5d44bb7f-f9sdh"]
Oct 30 07:15:44.388160 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:44.388128    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7c7884-5d44bb7f-f9sdh"]
Oct 30 07:15:45.504306 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:45.504260    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7877e784-053a-44d6-b681-ba3bdcb4110b" path="/var/lib/kubelet/pods/7877e784-053a-44d6-b681-ba3bdcb4110b/volumes"
Oct 30 07:15:46.895478 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:46.895437    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:15:49.968455 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:49.968420    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a6a14e07-bdc5d9c5f-j5wkj"]
Oct 30 07:15:54.969774 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:54.969724    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-a6a14e07-bdc5d9c5f-j5wkj" podUID="485a6676-1591-4e3c-b856-5bf04f8aef3b" containerName="payload-pre-processing" containerID="cri-o://80d58d6265926564828b11afa18523729a2defd492a7b0de1927595aa3a4163d" gracePeriod=25
Oct 30 07:15:55.249907 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:55.249879    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a6a14e07-bdc5d9c5f-j5wkj"
Oct 30 07:15:55.395622 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:55.395587    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/485a6676-1591-4e3c-b856-5bf04f8aef3b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/485a6676-1591-4e3c-b856-5bf04f8aef3b-plugins-config-volume\") pod \"485a6676-1591-4e3c-b856-5bf04f8aef3b\" (UID: \"485a6676-1591-4e3c-b856-5bf04f8aef3b\") "
Oct 30 07:15:55.395928 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:55.395907    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/485a6676-1591-4e3c-b856-5bf04f8aef3b-kube-api-access-drm9s\" (UniqueName: \"kubernetes.io/projected/485a6676-1591-4e3c-b856-5bf04f8aef3b-kube-api-access-drm9s\") pod \"485a6676-1591-4e3c-b856-5bf04f8aef3b\" (UID: \"485a6676-1591-4e3c-b856-5bf04f8aef3b\") "
Oct 30 07:15:55.396047 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:55.396011    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/485a6676-1591-4e3c-b856-5bf04f8aef3b-plugins-config-volume" pod "485a6676-1591-4e3c-b856-5bf04f8aef3b" (UID: "485a6676-1591-4e3c-b856-5bf04f8aef3b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:15:55.396417 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:55.396397    6224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/485a6676-1591-4e3c-b856-5bf04f8aef3b-plugins-config-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:15:55.397534 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:55.397510    6224 generic.go:356] "Generic (PLEG): container finished" podID="485a6676-1591-4e3c-b856-5bf04f8aef3b" containerID="80d58d6265926564828b11afa18523729a2defd492a7b0de1927595aa3a4163d" exitCode=0
Oct 30 07:15:55.397649 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:55.397569    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a6a14e07-bdc5d9c5f-j5wkj"
Oct 30 07:15:55.397649 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:55.397590    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a6a14e07-bdc5d9c5f-j5wkj" event={"ID":"485a6676-1591-4e3c-b856-5bf04f8aef3b","Type":"ContainerDied","Data":"80d58d6265926564828b11afa18523729a2defd492a7b0de1927595aa3a4163d"}
Oct 30 07:15:55.397763 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:55.397654    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a6a14e07-bdc5d9c5f-j5wkj" event={"ID":"485a6676-1591-4e3c-b856-5bf04f8aef3b","Type":"ContainerDied","Data":"2165bf000ee353a701385ded9d507ad0d3deb050d549965c16f04d11810281a6"}
Oct 30 07:15:55.397763 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:55.397678    6224 scope.go:122] "RemoveContainer" containerID="80d58d6265926564828b11afa18523729a2defd492a7b0de1927595aa3a4163d"
Oct 30 07:15:55.398306 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:55.398269    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/485a6676-1591-4e3c-b856-5bf04f8aef3b-kube-api-access-drm9s" pod "485a6676-1591-4e3c-b856-5bf04f8aef3b" (UID: "485a6676-1591-4e3c-b856-5bf04f8aef3b"). InnerVolumeSpecName "kube-api-access-drm9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:15:55.414939 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:55.414916    6224 scope.go:122] "RemoveContainer" containerID="80d58d6265926564828b11afa18523729a2defd492a7b0de1927595aa3a4163d"
Oct 30 07:15:55.415219 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:15:55.415201    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80d58d6265926564828b11afa18523729a2defd492a7b0de1927595aa3a4163d\": container with ID starting with 80d58d6265926564828b11afa18523729a2defd492a7b0de1927595aa3a4163d not found: ID does not exist" containerID="80d58d6265926564828b11afa18523729a2defd492a7b0de1927595aa3a4163d"
Oct 30 07:15:55.415274 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:55.415228    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80d58d6265926564828b11afa18523729a2defd492a7b0de1927595aa3a4163d"} err="failed to get container status \"80d58d6265926564828b11afa18523729a2defd492a7b0de1927595aa3a4163d\": rpc error: code = NotFound desc = could not find container \"80d58d6265926564828b11afa18523729a2defd492a7b0de1927595aa3a4163d\": container with ID starting with 80d58d6265926564828b11afa18523729a2defd492a7b0de1927595aa3a4163d not found: ID does not exist"
Oct 30 07:15:55.497895 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:55.497797    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drm9s\" (UniqueName: \"kubernetes.io/projected/485a6676-1591-4e3c-b856-5bf04f8aef3b-kube-api-access-drm9s\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:15:55.751867 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:55.751770    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a6a14e07-bdc5d9c5f-j5wkj"]
Oct 30 07:15:55.759161 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:55.759126    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a6a14e07-bdc5d9c5f-j5wkj"]
Oct 30 07:15:57.504060 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:15:57.504022    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="485a6676-1591-4e3c-b856-5bf04f8aef3b" path="/var/lib/kubelet/pods/485a6676-1591-4e3c-b856-5bf04f8aef3b/volumes"
Oct 30 07:16:01.099869 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:01.099786    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c0e3de-59fcb65d7d-68h76"]
Oct 30 07:16:01.135920 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:01.135888    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c0e3de-c4f94dbf9-p82jw"]
Oct 30 07:16:01.970146 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:16:01.970080    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:16:06.100978 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:06.100932    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-c0e3de-59fcb65d7d-68h76" podUID="71cfd33e-bb00-4995-a800-628145442737" containerName="payload-processing" containerID="cri-o://7443c480d6e893fbae14777cbbc817add86ed8a73826b5726b88040956ec0064" gracePeriod=25
Oct 30 07:16:06.136910 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:06.136865    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c0e3de-c4f94dbf9-p82jw" podUID="2c34be83-969f-4fef-8954-c014d0d2e5cc" containerName="payload-pre-processing" containerID="cri-o://b87aac45e21c8e18b2e83d3d89931562053533e30ae1cceca9aa50cfa14975f3" gracePeriod=25
Oct 30 07:16:06.276115 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:16:06.276089    6224 logging.go:55] [core] [Channel #317 SubChannel #318]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.76:50052", ServerName: "10.134.0.76:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.76:50052: connect: connection refused"
Oct 30 07:16:06.432837 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:06.432796    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c0e3de-59fcb65d7d-68h76"
Oct 30 07:16:06.481856 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:06.481833    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c0e3de-c4f94dbf9-p82jw"
Oct 30 07:16:06.496474 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:06.496438    6224 generic.go:356] "Generic (PLEG): container finished" podID="2c34be83-969f-4fef-8954-c014d0d2e5cc" containerID="b87aac45e21c8e18b2e83d3d89931562053533e30ae1cceca9aa50cfa14975f3" exitCode=0
Oct 30 07:16:06.496619 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:06.496493    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c0e3de-c4f94dbf9-p82jw"
Oct 30 07:16:06.496619 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:06.496577    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c0e3de-c4f94dbf9-p82jw" event={"ID":"2c34be83-969f-4fef-8954-c014d0d2e5cc","Type":"ContainerDied","Data":"b87aac45e21c8e18b2e83d3d89931562053533e30ae1cceca9aa50cfa14975f3"}
Oct 30 07:16:06.496719 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:06.496623    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c0e3de-c4f94dbf9-p82jw" event={"ID":"2c34be83-969f-4fef-8954-c014d0d2e5cc","Type":"ContainerDied","Data":"50cb6f1dde98fc0c40631dd833ed9322ac77d939bce16ae4a8d65fc23b68e9cd"}
Oct 30 07:16:06.496719 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:06.496654    6224 scope.go:122] "RemoveContainer" containerID="b87aac45e21c8e18b2e83d3d89931562053533e30ae1cceca9aa50cfa14975f3"
Oct 30 07:16:06.498586 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:06.498562    6224 generic.go:356] "Generic (PLEG): container finished" podID="71cfd33e-bb00-4995-a800-628145442737" containerID="7443c480d6e893fbae14777cbbc817add86ed8a73826b5726b88040956ec0064" exitCode=0
Oct 30 07:16:06.498713 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:06.498611    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c0e3de-59fcb65d7d-68h76"
Oct 30 07:16:06.498783 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:06.498727    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c0e3de-59fcb65d7d-68h76" event={"ID":"71cfd33e-bb00-4995-a800-628145442737","Type":"ContainerDied","Data":"7443c480d6e893fbae14777cbbc817add86ed8a73826b5726b88040956ec0064"}
Oct 30 07:16:06.498783 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:06.498765    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c0e3de-59fcb65d7d-68h76" event={"ID":"71cfd33e-bb00-4995-a800-628145442737","Type":"ContainerDied","Data":"7d180e36921f8bd1aa786fc5fe408c0441e3785b1071b2681d504325ab82a109"}
Oct 30 07:16:06.507827 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:06.507796    6224 scope.go:122] "RemoveContainer" containerID="b87aac45e21c8e18b2e83d3d89931562053533e30ae1cceca9aa50cfa14975f3"
Oct 30 07:16:06.508100 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:16:06.508083    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b87aac45e21c8e18b2e83d3d89931562053533e30ae1cceca9aa50cfa14975f3\": container with ID starting with b87aac45e21c8e18b2e83d3d89931562053533e30ae1cceca9aa50cfa14975f3 not found: ID does not exist" containerID="b87aac45e21c8e18b2e83d3d89931562053533e30ae1cceca9aa50cfa14975f3"
Oct 30 07:16:06.508183 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:06.508112    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b87aac45e21c8e18b2e83d3d89931562053533e30ae1cceca9aa50cfa14975f3"} err="failed to get container status \"b87aac45e21c8e18b2e83d3d89931562053533e30ae1cceca9aa50cfa14975f3\": rpc error: code = NotFound desc = could not find container \"b87aac45e21c8e18b2e83d3d89931562053533e30ae1cceca9aa50cfa14975f3\": container with ID starting with b87aac45e21c8e18b2e83d3d89931562053533e30ae1cceca9aa50cfa14975f3 not found: ID does not exist"
Oct 30 07:16:06.508183 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:06.508135    6224 scope.go:122] "RemoveContainer" containerID="7443c480d6e893fbae14777cbbc817add86ed8a73826b5726b88040956ec0064"
Oct 30 07:16:06.518649 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:06.518597    6224 scope.go:122] "RemoveContainer" containerID="7443c480d6e893fbae14777cbbc817add86ed8a73826b5726b88040956ec0064"
Oct 30 07:16:06.518952 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:16:06.518926    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7443c480d6e893fbae14777cbbc817add86ed8a73826b5726b88040956ec0064\": container with ID starting with 7443c480d6e893fbae14777cbbc817add86ed8a73826b5726b88040956ec0064 not found: ID does not exist" containerID="7443c480d6e893fbae14777cbbc817add86ed8a73826b5726b88040956ec0064"
Oct 30 07:16:06.519128 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:06.518961    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7443c480d6e893fbae14777cbbc817add86ed8a73826b5726b88040956ec0064"} err="failed to get container status \"7443c480d6e893fbae14777cbbc817add86ed8a73826b5726b88040956ec0064\": rpc error: code = NotFound desc = could not find container \"7443c480d6e893fbae14777cbbc817add86ed8a73826b5726b88040956ec0064\": container with ID starting with 7443c480d6e893fbae14777cbbc817add86ed8a73826b5726b88040956ec0064 not found: ID does not exist"
Oct 30 07:16:06.546883 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:06.546857    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71cfd33e-bb00-4995-a800-628145442737-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71cfd33e-bb00-4995-a800-628145442737-plugins-config-volume\") pod \"71cfd33e-bb00-4995-a800-628145442737\" (UID: \"71cfd33e-bb00-4995-a800-628145442737\") "
Oct 30 07:16:06.547028 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:06.546960    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c34be83-969f-4fef-8954-c014d0d2e5cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c34be83-969f-4fef-8954-c014d0d2e5cc-plugins-config-volume\") pod \"2c34be83-969f-4fef-8954-c014d0d2e5cc\" (UID: \"2c34be83-969f-4fef-8954-c014d0d2e5cc\") "
Oct 30 07:16:06.547028 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:06.546997    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c34be83-969f-4fef-8954-c014d0d2e5cc-kube-api-access-svpqj\" (UniqueName: \"kubernetes.io/projected/2c34be83-969f-4fef-8954-c014d0d2e5cc-kube-api-access-svpqj\") pod \"2c34be83-969f-4fef-8954-c014d0d2e5cc\" (UID: \"2c34be83-969f-4fef-8954-c014d0d2e5cc\") "
Oct 30 07:16:06.547525 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:06.547188    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71cfd33e-bb00-4995-a800-628145442737-kube-api-access-hsq9z\" (UniqueName: \"kubernetes.io/projected/71cfd33e-bb00-4995-a800-628145442737-kube-api-access-hsq9z\") pod \"71cfd33e-bb00-4995-a800-628145442737\" (UID: \"71cfd33e-bb00-4995-a800-628145442737\") "
Oct 30 07:16:06.547525 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:06.547243    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71cfd33e-bb00-4995-a800-628145442737-plugins-config-volume" pod "71cfd33e-bb00-4995-a800-628145442737" (UID: "71cfd33e-bb00-4995-a800-628145442737"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:16:06.547525 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:06.547292    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c34be83-969f-4fef-8954-c014d0d2e5cc-plugins-config-volume" pod "2c34be83-969f-4fef-8954-c014d0d2e5cc" (UID: "2c34be83-969f-4fef-8954-c014d0d2e5cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:16:06.547839 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:06.547822    6224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71cfd33e-bb00-4995-a800-628145442737-plugins-config-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:16:06.547912 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:06.547845    6224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c34be83-969f-4fef-8954-c014d0d2e5cc-plugins-config-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:16:06.549965 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:06.549940    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c34be83-969f-4fef-8954-c014d0d2e5cc-kube-api-access-svpqj" pod "2c34be83-969f-4fef-8954-c014d0d2e5cc" (UID: "2c34be83-969f-4fef-8954-c014d0d2e5cc"). InnerVolumeSpecName "kube-api-access-svpqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:16:06.550047 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:06.549961    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71cfd33e-bb00-4995-a800-628145442737-kube-api-access-hsq9z" pod "71cfd33e-bb00-4995-a800-628145442737" (UID: "71cfd33e-bb00-4995-a800-628145442737"). InnerVolumeSpecName "kube-api-access-hsq9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:16:06.649294 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:06.649215    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsq9z\" (UniqueName: \"kubernetes.io/projected/71cfd33e-bb00-4995-a800-628145442737-kube-api-access-hsq9z\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:16:06.649294 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:06.649244    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svpqj\" (UniqueName: \"kubernetes.io/projected/2c34be83-969f-4fef-8954-c014d0d2e5cc-kube-api-access-svpqj\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:16:06.852775 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:06.852733    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c0e3de-c4f94dbf9-p82jw"]
Oct 30 07:16:06.856764 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:06.856739    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c0e3de-c4f94dbf9-p82jw"]
Oct 30 07:16:06.898949 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:06.898922    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c0e3de-59fcb65d7d-68h76"]
Oct 30 07:16:06.902234 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:06.902180    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c0e3de-59fcb65d7d-68h76"]
Oct 30 07:16:07.276434 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:07.276349    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c0e3de-c4f94dbf9-p82jw" podUID="2c34be83-969f-4fef-8954-c014d0d2e5cc" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.76:50052\" within 1s: context deadline exceeded"
Oct 30 07:16:07.511539 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:07.511504    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c34be83-969f-4fef-8954-c014d0d2e5cc" path="/var/lib/kubelet/pods/2c34be83-969f-4fef-8954-c014d0d2e5cc/volumes"
Oct 30 07:16:07.511985 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:07.511962    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71cfd33e-bb00-4995-a800-628145442737" path="/var/lib/kubelet/pods/71cfd33e-bb00-4995-a800-628145442737/volumes"
Oct 30 07:16:13.521320 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:13.521284    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a6a14e07-84d574ddfc-7whtt"]
Oct 30 07:16:13.550436 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:16:13.550399    6224 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-a6a14e07-serving-cert: secret "maas-api-e2e-mt-a6a14e07-serving-cert" not found
Oct 30 07:16:13.550598 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:16:13.550523    6224 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee-maas-api-tls podName:fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee nodeName:}" failed. No retries permitted until 2026-10-30 07:16:14.050488639 +0000 UTC m=+1132.659895591 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee-maas-api-tls") pod "maas-api-e2e-mt-a6a14e07-84d574ddfc-7whtt" (UID: "fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee") : secret "maas-api-e2e-mt-a6a14e07-serving-cert" not found
Oct 30 07:16:13.558721 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:13.558677    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6a14e07-84d574ddfc-7whtt" podUID="fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee" containerName="maas-api" containerID="cri-o://5ceac104f1b97c77de0c5d629ce7c019120202e7180b027da4ff166dce515f57" gracePeriod=30
Oct 30 07:16:13.972641 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:13.972597    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6a14e07-84d574ddfc-7whtt"
Oct 30 07:16:14.064953 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:14.064919    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee-openshift-service-ca\") pod \"fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee\" (UID: \"fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee\") "
Oct 30 07:16:14.065184 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:14.065168    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee-metrics-certs\") pod \"fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee\" (UID: \"fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee\") "
Oct 30 07:16:14.065230 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:14.065215    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee-kube-api-access-fdb47\" (UniqueName: \"kubernetes.io/projected/fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee-kube-api-access-fdb47\") pod \"fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee\" (UID: \"fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee\") "
Oct 30 07:16:14.065291 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:14.065274    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee-maas-api-tls\") pod \"fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee\" (UID: \"fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee\") "
Oct 30 07:16:14.066651 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:14.066612    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee-openshift-service-ca" pod "fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee" (UID: "fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:16:14.096983 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:14.096267    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee-maas-api-tls" pod "fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee" (UID: "fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:16:14.096983 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:14.096309    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee-kube-api-access-fdb47" pod "fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee" (UID: "fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee"). InnerVolumeSpecName "kube-api-access-fdb47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:16:14.096983 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:14.096422    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee-metrics-certs" pod "fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee" (UID: "fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:16:14.167284 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:14.167244    6224 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee-metrics-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:16:14.167284 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:14.167273    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdb47\" (UniqueName: \"kubernetes.io/projected/fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee-kube-api-access-fdb47\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:16:14.167284 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:14.167282    6224 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee-maas-api-tls\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:16:14.167284 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:14.167295    6224 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee-openshift-service-ca\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:16:14.566162 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:14.566126    6224 generic.go:356] "Generic (PLEG): container finished" podID="fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee" containerID="5ceac104f1b97c77de0c5d629ce7c019120202e7180b027da4ff166dce515f57" exitCode=0
Oct 30 07:16:14.566569 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:14.566171    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6a14e07-84d574ddfc-7whtt"
Oct 30 07:16:14.566569 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:14.566212    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6a14e07-84d574ddfc-7whtt" event={"ID":"fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee","Type":"ContainerDied","Data":"5ceac104f1b97c77de0c5d629ce7c019120202e7180b027da4ff166dce515f57"}
Oct 30 07:16:14.566569 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:14.566265    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6a14e07-84d574ddfc-7whtt" event={"ID":"fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee","Type":"ContainerDied","Data":"96e48e00565dcc1491d3d2f5d53f4ceb4c67639beea0934a23c701f49daf3e1c"}
Oct 30 07:16:14.566569 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:14.566285    6224 scope.go:122] "RemoveContainer" containerID="5ceac104f1b97c77de0c5d629ce7c019120202e7180b027da4ff166dce515f57"
Oct 30 07:16:14.578981 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:14.578962    6224 scope.go:122] "RemoveContainer" containerID="5ceac104f1b97c77de0c5d629ce7c019120202e7180b027da4ff166dce515f57"
Oct 30 07:16:14.579282 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:16:14.579256    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ceac104f1b97c77de0c5d629ce7c019120202e7180b027da4ff166dce515f57\": container with ID starting with 5ceac104f1b97c77de0c5d629ce7c019120202e7180b027da4ff166dce515f57 not found: ID does not exist" containerID="5ceac104f1b97c77de0c5d629ce7c019120202e7180b027da4ff166dce515f57"
Oct 30 07:16:14.579395 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:14.579297    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ceac104f1b97c77de0c5d629ce7c019120202e7180b027da4ff166dce515f57"} err="failed to get container status \"5ceac104f1b97c77de0c5d629ce7c019120202e7180b027da4ff166dce515f57\": rpc error: code = NotFound desc = could not find container \"5ceac104f1b97c77de0c5d629ce7c019120202e7180b027da4ff166dce515f57\": container with ID starting with 5ceac104f1b97c77de0c5d629ce7c019120202e7180b027da4ff166dce515f57 not found: ID does not exist"
Oct 30 07:16:14.629374 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:14.629348    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a6a14e07-84d574ddfc-7whtt"]
Oct 30 07:16:14.634617 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:14.634593    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a6a14e07-84d574ddfc-7whtt"]
Oct 30 07:16:15.512677 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:15.512639    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee" path="/var/lib/kubelet/pods/fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee/volumes"
Oct 30 07:16:16.833730 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:16.833683    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6a14e07-84d574ddfc-7whtt" podUID="fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.85:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:16:20.176612 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:20.176551    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-a6a14e07/e2e-lifecycle-model-a6a14e07-kserve-5fddf987df-g5t8r"]
Oct 30 07:16:26.389831 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:26.389702    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:16:31.858286 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:16:31.858244    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:16:35.225903 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:35.225860    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-a6a14e07/e2e-lifecycle-model-a6a14e07-kserve-5fddf987df-g5t8r" podUID="9e157c86-b1bc-4b22-9a7a-2c75044d65db" containerName="main" containerID="cri-o://07513f38466ed270a704b1ebf33e315ef8b8aaecb33cdfc91efe5129b7d0ab41" gracePeriod=45
Oct 30 07:16:35.394018 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:35.393979    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-a6a14e07/e2e-lifecycle-model-a6a14e07-kserve-5fddf987df-g5t8r" podUID="9e157c86-b1bc-4b22-9a7a-2c75044d65db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": dial tcp 10.134.0.71:8000: connect: connection refused"
Oct 30 07:16:35.496008 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:35.495939    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-a6a14e07/e2e-lifecycle-model-a6a14e07-kserve-5fddf987df-g5t8r"
Oct 30 07:16:35.619705 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:35.619678    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e157c86-b1bc-4b22-9a7a-2c75044d65db-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e157c86-b1bc-4b22-9a7a-2c75044d65db-model-cache\") pod \"9e157c86-b1bc-4b22-9a7a-2c75044d65db\" (UID: \"9e157c86-b1bc-4b22-9a7a-2c75044d65db\") "
Oct 30 07:16:35.619910 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:35.619723    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e157c86-b1bc-4b22-9a7a-2c75044d65db-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e157c86-b1bc-4b22-9a7a-2c75044d65db-tmp-dir\") pod \"9e157c86-b1bc-4b22-9a7a-2c75044d65db\" (UID: \"9e157c86-b1bc-4b22-9a7a-2c75044d65db\") "
Oct 30 07:16:35.619910 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:35.619750    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e157c86-b1bc-4b22-9a7a-2c75044d65db-home\" (UniqueName: \"kubernetes.io/empty-dir/9e157c86-b1bc-4b22-9a7a-2c75044d65db-home\") pod \"9e157c86-b1bc-4b22-9a7a-2c75044d65db\" (UID: \"9e157c86-b1bc-4b22-9a7a-2c75044d65db\") "
Oct 30 07:16:35.619910 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:35.619798    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e157c86-b1bc-4b22-9a7a-2c75044d65db-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e157c86-b1bc-4b22-9a7a-2c75044d65db-dshm\") pod \"9e157c86-b1bc-4b22-9a7a-2c75044d65db\" (UID: \"9e157c86-b1bc-4b22-9a7a-2c75044d65db\") "
Oct 30 07:16:35.620042 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:35.620006    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e157c86-b1bc-4b22-9a7a-2c75044d65db-model-cache" pod "9e157c86-b1bc-4b22-9a7a-2c75044d65db" (UID: "9e157c86-b1bc-4b22-9a7a-2c75044d65db"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:16:35.620084 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:35.620052    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e157c86-b1bc-4b22-9a7a-2c75044d65db-tls-certs\" (UniqueName: \"kubernetes.io/secret/9e157c86-b1bc-4b22-9a7a-2c75044d65db-tls-certs\") pod \"9e157c86-b1bc-4b22-9a7a-2c75044d65db\" (UID: \"9e157c86-b1bc-4b22-9a7a-2c75044d65db\") "
Oct 30 07:16:35.620084 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:35.620059    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e157c86-b1bc-4b22-9a7a-2c75044d65db-home" pod "9e157c86-b1bc-4b22-9a7a-2c75044d65db" (UID: "9e157c86-b1bc-4b22-9a7a-2c75044d65db"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:16:35.620084 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:35.620070    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e157c86-b1bc-4b22-9a7a-2c75044d65db-tmp-dir" pod "9e157c86-b1bc-4b22-9a7a-2c75044d65db" (UID: "9e157c86-b1bc-4b22-9a7a-2c75044d65db"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:16:35.620190 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:35.620172    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e157c86-b1bc-4b22-9a7a-2c75044d65db-kube-api-access-qf96p\" (UniqueName: \"kubernetes.io/projected/9e157c86-b1bc-4b22-9a7a-2c75044d65db-kube-api-access-qf96p\") pod \"9e157c86-b1bc-4b22-9a7a-2c75044d65db\" (UID: \"9e157c86-b1bc-4b22-9a7a-2c75044d65db\") "
Oct 30 07:16:35.620945 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:35.620918    6224 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e157c86-b1bc-4b22-9a7a-2c75044d65db-model-cache\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:16:35.620945 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:35.620946    6224 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e157c86-b1bc-4b22-9a7a-2c75044d65db-tmp-dir\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:16:35.621150 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:35.620959    6224 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e157c86-b1bc-4b22-9a7a-2c75044d65db-home\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:16:35.622615 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:35.622592    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e157c86-b1bc-4b22-9a7a-2c75044d65db-dshm" pod "9e157c86-b1bc-4b22-9a7a-2c75044d65db" (UID: "9e157c86-b1bc-4b22-9a7a-2c75044d65db"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:16:35.622724 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:35.622704    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e157c86-b1bc-4b22-9a7a-2c75044d65db-tls-certs" pod "9e157c86-b1bc-4b22-9a7a-2c75044d65db" (UID: "9e157c86-b1bc-4b22-9a7a-2c75044d65db"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:16:35.622959 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:35.622941    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e157c86-b1bc-4b22-9a7a-2c75044d65db-kube-api-access-qf96p" pod "9e157c86-b1bc-4b22-9a7a-2c75044d65db" (UID: "9e157c86-b1bc-4b22-9a7a-2c75044d65db"). InnerVolumeSpecName "kube-api-access-qf96p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:16:35.722233 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:35.722201    6224 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e157c86-b1bc-4b22-9a7a-2c75044d65db-tls-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:16:35.722233 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:35.722229    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qf96p\" (UniqueName: \"kubernetes.io/projected/9e157c86-b1bc-4b22-9a7a-2c75044d65db-kube-api-access-qf96p\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:16:35.722233 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:35.722239    6224 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e157c86-b1bc-4b22-9a7a-2c75044d65db-dshm\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:16:35.802009 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:35.801976    6224 generic.go:356] "Generic (PLEG): container finished" podID="9e157c86-b1bc-4b22-9a7a-2c75044d65db" containerID="07513f38466ed270a704b1ebf33e315ef8b8aaecb33cdfc91efe5129b7d0ab41" exitCode=0
Oct 30 07:16:35.802164 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:35.802019    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-a6a14e07/e2e-lifecycle-model-a6a14e07-kserve-5fddf987df-g5t8r"
Oct 30 07:16:35.802164 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:35.802053    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-a6a14e07/e2e-lifecycle-model-a6a14e07-kserve-5fddf987df-g5t8r" event={"ID":"9e157c86-b1bc-4b22-9a7a-2c75044d65db","Type":"ContainerDied","Data":"07513f38466ed270a704b1ebf33e315ef8b8aaecb33cdfc91efe5129b7d0ab41"}
Oct 30 07:16:35.802164 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:35.802101    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-a6a14e07/e2e-lifecycle-model-a6a14e07-kserve-5fddf987df-g5t8r" event={"ID":"9e157c86-b1bc-4b22-9a7a-2c75044d65db","Type":"ContainerDied","Data":"d5df3814f9d1f05da66107bca282ead2e4252b6e4a67997d5d493343697a0232"}
Oct 30 07:16:35.802164 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:35.802120    6224 scope.go:122] "RemoveContainer" containerID="07513f38466ed270a704b1ebf33e315ef8b8aaecb33cdfc91efe5129b7d0ab41"
Oct 30 07:16:35.812793 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:35.812776    6224 scope.go:122] "RemoveContainer" containerID="07513f38466ed270a704b1ebf33e315ef8b8aaecb33cdfc91efe5129b7d0ab41"
Oct 30 07:16:35.813106 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:16:35.813085    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07513f38466ed270a704b1ebf33e315ef8b8aaecb33cdfc91efe5129b7d0ab41\": container with ID starting with 07513f38466ed270a704b1ebf33e315ef8b8aaecb33cdfc91efe5129b7d0ab41 not found: ID does not exist" containerID="07513f38466ed270a704b1ebf33e315ef8b8aaecb33cdfc91efe5129b7d0ab41"
Oct 30 07:16:35.813175 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:35.813116    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07513f38466ed270a704b1ebf33e315ef8b8aaecb33cdfc91efe5129b7d0ab41"} err="failed to get container status \"07513f38466ed270a704b1ebf33e315ef8b8aaecb33cdfc91efe5129b7d0ab41\": rpc error: code = NotFound desc = could not find container \"07513f38466ed270a704b1ebf33e315ef8b8aaecb33cdfc91efe5129b7d0ab41\": container with ID starting with 07513f38466ed270a704b1ebf33e315ef8b8aaecb33cdfc91efe5129b7d0ab41 not found: ID does not exist"
Oct 30 07:16:35.864139 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:35.864112    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-a6a14e07/e2e-lifecycle-model-a6a14e07-kserve-5fddf987df-g5t8r"]
Oct 30 07:16:35.878102 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:35.878076    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-a6a14e07/e2e-lifecycle-model-a6a14e07-kserve-5fddf987df-g5t8r"]
Oct 30 07:16:37.504390 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:37.504359    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e157c86-b1bc-4b22-9a7a-2c75044d65db" path="/var/lib/kubelet/pods/9e157c86-b1bc-4b22-9a7a-2c75044d65db/volumes"
Oct 30 07:16:54.067103 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.067069    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb"]
Oct 30 07:16:54.067596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.067582    6224 cpu_manager.go:414] "containerMap: removing container" podUID="32a34b72-254d-4660-8c61-edf07e947407" containerName="cleanup"
Oct 30 07:16:54.067639 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.067597    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32a34b72-254d-4660-8c61-edf07e947407" containerName="cleanup"
Oct 30 07:16:54.067639 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.067608    6224 cpu_manager.go:414] "containerMap: removing container" podUID="fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee" containerName="maas-api"
Oct 30 07:16:54.067639 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.067613    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee" containerName="maas-api"
Oct 30 07:16:54.067639 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.067622    6224 cpu_manager.go:414] "containerMap: removing container" podUID="272f437e-01c1-4ce3-b2a6-3ec21c50aab1" containerName="cleanup"
Oct 30 07:16:54.067639 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.067628    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="272f437e-01c1-4ce3-b2a6-3ec21c50aab1" containerName="cleanup"
Oct 30 07:16:54.067792 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.067643    6224 cpu_manager.go:414] "containerMap: removing container" podUID="9e157c86-b1bc-4b22-9a7a-2c75044d65db" containerName="main"
Oct 30 07:16:54.067792 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.067648    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e157c86-b1bc-4b22-9a7a-2c75044d65db" containerName="main"
Oct 30 07:16:54.067792 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.067658    6224 cpu_manager.go:414] "containerMap: removing container" podUID="bc136a4a-c414-48a8-9759-39010f5d76e1" containerName="cleanup"
Oct 30 07:16:54.067792 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.067664    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc136a4a-c414-48a8-9759-39010f5d76e1" containerName="cleanup"
Oct 30 07:16:54.067792 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.067673    6224 cpu_manager.go:414] "containerMap: removing container" podUID="272f437e-01c1-4ce3-b2a6-3ec21c50aab1" containerName="cleanup"
Oct 30 07:16:54.067792 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.067677    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="272f437e-01c1-4ce3-b2a6-3ec21c50aab1" containerName="cleanup"
Oct 30 07:16:54.067792 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.067694    6224 cpu_manager.go:414] "containerMap: removing container" podUID="bc136a4a-c414-48a8-9759-39010f5d76e1" containerName="cleanup"
Oct 30 07:16:54.067792 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.067699    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc136a4a-c414-48a8-9759-39010f5d76e1" containerName="cleanup"
Oct 30 07:16:54.067792 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.067708    6224 cpu_manager.go:414] "containerMap: removing container" podUID="7877e784-053a-44d6-b681-ba3bdcb4110b" containerName="maas-api"
Oct 30 07:16:54.067792 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.067713    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7877e784-053a-44d6-b681-ba3bdcb4110b" containerName="maas-api"
Oct 30 07:16:54.067792 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.067722    6224 cpu_manager.go:414] "containerMap: removing container" podUID="485a6676-1591-4e3c-b856-5bf04f8aef3b" containerName="payload-pre-processing"
Oct 30 07:16:54.067792 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.067727    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="485a6676-1591-4e3c-b856-5bf04f8aef3b" containerName="payload-pre-processing"
Oct 30 07:16:54.067792 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.067739    6224 cpu_manager.go:414] "containerMap: removing container" podUID="32a34b72-254d-4660-8c61-edf07e947407" containerName="cleanup"
Oct 30 07:16:54.067792 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.067743    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32a34b72-254d-4660-8c61-edf07e947407" containerName="cleanup"
Oct 30 07:16:54.067792 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.067750    6224 cpu_manager.go:414] "containerMap: removing container" podUID="71cfd33e-bb00-4995-a800-628145442737" containerName="payload-processing"
Oct 30 07:16:54.067792 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.067754    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71cfd33e-bb00-4995-a800-628145442737" containerName="payload-processing"
Oct 30 07:16:54.067792 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.067763    6224 cpu_manager.go:414] "containerMap: removing container" podUID="2c34be83-969f-4fef-8954-c014d0d2e5cc" containerName="payload-pre-processing"
Oct 30 07:16:54.067792 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.067768    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c34be83-969f-4fef-8954-c014d0d2e5cc" containerName="payload-pre-processing"
Oct 30 07:16:54.068344 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.067865    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e157c86-b1bc-4b22-9a7a-2c75044d65db" containerName="main"
Oct 30 07:16:54.068344 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.067873    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc136a4a-c414-48a8-9759-39010f5d76e1" containerName="cleanup"
Oct 30 07:16:54.068344 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.067879    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="272f437e-01c1-4ce3-b2a6-3ec21c50aab1" containerName="cleanup"
Oct 30 07:16:54.068344 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.067889    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc136a4a-c414-48a8-9759-39010f5d76e1" containerName="cleanup"
Oct 30 07:16:54.068344 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.067895    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="7877e784-053a-44d6-b681-ba3bdcb4110b" containerName="maas-api"
Oct 30 07:16:54.068344 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.067901    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="32a34b72-254d-4660-8c61-edf07e947407" containerName="cleanup"
Oct 30 07:16:54.068344 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.067906    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="485a6676-1591-4e3c-b856-5bf04f8aef3b" containerName="payload-pre-processing"
Oct 30 07:16:54.068344 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.067912    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc136a4a-c414-48a8-9759-39010f5d76e1" containerName="cleanup"
Oct 30 07:16:54.068344 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.067918    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="32a34b72-254d-4660-8c61-edf07e947407" containerName="cleanup"
Oct 30 07:16:54.068344 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.067922    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="71cfd33e-bb00-4995-a800-628145442737" containerName="payload-processing"
Oct 30 07:16:54.068344 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.067927    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c34be83-969f-4fef-8954-c014d0d2e5cc" containerName="payload-pre-processing"
Oct 30 07:16:54.068344 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.067933    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="32a34b72-254d-4660-8c61-edf07e947407" containerName="cleanup"
Oct 30 07:16:54.068344 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.067938    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd5b0ba9-f329-49c2-80a2-f1ea3f5a09ee" containerName="maas-api"
Oct 30 07:16:54.068344 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.067943    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="272f437e-01c1-4ce3-b2a6-3ec21c50aab1" containerName="cleanup"
Oct 30 07:16:54.071593 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.071574    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb"
Oct 30 07:16:54.109075 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.109035    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb"]
Oct 30 07:16:54.172390 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.172360    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-ba67d8-openshift-default-dockercfg-r2cwg\""
Oct 30 07:16:54.195004 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.194970    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eda91969-699f-446c-96ab-d49d54c97d26-istio-envoy\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb\" (UID: \"eda91969-699f-446c-96ab-d49d54c97d26\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb"
Oct 30 07:16:54.195153 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.195010    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eda91969-699f-446c-96ab-d49d54c97d26-istio-data\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb\" (UID: \"eda91969-699f-446c-96ab-d49d54c97d26\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb"
Oct 30 07:16:54.195153 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.195030    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrrg9\" (UniqueName: \"kubernetes.io/projected/eda91969-699f-446c-96ab-d49d54c97d26-kube-api-access-xrrg9\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb\" (UID: \"eda91969-699f-446c-96ab-d49d54c97d26\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb"
Oct 30 07:16:54.195153 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.195115    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eda91969-699f-446c-96ab-d49d54c97d26-istio-podinfo\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb\" (UID: \"eda91969-699f-446c-96ab-d49d54c97d26\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb"
Oct 30 07:16:54.195153 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.195138    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eda91969-699f-446c-96ab-d49d54c97d26-credential-socket\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb\" (UID: \"eda91969-699f-446c-96ab-d49d54c97d26\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb"
Oct 30 07:16:54.195307 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.195178    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eda91969-699f-446c-96ab-d49d54c97d26-istiod-ca-cert\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb\" (UID: \"eda91969-699f-446c-96ab-d49d54c97d26\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb"
Oct 30 07:16:54.195307 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.195270    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eda91969-699f-446c-96ab-d49d54c97d26-workload-socket\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb\" (UID: \"eda91969-699f-446c-96ab-d49d54c97d26\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb"
Oct 30 07:16:54.195390 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.195341    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eda91969-699f-446c-96ab-d49d54c97d26-workload-certs\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb\" (UID: \"eda91969-699f-446c-96ab-d49d54c97d26\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb"
Oct 30 07:16:54.195426 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.195406    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eda91969-699f-446c-96ab-d49d54c97d26-istio-token\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb\" (UID: \"eda91969-699f-446c-96ab-d49d54c97d26\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb"
Oct 30 07:16:54.297018 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.296982    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eda91969-699f-446c-96ab-d49d54c97d26-workload-certs\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb\" (UID: \"eda91969-699f-446c-96ab-d49d54c97d26\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb"
Oct 30 07:16:54.297018 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.297025    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eda91969-699f-446c-96ab-d49d54c97d26-istio-token\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb\" (UID: \"eda91969-699f-446c-96ab-d49d54c97d26\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb"
Oct 30 07:16:54.297267 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.297116    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eda91969-699f-446c-96ab-d49d54c97d26-istio-envoy\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb\" (UID: \"eda91969-699f-446c-96ab-d49d54c97d26\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb"
Oct 30 07:16:54.297267 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.297138    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eda91969-699f-446c-96ab-d49d54c97d26-istio-data\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb\" (UID: \"eda91969-699f-446c-96ab-d49d54c97d26\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb"
Oct 30 07:16:54.297267 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.297161    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrrg9\" (UniqueName: \"kubernetes.io/projected/eda91969-699f-446c-96ab-d49d54c97d26-kube-api-access-xrrg9\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb\" (UID: \"eda91969-699f-446c-96ab-d49d54c97d26\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb"
Oct 30 07:16:54.297267 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.297200    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eda91969-699f-446c-96ab-d49d54c97d26-istio-podinfo\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb\" (UID: \"eda91969-699f-446c-96ab-d49d54c97d26\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb"
Oct 30 07:16:54.297267 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.297249    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eda91969-699f-446c-96ab-d49d54c97d26-credential-socket\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb\" (UID: \"eda91969-699f-446c-96ab-d49d54c97d26\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb"
Oct 30 07:16:54.297535 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.297353    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eda91969-699f-446c-96ab-d49d54c97d26-istiod-ca-cert\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb\" (UID: \"eda91969-699f-446c-96ab-d49d54c97d26\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb"
Oct 30 07:16:54.297535 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.297394    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eda91969-699f-446c-96ab-d49d54c97d26-workload-socket\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb\" (UID: \"eda91969-699f-446c-96ab-d49d54c97d26\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb"
Oct 30 07:16:54.297535 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.297438    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eda91969-699f-446c-96ab-d49d54c97d26-workload-certs\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb\" (UID: \"eda91969-699f-446c-96ab-d49d54c97d26\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb"
Oct 30 07:16:54.297535 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.297529    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eda91969-699f-446c-96ab-d49d54c97d26-istio-data\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb\" (UID: \"eda91969-699f-446c-96ab-d49d54c97d26\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb"
Oct 30 07:16:54.297731 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.297610    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eda91969-699f-446c-96ab-d49d54c97d26-credential-socket\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb\" (UID: \"eda91969-699f-446c-96ab-d49d54c97d26\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb"
Oct 30 07:16:54.297769 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.297734    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eda91969-699f-446c-96ab-d49d54c97d26-workload-socket\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb\" (UID: \"eda91969-699f-446c-96ab-d49d54c97d26\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb"
Oct 30 07:16:54.298054 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.298035    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eda91969-699f-446c-96ab-d49d54c97d26-istiod-ca-cert\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb\" (UID: \"eda91969-699f-446c-96ab-d49d54c97d26\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb"
Oct 30 07:16:54.299648 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.299627    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eda91969-699f-446c-96ab-d49d54c97d26-istio-envoy\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb\" (UID: \"eda91969-699f-446c-96ab-d49d54c97d26\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb"
Oct 30 07:16:54.299778 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.299754    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eda91969-699f-446c-96ab-d49d54c97d26-istio-podinfo\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb\" (UID: \"eda91969-699f-446c-96ab-d49d54c97d26\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb"
Oct 30 07:16:54.322236 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.322176    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eda91969-699f-446c-96ab-d49d54c97d26-istio-token\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb\" (UID: \"eda91969-699f-446c-96ab-d49d54c97d26\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb"
Oct 30 07:16:54.322416 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.322394    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrrg9\" (UniqueName: \"kubernetes.io/projected/eda91969-699f-446c-96ab-d49d54c97d26-kube-api-access-xrrg9\") pod \"e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb\" (UID: \"eda91969-699f-446c-96ab-d49d54c97d26\") " pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb"
Oct 30 07:16:54.386776 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.386742    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb"
Oct 30 07:16:54.710066 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:16:54.710021    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeda91969_699f_446c_96ab_d49d54c97d26.slice/crio-26bafe63d1d5d0456c5bdc130bcc9eb0ac6eb855384787de75cfbb075a96b162 WatchSource:0}: Error finding container 26bafe63d1d5d0456c5bdc130bcc9eb0ac6eb855384787de75cfbb075a96b162: Status 404 returned error can't find the container with id 26bafe63d1d5d0456c5bdc130bcc9eb0ac6eb855384787de75cfbb075a96b162
Oct 30 07:16:54.711072 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.711042    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb"]
Oct 30 07:16:54.712871 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.712796    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:16:54.712986 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.712924    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:16:54.712986 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.712970    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:16:54.942719 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.942680    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb" event={"ID":"eda91969-699f-446c-96ab-d49d54c97d26","Type":"ContainerStarted","Data":"357563b84dc3c267e9cd7df0ffe65ad8306af5b49d2ea0c9f1079b8be99422ab"}
Oct 30 07:16:54.942957 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.942772    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb" event={"ID":"eda91969-699f-446c-96ab-d49d54c97d26","Type":"ContainerStarted","Data":"26bafe63d1d5d0456c5bdc130bcc9eb0ac6eb855384787de75cfbb075a96b162"}
Oct 30 07:16:54.990952 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:54.990832    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb" podStartSLOduration=0.990801493 podStartE2EDuration="990.801493ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:16:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:16:54.990346471 +0000 UTC m=+1173.599753442" watchObservedRunningTime="2026-10-30 07:16:54.990801493 +0000 UTC m=+1173.600208464"
Oct 30 07:16:55.387361 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:55.387325    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb"
Oct 30 07:16:55.389052 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:55.389024    6224 patch_prober.go:29] interesting pod/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 30 07:16:55.389181 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:55.389087    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb" podUID="eda91969-699f-446c-96ab-d49d54c97d26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 30 07:16:56.390400 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:56.388223    6224 patch_prober.go:29] interesting pod/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 30 07:16:56.390400 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:56.388289    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb" podUID="eda91969-699f-446c-96ab-d49d54c97d26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 30 07:16:57.387834 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:57.387149    6224 patch_prober.go:29] interesting pod/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 30 07:16:57.387834 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:57.387214    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb" podUID="eda91969-699f-446c-96ab-d49d54c97d26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 30 07:16:58.388281 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:58.388241    6224 patch_prober.go:29] interesting pod/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 30 07:16:58.388785 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:58.388305    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb" podUID="eda91969-699f-446c-96ab-d49d54c97d26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 30 07:16:59.387275 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:59.387233    6224 patch_prober.go:29] interesting pod/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 30 07:16:59.387540 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:59.387509    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb" podUID="eda91969-699f-446c-96ab-d49d54c97d26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 30 07:16:59.786925 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:59.784865    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4e42c8-64d8f55c96-6wxtx"]
Oct 30 07:16:59.804877 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:16:59.804446    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4e42c8-658945d447-wkthc"]
Oct 30 07:17:00.388363 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:00.388324    6224 patch_prober.go:29] interesting pod/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 30 07:17:00.388575 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:00.388498    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb" podUID="eda91969-699f-446c-96ab-d49d54c97d26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 30 07:17:01.390011 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:01.389973    6224 patch_prober.go:29] interesting pod/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 30 07:17:01.390658 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:01.390620    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb" podUID="eda91969-699f-446c-96ab-d49d54c97d26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 30 07:17:01.857760 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:17:01.857721    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:17:02.391866 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:02.391831    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb"
Oct 30 07:17:02.392256 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:02.392187    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb"
Oct 30 07:17:02.392835 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:02.392796    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb"
Oct 30 07:17:02.515674 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:02.515624    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:17:04.786060 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:04.785995    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-4e42c8-64d8f55c96-6wxtx" podUID="8beda270-8cb9-4211-889b-1e8a42cf3891" containerName="payload-processing" containerID="cri-o://e470451d24cb0d55996889804f23dcc2ace9bb8b7b5bd2c58760657d9057d00b" gracePeriod=25
Oct 30 07:17:04.805384 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:04.805180    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4e42c8-658945d447-wkthc" podUID="dcc1ddda-8a2c-40cb-b3fa-82b7e931e7b7" containerName="payload-pre-processing" containerID="cri-o://6f7d1dc929970778cc5bb8f83314729f79ca623523a914db44a8e688bd19efd4" gracePeriod=25
Oct 30 07:17:05.060197 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:05.060168    6224 generic.go:356] "Generic (PLEG): container finished" podID="8beda270-8cb9-4211-889b-1e8a42cf3891" containerID="e470451d24cb0d55996889804f23dcc2ace9bb8b7b5bd2c58760657d9057d00b" exitCode=0
Oct 30 07:17:05.060321 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:05.060252    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4e42c8-64d8f55c96-6wxtx" event={"ID":"8beda270-8cb9-4211-889b-1e8a42cf3891","Type":"ContainerDied","Data":"e470451d24cb0d55996889804f23dcc2ace9bb8b7b5bd2c58760657d9057d00b"}
Oct 30 07:17:05.060321 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:05.060288    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4e42c8-64d8f55c96-6wxtx" event={"ID":"8beda270-8cb9-4211-889b-1e8a42cf3891","Type":"ContainerDied","Data":"9684516c85b76b144b1c0912ba5007f6532f0f75a2c5a34c08ee1c1689c47912"}
Oct 30 07:17:05.060321 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:05.060311    6224 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9684516c85b76b144b1c0912ba5007f6532f0f75a2c5a34c08ee1c1689c47912"
Oct 30 07:17:05.063777 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:05.063750    6224 generic.go:356] "Generic (PLEG): container finished" podID="dcc1ddda-8a2c-40cb-b3fa-82b7e931e7b7" containerID="6f7d1dc929970778cc5bb8f83314729f79ca623523a914db44a8e688bd19efd4" exitCode=0
Oct 30 07:17:05.063913 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:05.063851    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4e42c8-658945d447-wkthc" event={"ID":"dcc1ddda-8a2c-40cb-b3fa-82b7e931e7b7","Type":"ContainerDied","Data":"6f7d1dc929970778cc5bb8f83314729f79ca623523a914db44a8e688bd19efd4"}
Oct 30 07:17:05.078942 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:05.078915    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4e42c8-64d8f55c96-6wxtx"
Oct 30 07:17:05.104514 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:05.104485    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8beda270-8cb9-4211-889b-1e8a42cf3891-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8beda270-8cb9-4211-889b-1e8a42cf3891-plugins-config-volume\") pod \"8beda270-8cb9-4211-889b-1e8a42cf3891\" (UID: \"8beda270-8cb9-4211-889b-1e8a42cf3891\") "
Oct 30 07:17:05.104744 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:05.104720    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8beda270-8cb9-4211-889b-1e8a42cf3891-kube-api-access-wqnnv\" (UniqueName: \"kubernetes.io/projected/8beda270-8cb9-4211-889b-1e8a42cf3891-kube-api-access-wqnnv\") pod \"8beda270-8cb9-4211-889b-1e8a42cf3891\" (UID: \"8beda270-8cb9-4211-889b-1e8a42cf3891\") "
Oct 30 07:17:05.104912 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:05.104892    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8beda270-8cb9-4211-889b-1e8a42cf3891-plugins-config-volume" pod "8beda270-8cb9-4211-889b-1e8a42cf3891" (UID: "8beda270-8cb9-4211-889b-1e8a42cf3891"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:17:05.105210 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:05.105193    6224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8beda270-8cb9-4211-889b-1e8a42cf3891-plugins-config-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:17:05.106754 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:05.106728    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8beda270-8cb9-4211-889b-1e8a42cf3891-kube-api-access-wqnnv" pod "8beda270-8cb9-4211-889b-1e8a42cf3891" (UID: "8beda270-8cb9-4211-889b-1e8a42cf3891"). InnerVolumeSpecName "kube-api-access-wqnnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:17:05.130401 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:05.130381    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4e42c8-658945d447-wkthc"
Oct 30 07:17:05.207146 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:05.205871    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcc1ddda-8a2c-40cb-b3fa-82b7e931e7b7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcc1ddda-8a2c-40cb-b3fa-82b7e931e7b7-plugins-config-volume\") pod \"dcc1ddda-8a2c-40cb-b3fa-82b7e931e7b7\" (UID: \"dcc1ddda-8a2c-40cb-b3fa-82b7e931e7b7\") "
Oct 30 07:17:05.207146 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:05.206113    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcc1ddda-8a2c-40cb-b3fa-82b7e931e7b7-kube-api-access-tk5zg\" (UniqueName: \"kubernetes.io/projected/dcc1ddda-8a2c-40cb-b3fa-82b7e931e7b7-kube-api-access-tk5zg\") pod \"dcc1ddda-8a2c-40cb-b3fa-82b7e931e7b7\" (UID: \"dcc1ddda-8a2c-40cb-b3fa-82b7e931e7b7\") "
Oct 30 07:17:05.207146 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:05.206248    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcc1ddda-8a2c-40cb-b3fa-82b7e931e7b7-plugins-config-volume" pod "dcc1ddda-8a2c-40cb-b3fa-82b7e931e7b7" (UID: "dcc1ddda-8a2c-40cb-b3fa-82b7e931e7b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:17:05.207146 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:05.206616    6224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcc1ddda-8a2c-40cb-b3fa-82b7e931e7b7-plugins-config-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:17:05.207146 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:05.206639    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqnnv\" (UniqueName: \"kubernetes.io/projected/8beda270-8cb9-4211-889b-1e8a42cf3891-kube-api-access-wqnnv\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:17:05.208977 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:05.208951    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcc1ddda-8a2c-40cb-b3fa-82b7e931e7b7-kube-api-access-tk5zg" pod "dcc1ddda-8a2c-40cb-b3fa-82b7e931e7b7" (UID: "dcc1ddda-8a2c-40cb-b3fa-82b7e931e7b7"). InnerVolumeSpecName "kube-api-access-tk5zg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:17:05.308109 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:05.308031    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tk5zg\" (UniqueName: \"kubernetes.io/projected/dcc1ddda-8a2c-40cb-b3fa-82b7e931e7b7-kube-api-access-tk5zg\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:17:06.071285 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:06.071213    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4e42c8-658945d447-wkthc"
Oct 30 07:17:06.071285 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:06.071254    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4e42c8-658945d447-wkthc" event={"ID":"dcc1ddda-8a2c-40cb-b3fa-82b7e931e7b7","Type":"ContainerDied","Data":"646f90edf58d2300b6149046e346451e1e30e6231d30636be625d232be9c4118"}
Oct 30 07:17:06.071747 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:06.071313    6224 scope.go:122] "RemoveContainer" containerID="6f7d1dc929970778cc5bb8f83314729f79ca623523a914db44a8e688bd19efd4"
Oct 30 07:17:06.071747 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:06.071435    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4e42c8-64d8f55c96-6wxtx"
Oct 30 07:17:06.128520 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:06.128486    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4e42c8-64d8f55c96-6wxtx"]
Oct 30 07:17:06.142040 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:06.142014    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4e42c8-64d8f55c96-6wxtx"]
Oct 30 07:17:06.167980 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:06.167947    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4e42c8-658945d447-wkthc"]
Oct 30 07:17:06.175416 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:06.175387    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4e42c8-658945d447-wkthc"]
Oct 30 07:17:07.503889 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:07.503855    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8beda270-8cb9-4211-889b-1e8a42cf3891" path="/var/lib/kubelet/pods/8beda270-8cb9-4211-889b-1e8a42cf3891/volumes"
Oct 30 07:17:07.504291 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:07.504282    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcc1ddda-8a2c-40cb-b3fa-82b7e931e7b7" path="/var/lib/kubelet/pods/dcc1ddda-8a2c-40cb-b3fa-82b7e931e7b7/volumes"
Oct 30 07:17:31.867528 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:17:31.867492    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:17:33.080133 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:33.080086    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c1f203-6bb4f7d48c-wvz6l"]
Oct 30 07:17:33.080653 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:33.080639    6224 cpu_manager.go:414] "containerMap: removing container" podUID="272f437e-01c1-4ce3-b2a6-3ec21c50aab1" containerName="cleanup"
Oct 30 07:17:33.080697 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:33.080654    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="272f437e-01c1-4ce3-b2a6-3ec21c50aab1" containerName="cleanup"
Oct 30 07:17:33.080697 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:33.080676    6224 cpu_manager.go:414] "containerMap: removing container" podUID="bc136a4a-c414-48a8-9759-39010f5d76e1" containerName="cleanup"
Oct 30 07:17:33.080697 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:33.080681    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc136a4a-c414-48a8-9759-39010f5d76e1" containerName="cleanup"
Oct 30 07:17:33.080697 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:33.080692    6224 cpu_manager.go:414] "containerMap: removing container" podUID="dcc1ddda-8a2c-40cb-b3fa-82b7e931e7b7" containerName="payload-pre-processing"
Oct 30 07:17:33.080843 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:33.080698    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcc1ddda-8a2c-40cb-b3fa-82b7e931e7b7" containerName="payload-pre-processing"
Oct 30 07:17:33.080843 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:33.080706    6224 cpu_manager.go:414] "containerMap: removing container" podUID="8beda270-8cb9-4211-889b-1e8a42cf3891" containerName="payload-processing"
Oct 30 07:17:33.080843 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:33.080711    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8beda270-8cb9-4211-889b-1e8a42cf3891" containerName="payload-processing"
Oct 30 07:17:33.080843 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:33.080727    6224 cpu_manager.go:414] "containerMap: removing container" podUID="32a34b72-254d-4660-8c61-edf07e947407" containerName="cleanup"
Oct 30 07:17:33.080843 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:33.080731    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32a34b72-254d-4660-8c61-edf07e947407" containerName="cleanup"
Oct 30 07:17:33.080843 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:33.080825    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcc1ddda-8a2c-40cb-b3fa-82b7e931e7b7" containerName="payload-pre-processing"
Oct 30 07:17:33.080843 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:33.080832    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="8beda270-8cb9-4211-889b-1e8a42cf3891" containerName="payload-processing"
Oct 30 07:17:33.080843 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:33.080844    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="272f437e-01c1-4ce3-b2a6-3ec21c50aab1" containerName="cleanup"
Oct 30 07:17:33.085823 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:33.085794    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1f203-6bb4f7d48c-wvz6l"
Oct 30 07:17:33.105445 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:33.105415    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c1f203-6bb4f7d48c-wvz6l"]
Oct 30 07:17:33.115388 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:33.115359    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-c1f203\""
Oct 30 07:17:33.115518 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:33.115406    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-c1f203-dockercfg-kgjbb\""
Oct 30 07:17:33.229907 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:33.229866    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c1f203-6ffbd89664-zzf48"]
Oct 30 07:17:33.236247 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:33.235410    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c1f203-6ffbd89664-zzf48"]
Oct 30 07:17:33.236247 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:33.235539    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c1f203-6ffbd89664-zzf48"
Oct 30 07:17:33.258117 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:33.258066    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76w4s\" (UniqueName: \"kubernetes.io/projected/bd9eda1c-1c26-4617-8f10-886e51c2b3e9-kube-api-access-76w4s\") pod \"payload-pre-processing-e2e-ipp-b-c1f203-6bb4f7d48c-wvz6l\" (UID: \"bd9eda1c-1c26-4617-8f10-886e51c2b3e9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1f203-6bb4f7d48c-wvz6l"
Oct 30 07:17:33.258448 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:33.258425    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd9eda1c-1c26-4617-8f10-886e51c2b3e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c1f203-6bb4f7d48c-wvz6l\" (UID: \"bd9eda1c-1c26-4617-8f10-886e51c2b3e9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1f203-6bb4f7d48c-wvz6l"
Oct 30 07:17:33.360508 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:33.360414    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a583c68f-99d3-4e7f-ab2c-5b7343cf44b8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c1f203-6ffbd89664-zzf48\" (UID: \"a583c68f-99d3-4e7f-ab2c-5b7343cf44b8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c1f203-6ffbd89664-zzf48"
Oct 30 07:17:33.360682 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:33.360663    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76w4s\" (UniqueName: \"kubernetes.io/projected/bd9eda1c-1c26-4617-8f10-886e51c2b3e9-kube-api-access-76w4s\") pod \"payload-pre-processing-e2e-ipp-b-c1f203-6bb4f7d48c-wvz6l\" (UID: \"bd9eda1c-1c26-4617-8f10-886e51c2b3e9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1f203-6bb4f7d48c-wvz6l"
Oct 30 07:17:33.360749 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:33.360714    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktfm7\" (UniqueName: \"kubernetes.io/projected/a583c68f-99d3-4e7f-ab2c-5b7343cf44b8-kube-api-access-ktfm7\") pod \"payload-processing-e2e-ipp-b-c1f203-6ffbd89664-zzf48\" (UID: \"a583c68f-99d3-4e7f-ab2c-5b7343cf44b8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c1f203-6ffbd89664-zzf48"
Oct 30 07:17:33.360853 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:33.360796    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd9eda1c-1c26-4617-8f10-886e51c2b3e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c1f203-6bb4f7d48c-wvz6l\" (UID: \"bd9eda1c-1c26-4617-8f10-886e51c2b3e9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1f203-6bb4f7d48c-wvz6l"
Oct 30 07:17:33.361461 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:33.361440    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd9eda1c-1c26-4617-8f10-886e51c2b3e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c1f203-6bb4f7d48c-wvz6l\" (UID: \"bd9eda1c-1c26-4617-8f10-886e51c2b3e9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1f203-6bb4f7d48c-wvz6l"
Oct 30 07:17:33.385777 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:33.385747    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76w4s\" (UniqueName: \"kubernetes.io/projected/bd9eda1c-1c26-4617-8f10-886e51c2b3e9-kube-api-access-76w4s\") pod \"payload-pre-processing-e2e-ipp-b-c1f203-6bb4f7d48c-wvz6l\" (UID: \"bd9eda1c-1c26-4617-8f10-886e51c2b3e9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1f203-6bb4f7d48c-wvz6l"
Oct 30 07:17:33.399636 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:33.399606    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1f203-6bb4f7d48c-wvz6l"
Oct 30 07:17:33.463302 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:33.462139    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktfm7\" (UniqueName: \"kubernetes.io/projected/a583c68f-99d3-4e7f-ab2c-5b7343cf44b8-kube-api-access-ktfm7\") pod \"payload-processing-e2e-ipp-b-c1f203-6ffbd89664-zzf48\" (UID: \"a583c68f-99d3-4e7f-ab2c-5b7343cf44b8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c1f203-6ffbd89664-zzf48"
Oct 30 07:17:33.463302 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:33.462713    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a583c68f-99d3-4e7f-ab2c-5b7343cf44b8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c1f203-6ffbd89664-zzf48\" (UID: \"a583c68f-99d3-4e7f-ab2c-5b7343cf44b8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c1f203-6ffbd89664-zzf48"
Oct 30 07:17:33.463843 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:33.463750    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a583c68f-99d3-4e7f-ab2c-5b7343cf44b8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c1f203-6ffbd89664-zzf48\" (UID: \"a583c68f-99d3-4e7f-ab2c-5b7343cf44b8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c1f203-6ffbd89664-zzf48"
Oct 30 07:17:33.500333 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:33.488565    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktfm7\" (UniqueName: \"kubernetes.io/projected/a583c68f-99d3-4e7f-ab2c-5b7343cf44b8-kube-api-access-ktfm7\") pod \"payload-processing-e2e-ipp-b-c1f203-6ffbd89664-zzf48\" (UID: \"a583c68f-99d3-4e7f-ab2c-5b7343cf44b8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c1f203-6ffbd89664-zzf48"
Oct 30 07:17:33.560463 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:33.560429    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c1f203-6ffbd89664-zzf48"
Oct 30 07:17:33.713328 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:17:33.713299    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd9eda1c_1c26_4617_8f10_886e51c2b3e9.slice/crio-72a67ae4634093f3b24178de9b208d54ce5e59c31f2b8feda2d27936882c8f9c WatchSource:0}: Error finding container 72a67ae4634093f3b24178de9b208d54ce5e59c31f2b8feda2d27936882c8f9c: Status 404 returned error can't find the container with id 72a67ae4634093f3b24178de9b208d54ce5e59c31f2b8feda2d27936882c8f9c
Oct 30 07:17:33.714589 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:33.714566    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c1f203-6bb4f7d48c-wvz6l"]
Oct 30 07:17:33.861071 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:17:33.861035    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda583c68f_99d3_4e7f_ab2c_5b7343cf44b8.slice/crio-bed3a905b53ec2ec22f38867e804f421fd73793498bfdc97f418a8e8c170242c WatchSource:0}: Error finding container bed3a905b53ec2ec22f38867e804f421fd73793498bfdc97f418a8e8c170242c: Status 404 returned error can't find the container with id bed3a905b53ec2ec22f38867e804f421fd73793498bfdc97f418a8e8c170242c
Oct 30 07:17:33.862095 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:33.862070    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c1f203-6ffbd89664-zzf48"]
Oct 30 07:17:34.476599 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:34.476548    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c1f203-6ffbd89664-zzf48" event={"ID":"a583c68f-99d3-4e7f-ab2c-5b7343cf44b8","Type":"ContainerStarted","Data":"9900d4e863cb1d16da975c6e64612b51357d3ef40e44f29a001934ba08d434f4"}
Oct 30 07:17:34.477117 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:34.476619    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c1f203-6ffbd89664-zzf48" event={"ID":"a583c68f-99d3-4e7f-ab2c-5b7343cf44b8","Type":"ContainerStarted","Data":"bed3a905b53ec2ec22f38867e804f421fd73793498bfdc97f418a8e8c170242c"}
Oct 30 07:17:34.477117 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:34.476650    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c1f203-6ffbd89664-zzf48"
Oct 30 07:17:34.478284 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:34.478256    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1f203-6bb4f7d48c-wvz6l" event={"ID":"bd9eda1c-1c26-4617-8f10-886e51c2b3e9","Type":"ContainerStarted","Data":"0a3f2006387cd3160b70b15a028f042f0028425f6d0941c4ff5af5cb35a9e1d8"}
Oct 30 07:17:34.478403 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:34.478305    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1f203-6bb4f7d48c-wvz6l" event={"ID":"bd9eda1c-1c26-4617-8f10-886e51c2b3e9","Type":"ContainerStarted","Data":"72a67ae4634093f3b24178de9b208d54ce5e59c31f2b8feda2d27936882c8f9c"}
Oct 30 07:17:34.478403 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:34.478345    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1f203-6bb4f7d48c-wvz6l"
Oct 30 07:17:34.520878 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:34.520795    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-c1f203-6ffbd89664-zzf48" podStartSLOduration=1.520781222 podStartE2EDuration="1.520781222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:17:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:17:34.520422732 +0000 UTC m=+1213.129829703" watchObservedRunningTime="2026-10-30 07:17:34.520781222 +0000 UTC m=+1213.130188192"
Oct 30 07:17:34.566472 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:34.566399    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1f203-6bb4f7d48c-wvz6l" podStartSLOduration=1.56638067 podStartE2EDuration="1.56638067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:17:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:17:34.564689343 +0000 UTC m=+1213.174096313" watchObservedRunningTime="2026-10-30 07:17:34.56638067 +0000 UTC m=+1213.175787643"
Oct 30 07:17:40.634909 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:40.634869    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj"]
Oct 30 07:17:40.635380 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:40.635129    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj" podUID="12729552-3e12-4b64-8779-a1584734c16e" containerName="istio-proxy" containerID="cri-o://2f914474e5c2adc796e469d46b532e410e51805ff14b3566ef7c8bc8ea2adce0" gracePeriod=30
Oct 30 07:17:41.027393 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:41.027298    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:17:45.489294 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:45.489261    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1f203-6bb4f7d48c-wvz6l"
Oct 30 07:17:45.489700 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:45.489337    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c1f203-6ffbd89664-zzf48"
Oct 30 07:17:45.791230 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:45.791197    6224 generic.go:356] "Generic (PLEG): container finished" podID="12729552-3e12-4b64-8779-a1584734c16e" containerID="2f914474e5c2adc796e469d46b532e410e51805ff14b3566ef7c8bc8ea2adce0" exitCode=0
Oct 30 07:17:45.791386 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:45.791275    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj" event={"ID":"12729552-3e12-4b64-8779-a1584734c16e","Type":"ContainerDied","Data":"2f914474e5c2adc796e469d46b532e410e51805ff14b3566ef7c8bc8ea2adce0"}
Oct 30 07:17:45.903359 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:45.903337    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj"
Oct 30 07:17:45.973138 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:45.973102    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12729552-3e12-4b64-8779-a1584734c16e-istio-token\" (UniqueName: \"kubernetes.io/projected/12729552-3e12-4b64-8779-a1584734c16e-istio-token\") pod \"12729552-3e12-4b64-8779-a1584734c16e\" (UID: \"12729552-3e12-4b64-8779-a1584734c16e\") "
Oct 30 07:17:45.973287 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:45.973155    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12729552-3e12-4b64-8779-a1584734c16e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12729552-3e12-4b64-8779-a1584734c16e-credential-socket\") pod \"12729552-3e12-4b64-8779-a1584734c16e\" (UID: \"12729552-3e12-4b64-8779-a1584734c16e\") "
Oct 30 07:17:45.973287 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:45.973191    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12729552-3e12-4b64-8779-a1584734c16e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12729552-3e12-4b64-8779-a1584734c16e-istiod-ca-cert\") pod \"12729552-3e12-4b64-8779-a1584734c16e\" (UID: \"12729552-3e12-4b64-8779-a1584734c16e\") "
Oct 30 07:17:45.973287 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:45.973279    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12729552-3e12-4b64-8779-a1584734c16e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12729552-3e12-4b64-8779-a1584734c16e-workload-certs\") pod \"12729552-3e12-4b64-8779-a1584734c16e\" (UID: \"12729552-3e12-4b64-8779-a1584734c16e\") "
Oct 30 07:17:45.973392 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:45.973358    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12729552-3e12-4b64-8779-a1584734c16e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12729552-3e12-4b64-8779-a1584734c16e-workload-socket\") pod \"12729552-3e12-4b64-8779-a1584734c16e\" (UID: \"12729552-3e12-4b64-8779-a1584734c16e\") "
Oct 30 07:17:45.973436 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:45.973421    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12729552-3e12-4b64-8779-a1584734c16e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12729552-3e12-4b64-8779-a1584734c16e-istio-data\") pod \"12729552-3e12-4b64-8779-a1584734c16e\" (UID: \"12729552-3e12-4b64-8779-a1584734c16e\") "
Oct 30 07:17:45.973501 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:45.973487    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/12729552-3e12-4b64-8779-a1584734c16e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12729552-3e12-4b64-8779-a1584734c16e-istio-podinfo\") pod \"12729552-3e12-4b64-8779-a1584734c16e\" (UID: \"12729552-3e12-4b64-8779-a1584734c16e\") "
Oct 30 07:17:45.973589 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:45.973574    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12729552-3e12-4b64-8779-a1584734c16e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12729552-3e12-4b64-8779-a1584734c16e-istio-envoy\") pod \"12729552-3e12-4b64-8779-a1584734c16e\" (UID: \"12729552-3e12-4b64-8779-a1584734c16e\") "
Oct 30 07:17:45.973681 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:45.973665    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12729552-3e12-4b64-8779-a1584734c16e-kube-api-access-5dhbr\" (UniqueName: \"kubernetes.io/projected/12729552-3e12-4b64-8779-a1584734c16e-kube-api-access-5dhbr\") pod \"12729552-3e12-4b64-8779-a1584734c16e\" (UID: \"12729552-3e12-4b64-8779-a1584734c16e\") "
Oct 30 07:17:45.975177 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:45.975145    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12729552-3e12-4b64-8779-a1584734c16e-workload-socket" pod "12729552-3e12-4b64-8779-a1584734c16e" (UID: "12729552-3e12-4b64-8779-a1584734c16e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:17:45.975376 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:45.975353    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12729552-3e12-4b64-8779-a1584734c16e-workload-certs" pod "12729552-3e12-4b64-8779-a1584734c16e" (UID: "12729552-3e12-4b64-8779-a1584734c16e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:17:45.975464 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:45.975412    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12729552-3e12-4b64-8779-a1584734c16e-istiod-ca-cert" pod "12729552-3e12-4b64-8779-a1584734c16e" (UID: "12729552-3e12-4b64-8779-a1584734c16e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:17:45.975850 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:45.975685    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12729552-3e12-4b64-8779-a1584734c16e-credential-socket" pod "12729552-3e12-4b64-8779-a1584734c16e" (UID: "12729552-3e12-4b64-8779-a1584734c16e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:17:45.976011 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:45.975979    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12729552-3e12-4b64-8779-a1584734c16e-istio-data" pod "12729552-3e12-4b64-8779-a1584734c16e" (UID: "12729552-3e12-4b64-8779-a1584734c16e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:17:45.978221 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:45.978195    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12729552-3e12-4b64-8779-a1584734c16e-kube-api-access-5dhbr" pod "12729552-3e12-4b64-8779-a1584734c16e" (UID: "12729552-3e12-4b64-8779-a1584734c16e"). InnerVolumeSpecName "kube-api-access-5dhbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:17:45.978924 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:45.978901    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12729552-3e12-4b64-8779-a1584734c16e-istio-envoy" pod "12729552-3e12-4b64-8779-a1584734c16e" (UID: "12729552-3e12-4b64-8779-a1584734c16e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:17:45.978924 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:45.978917    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12729552-3e12-4b64-8779-a1584734c16e-istio-token" pod "12729552-3e12-4b64-8779-a1584734c16e" (UID: "12729552-3e12-4b64-8779-a1584734c16e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:17:45.980167 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:45.980151    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/12729552-3e12-4b64-8779-a1584734c16e-istio-podinfo" pod "12729552-3e12-4b64-8779-a1584734c16e" (UID: "12729552-3e12-4b64-8779-a1584734c16e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:17:46.076081 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:46.076046    6224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12729552-3e12-4b64-8779-a1584734c16e-workload-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:17:46.076081 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:46.076084    6224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12729552-3e12-4b64-8779-a1584734c16e-istio-data\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:17:46.076316 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:46.076099    6224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12729552-3e12-4b64-8779-a1584734c16e-istio-podinfo\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:17:46.076316 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:46.076111    6224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12729552-3e12-4b64-8779-a1584734c16e-istio-envoy\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:17:46.076316 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:46.076123    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dhbr\" (UniqueName: \"kubernetes.io/projected/12729552-3e12-4b64-8779-a1584734c16e-kube-api-access-5dhbr\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:17:46.076316 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:46.076136    6224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12729552-3e12-4b64-8779-a1584734c16e-istio-token\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:17:46.076316 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:46.076149    6224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12729552-3e12-4b64-8779-a1584734c16e-credential-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:17:46.076316 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:46.076162    6224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12729552-3e12-4b64-8779-a1584734c16e-istiod-ca-cert\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:17:46.076316 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:46.076175    6224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12729552-3e12-4b64-8779-a1584734c16e-workload-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:17:46.801153 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:46.801124    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj"
Oct 30 07:17:46.801634 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:46.801157    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj" event={"ID":"12729552-3e12-4b64-8779-a1584734c16e","Type":"ContainerDied","Data":"f1407b4cf0a8f71b251e66d4888bb178c4c57851385169730ba3f3d86068c855"}
Oct 30 07:17:46.801634 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:46.801228    6224 scope.go:122] "RemoveContainer" containerID="2f914474e5c2adc796e469d46b532e410e51805ff14b3566ef7c8bc8ea2adce0"
Oct 30 07:17:46.875822 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:46.875782    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj"]
Oct 30 07:17:46.888352 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:46.888328    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-ssfvj"]
Oct 30 07:17:47.504745 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:17:47.504704    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12729552-3e12-4b64-8779-a1584734c16e" path="/var/lib/kubelet/pods/12729552-3e12-4b64-8779-a1584734c16e/volumes"
Oct 30 07:18:01.858528 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:18:01.858491    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:18:11.375865 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:18:11.372402    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:18:27.274620 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:18:27.274586    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c1f203-6ffbd89664-zzf48"]
Oct 30 07:18:27.311315 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:18:27.311278    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c1f203-6bb4f7d48c-wvz6l"]
Oct 30 07:18:31.857867 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:18:31.857797    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:18:32.275712 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:18:32.275613    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-c1f203-6ffbd89664-zzf48" podUID="a583c68f-99d3-4e7f-ab2c-5b7343cf44b8" containerName="payload-processing" containerID="cri-o://9900d4e863cb1d16da975c6e64612b51357d3ef40e44f29a001934ba08d434f4" gracePeriod=25
Oct 30 07:18:32.312081 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:18:32.312049    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1f203-6bb4f7d48c-wvz6l" podUID="bd9eda1c-1c26-4617-8f10-886e51c2b3e9" containerName="payload-pre-processing" containerID="cri-o://0a3f2006387cd3160b70b15a028f042f0028425f6d0941c4ff5af5cb35a9e1d8" gracePeriod=25
Oct 30 07:18:32.570680 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:18:32.570650    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c1f203-6ffbd89664-zzf48"
Oct 30 07:18:32.633587 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:18:32.633559    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1f203-6bb4f7d48c-wvz6l"
Oct 30 07:18:32.660799 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:18:32.660774    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a583c68f-99d3-4e7f-ab2c-5b7343cf44b8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a583c68f-99d3-4e7f-ab2c-5b7343cf44b8-plugins-config-volume\") pod \"a583c68f-99d3-4e7f-ab2c-5b7343cf44b8\" (UID: \"a583c68f-99d3-4e7f-ab2c-5b7343cf44b8\") "
Oct 30 07:18:32.660948 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:18:32.660926    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a583c68f-99d3-4e7f-ab2c-5b7343cf44b8-kube-api-access-ktfm7\" (UniqueName: \"kubernetes.io/projected/a583c68f-99d3-4e7f-ab2c-5b7343cf44b8-kube-api-access-ktfm7\") pod \"a583c68f-99d3-4e7f-ab2c-5b7343cf44b8\" (UID: \"a583c68f-99d3-4e7f-ab2c-5b7343cf44b8\") "
Oct 30 07:18:32.661241 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:18:32.661212    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a583c68f-99d3-4e7f-ab2c-5b7343cf44b8-plugins-config-volume" pod "a583c68f-99d3-4e7f-ab2c-5b7343cf44b8" (UID: "a583c68f-99d3-4e7f-ab2c-5b7343cf44b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:18:32.661711 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:18:32.661674    6224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a583c68f-99d3-4e7f-ab2c-5b7343cf44b8-plugins-config-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:18:32.663087 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:18:32.663068    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a583c68f-99d3-4e7f-ab2c-5b7343cf44b8-kube-api-access-ktfm7" pod "a583c68f-99d3-4e7f-ab2c-5b7343cf44b8" (UID: "a583c68f-99d3-4e7f-ab2c-5b7343cf44b8"). InnerVolumeSpecName "kube-api-access-ktfm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:18:32.762771 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:18:32.762696    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd9eda1c-1c26-4617-8f10-886e51c2b3e9-kube-api-access-76w4s\" (UniqueName: \"kubernetes.io/projected/bd9eda1c-1c26-4617-8f10-886e51c2b3e9-kube-api-access-76w4s\") pod \"bd9eda1c-1c26-4617-8f10-886e51c2b3e9\" (UID: \"bd9eda1c-1c26-4617-8f10-886e51c2b3e9\") "
Oct 30 07:18:32.762959 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:18:32.762825    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd9eda1c-1c26-4617-8f10-886e51c2b3e9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd9eda1c-1c26-4617-8f10-886e51c2b3e9-plugins-config-volume\") pod \"bd9eda1c-1c26-4617-8f10-886e51c2b3e9\" (UID: \"bd9eda1c-1c26-4617-8f10-886e51c2b3e9\") "
Oct 30 07:18:32.763193 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:18:32.763170    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd9eda1c-1c26-4617-8f10-886e51c2b3e9-plugins-config-volume" pod "bd9eda1c-1c26-4617-8f10-886e51c2b3e9" (UID: "bd9eda1c-1c26-4617-8f10-886e51c2b3e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:18:32.763365 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:18:32.763350    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktfm7\" (UniqueName: \"kubernetes.io/projected/a583c68f-99d3-4e7f-ab2c-5b7343cf44b8-kube-api-access-ktfm7\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:18:32.763409 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:18:32.763371    6224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd9eda1c-1c26-4617-8f10-886e51c2b3e9-plugins-config-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:18:32.764686 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:18:32.764666    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd9eda1c-1c26-4617-8f10-886e51c2b3e9-kube-api-access-76w4s" pod "bd9eda1c-1c26-4617-8f10-886e51c2b3e9" (UID: "bd9eda1c-1c26-4617-8f10-886e51c2b3e9"). InnerVolumeSpecName "kube-api-access-76w4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:18:32.869255 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:18:32.868940    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76w4s\" (UniqueName: \"kubernetes.io/projected/bd9eda1c-1c26-4617-8f10-886e51c2b3e9-kube-api-access-76w4s\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:18:33.304086 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:18:33.304046    6224 generic.go:356] "Generic (PLEG): container finished" podID="bd9eda1c-1c26-4617-8f10-886e51c2b3e9" containerID="0a3f2006387cd3160b70b15a028f042f0028425f6d0941c4ff5af5cb35a9e1d8" exitCode=0
Oct 30 07:18:33.304231 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:18:33.304106    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1f203-6bb4f7d48c-wvz6l"
Oct 30 07:18:33.304231 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:18:33.304100    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1f203-6bb4f7d48c-wvz6l" event={"ID":"bd9eda1c-1c26-4617-8f10-886e51c2b3e9","Type":"ContainerDied","Data":"0a3f2006387cd3160b70b15a028f042f0028425f6d0941c4ff5af5cb35a9e1d8"}
Oct 30 07:18:33.304352 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:18:33.304249    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1f203-6bb4f7d48c-wvz6l" event={"ID":"bd9eda1c-1c26-4617-8f10-886e51c2b3e9","Type":"ContainerDied","Data":"72a67ae4634093f3b24178de9b208d54ce5e59c31f2b8feda2d27936882c8f9c"}
Oct 30 07:18:33.304352 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:18:33.304284    6224 scope.go:122] "RemoveContainer" containerID="0a3f2006387cd3160b70b15a028f042f0028425f6d0941c4ff5af5cb35a9e1d8"
Oct 30 07:18:33.315144 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:18:33.314065    6224 generic.go:356] "Generic (PLEG): container finished" podID="a583c68f-99d3-4e7f-ab2c-5b7343cf44b8" containerID="9900d4e863cb1d16da975c6e64612b51357d3ef40e44f29a001934ba08d434f4" exitCode=0
Oct 30 07:18:33.315144 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:18:33.314137    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c1f203-6ffbd89664-zzf48" event={"ID":"a583c68f-99d3-4e7f-ab2c-5b7343cf44b8","Type":"ContainerDied","Data":"9900d4e863cb1d16da975c6e64612b51357d3ef40e44f29a001934ba08d434f4"}
Oct 30 07:18:33.315144 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:18:33.314181    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c1f203-6ffbd89664-zzf48" event={"ID":"a583c68f-99d3-4e7f-ab2c-5b7343cf44b8","Type":"ContainerDied","Data":"bed3a905b53ec2ec22f38867e804f421fd73793498bfdc97f418a8e8c170242c"}
Oct 30 07:18:33.315144 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:18:33.314251    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c1f203-6ffbd89664-zzf48"
Oct 30 07:18:33.324800 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:18:33.316965    6224 scope.go:122] "RemoveContainer" containerID="0a3f2006387cd3160b70b15a028f042f0028425f6d0941c4ff5af5cb35a9e1d8"
Oct 30 07:18:33.324800 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:18:33.324115    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a3f2006387cd3160b70b15a028f042f0028425f6d0941c4ff5af5cb35a9e1d8\": container with ID starting with 0a3f2006387cd3160b70b15a028f042f0028425f6d0941c4ff5af5cb35a9e1d8 not found: ID does not exist" containerID="0a3f2006387cd3160b70b15a028f042f0028425f6d0941c4ff5af5cb35a9e1d8"
Oct 30 07:18:33.324800 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:18:33.324149    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a3f2006387cd3160b70b15a028f042f0028425f6d0941c4ff5af5cb35a9e1d8"} err="failed to get container status \"0a3f2006387cd3160b70b15a028f042f0028425f6d0941c4ff5af5cb35a9e1d8\": rpc error: code = NotFound desc = could not find container \"0a3f2006387cd3160b70b15a028f042f0028425f6d0941c4ff5af5cb35a9e1d8\": container with ID starting with 0a3f2006387cd3160b70b15a028f042f0028425f6d0941c4ff5af5cb35a9e1d8 not found: ID does not exist"
Oct 30 07:18:33.324800 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:18:33.324171    6224 scope.go:122] "RemoveContainer" containerID="9900d4e863cb1d16da975c6e64612b51357d3ef40e44f29a001934ba08d434f4"
Oct 30 07:18:33.334849 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:18:33.334795    6224 scope.go:122] "RemoveContainer" containerID="9900d4e863cb1d16da975c6e64612b51357d3ef40e44f29a001934ba08d434f4"
Oct 30 07:18:33.335112 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:18:33.335089    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9900d4e863cb1d16da975c6e64612b51357d3ef40e44f29a001934ba08d434f4\": container with ID starting with 9900d4e863cb1d16da975c6e64612b51357d3ef40e44f29a001934ba08d434f4 not found: ID does not exist" containerID="9900d4e863cb1d16da975c6e64612b51357d3ef40e44f29a001934ba08d434f4"
Oct 30 07:18:33.335202 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:18:33.335116    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9900d4e863cb1d16da975c6e64612b51357d3ef40e44f29a001934ba08d434f4"} err="failed to get container status \"9900d4e863cb1d16da975c6e64612b51357d3ef40e44f29a001934ba08d434f4\": rpc error: code = NotFound desc = could not find container \"9900d4e863cb1d16da975c6e64612b51357d3ef40e44f29a001934ba08d434f4\": container with ID starting with 9900d4e863cb1d16da975c6e64612b51357d3ef40e44f29a001934ba08d434f4 not found: ID does not exist"
Oct 30 07:18:33.381601 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:18:33.381575    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c1f203-6bb4f7d48c-wvz6l"]
Oct 30 07:18:33.392946 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:18:33.392922    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c1f203-6bb4f7d48c-wvz6l"]
Oct 30 07:18:33.420634 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:18:33.420609    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c1f203-6ffbd89664-zzf48"]
Oct 30 07:18:33.424604 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:18:33.424579    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c1f203-6ffbd89664-zzf48"]
Oct 30 07:18:33.504871 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:18:33.504841    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a583c68f-99d3-4e7f-ab2c-5b7343cf44b8" path="/var/lib/kubelet/pods/a583c68f-99d3-4e7f-ab2c-5b7343cf44b8/volumes"
Oct 30 07:18:33.505312 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:18:33.505296    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd9eda1c-1c26-4617-8f10-886e51c2b3e9" path="/var/lib/kubelet/pods/bd9eda1c-1c26-4617-8f10-886e51c2b3e9/volumes"
Oct 30 07:18:55.267829 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:18:55.262711    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:19:01.857045 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:19:01.857002    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:19:31.857184 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:19:31.857146    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:19:32.022958 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:19:32.022929    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb"]
Oct 30 07:19:32.023173 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:19:32.023146    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb" podUID="eda91969-699f-446c-96ab-d49d54c97d26" containerName="istio-proxy" containerID="cri-o://357563b84dc3c267e9cd7df0ffe65ad8306af5b49d2ea0c9f1079b8be99422ab" gracePeriod=30
Oct 30 07:19:32.395271 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:19:32.395238    6224 patch_prober.go:29] interesting pod/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 07:19:32.395436 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:19:32.395289    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb" podUID="eda91969-699f-446c-96ab-d49d54c97d26" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 07:19:36.595245 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:19:36.595209    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:19:37.322556 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:19:37.311139    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb"
Oct 30 07:19:37.494787 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:19:37.494755    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eda91969-699f-446c-96ab-d49d54c97d26-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eda91969-699f-446c-96ab-d49d54c97d26-istio-data\") pod \"eda91969-699f-446c-96ab-d49d54c97d26\" (UID: \"eda91969-699f-446c-96ab-d49d54c97d26\") "
Oct 30 07:19:37.494987 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:19:37.494867    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eda91969-699f-446c-96ab-d49d54c97d26-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eda91969-699f-446c-96ab-d49d54c97d26-workload-certs\") pod \"eda91969-699f-446c-96ab-d49d54c97d26\" (UID: \"eda91969-699f-446c-96ab-d49d54c97d26\") "
Oct 30 07:19:37.494987 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:19:37.494913    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eda91969-699f-446c-96ab-d49d54c97d26-istio-token\" (UniqueName: \"kubernetes.io/projected/eda91969-699f-446c-96ab-d49d54c97d26-istio-token\") pod \"eda91969-699f-446c-96ab-d49d54c97d26\" (UID: \"eda91969-699f-446c-96ab-d49d54c97d26\") "
Oct 30 07:19:37.494987 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:19:37.494952    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eda91969-699f-446c-96ab-d49d54c97d26-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eda91969-699f-446c-96ab-d49d54c97d26-istio-podinfo\") pod \"eda91969-699f-446c-96ab-d49d54c97d26\" (UID: \"eda91969-699f-446c-96ab-d49d54c97d26\") "
Oct 30 07:19:37.495170 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:19:37.494992    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eda91969-699f-446c-96ab-d49d54c97d26-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eda91969-699f-446c-96ab-d49d54c97d26-istio-envoy\") pod \"eda91969-699f-446c-96ab-d49d54c97d26\" (UID: \"eda91969-699f-446c-96ab-d49d54c97d26\") "
Oct 30 07:19:37.495170 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:19:37.495042    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eda91969-699f-446c-96ab-d49d54c97d26-istio-data" pod "eda91969-699f-446c-96ab-d49d54c97d26" (UID: "eda91969-699f-446c-96ab-d49d54c97d26"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:19:37.495170 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:19:37.495049    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eda91969-699f-446c-96ab-d49d54c97d26-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eda91969-699f-446c-96ab-d49d54c97d26-credential-socket\") pod \"eda91969-699f-446c-96ab-d49d54c97d26\" (UID: \"eda91969-699f-446c-96ab-d49d54c97d26\") "
Oct 30 07:19:37.495170 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:19:37.495147    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eda91969-699f-446c-96ab-d49d54c97d26-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eda91969-699f-446c-96ab-d49d54c97d26-istiod-ca-cert\") pod \"eda91969-699f-446c-96ab-d49d54c97d26\" (UID: \"eda91969-699f-446c-96ab-d49d54c97d26\") "
Oct 30 07:19:37.495170 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:19:37.495150    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eda91969-699f-446c-96ab-d49d54c97d26-workload-certs" pod "eda91969-699f-446c-96ab-d49d54c97d26" (UID: "eda91969-699f-446c-96ab-d49d54c97d26"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:19:37.495437 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:19:37.495263    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eda91969-699f-446c-96ab-d49d54c97d26-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eda91969-699f-446c-96ab-d49d54c97d26-workload-socket\") pod \"eda91969-699f-446c-96ab-d49d54c97d26\" (UID: \"eda91969-699f-446c-96ab-d49d54c97d26\") "
Oct 30 07:19:37.495437 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:19:37.495339    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eda91969-699f-446c-96ab-d49d54c97d26-kube-api-access-xrrg9\" (UniqueName: \"kubernetes.io/projected/eda91969-699f-446c-96ab-d49d54c97d26-kube-api-access-xrrg9\") pod \"eda91969-699f-446c-96ab-d49d54c97d26\" (UID: \"eda91969-699f-446c-96ab-d49d54c97d26\") "
Oct 30 07:19:37.496007 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:19:37.495642    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eda91969-699f-446c-96ab-d49d54c97d26-credential-socket" pod "eda91969-699f-446c-96ab-d49d54c97d26" (UID: "eda91969-699f-446c-96ab-d49d54c97d26"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:19:37.496007 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:19:37.495854    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eda91969-699f-446c-96ab-d49d54c97d26-istiod-ca-cert" pod "eda91969-699f-446c-96ab-d49d54c97d26" (UID: "eda91969-699f-446c-96ab-d49d54c97d26"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:19:37.496216 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:19:37.496123    6224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eda91969-699f-446c-96ab-d49d54c97d26-istiod-ca-cert\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:19:37.496216 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:19:37.496141    6224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eda91969-699f-446c-96ab-d49d54c97d26-istio-data\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:19:37.496216 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:19:37.496154    6224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eda91969-699f-446c-96ab-d49d54c97d26-workload-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:19:37.496216 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:19:37.496168    6224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eda91969-699f-446c-96ab-d49d54c97d26-credential-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:19:37.496523 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:19:37.496473    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eda91969-699f-446c-96ab-d49d54c97d26-workload-socket" pod "eda91969-699f-446c-96ab-d49d54c97d26" (UID: "eda91969-699f-446c-96ab-d49d54c97d26"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:19:37.498560 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:19:37.498514    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eda91969-699f-446c-96ab-d49d54c97d26-kube-api-access-xrrg9" pod "eda91969-699f-446c-96ab-d49d54c97d26" (UID: "eda91969-699f-446c-96ab-d49d54c97d26"). InnerVolumeSpecName "kube-api-access-xrrg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:19:37.501839 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:19:37.501512    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eda91969-699f-446c-96ab-d49d54c97d26-istio-envoy" pod "eda91969-699f-446c-96ab-d49d54c97d26" (UID: "eda91969-699f-446c-96ab-d49d54c97d26"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:19:37.501839 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:19:37.501789    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eda91969-699f-446c-96ab-d49d54c97d26-istio-podinfo" pod "eda91969-699f-446c-96ab-d49d54c97d26" (UID: "eda91969-699f-446c-96ab-d49d54c97d26"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:19:37.504270 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:19:37.503347    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eda91969-699f-446c-96ab-d49d54c97d26-istio-token" pod "eda91969-699f-446c-96ab-d49d54c97d26" (UID: "eda91969-699f-446c-96ab-d49d54c97d26"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:19:37.598423 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:19:37.598326    6224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eda91969-699f-446c-96ab-d49d54c97d26-workload-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:19:37.598423 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:19:37.598369    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrrg9\" (UniqueName: \"kubernetes.io/projected/eda91969-699f-446c-96ab-d49d54c97d26-kube-api-access-xrrg9\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:19:37.598423 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:19:37.598389    6224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eda91969-699f-446c-96ab-d49d54c97d26-istio-token\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:19:37.598423 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:19:37.598404    6224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eda91969-699f-446c-96ab-d49d54c97d26-istio-podinfo\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:19:37.598423 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:19:37.598418    6224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eda91969-699f-446c-96ab-d49d54c97d26-istio-envoy\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:19:37.861514 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:19:37.861429    6224 generic.go:356] "Generic (PLEG): container finished" podID="eda91969-699f-446c-96ab-d49d54c97d26" containerID="357563b84dc3c267e9cd7df0ffe65ad8306af5b49d2ea0c9f1079b8be99422ab" exitCode=0
Oct 30 07:19:37.861514 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:19:37.861479    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb"
Oct 30 07:19:37.861702 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:19:37.861517    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb" event={"ID":"eda91969-699f-446c-96ab-d49d54c97d26","Type":"ContainerDied","Data":"357563b84dc3c267e9cd7df0ffe65ad8306af5b49d2ea0c9f1079b8be99422ab"}
Oct 30 07:19:37.861702 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:19:37.861584    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb" event={"ID":"eda91969-699f-446c-96ab-d49d54c97d26","Type":"ContainerDied","Data":"26bafe63d1d5d0456c5bdc130bcc9eb0ac6eb855384787de75cfbb075a96b162"}
Oct 30 07:19:37.861702 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:19:37.861613    6224 scope.go:122] "RemoveContainer" containerID="357563b84dc3c267e9cd7df0ffe65ad8306af5b49d2ea0c9f1079b8be99422ab"
Oct 30 07:19:37.871903 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:19:37.871886    6224 scope.go:122] "RemoveContainer" containerID="357563b84dc3c267e9cd7df0ffe65ad8306af5b49d2ea0c9f1079b8be99422ab"
Oct 30 07:19:37.872180 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:19:37.872159    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"357563b84dc3c267e9cd7df0ffe65ad8306af5b49d2ea0c9f1079b8be99422ab\": container with ID starting with 357563b84dc3c267e9cd7df0ffe65ad8306af5b49d2ea0c9f1079b8be99422ab not found: ID does not exist" containerID="357563b84dc3c267e9cd7df0ffe65ad8306af5b49d2ea0c9f1079b8be99422ab"
Oct 30 07:19:37.872238 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:19:37.872200    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"357563b84dc3c267e9cd7df0ffe65ad8306af5b49d2ea0c9f1079b8be99422ab"} err="failed to get container status \"357563b84dc3c267e9cd7df0ffe65ad8306af5b49d2ea0c9f1079b8be99422ab\": rpc error: code = NotFound desc = could not find container \"357563b84dc3c267e9cd7df0ffe65ad8306af5b49d2ea0c9f1079b8be99422ab\": container with ID starting with 357563b84dc3c267e9cd7df0ffe65ad8306af5b49d2ea0c9f1079b8be99422ab not found: ID does not exist"
Oct 30 07:19:37.933678 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:19:37.933648    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb"]
Oct 30 07:19:37.942143 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:19:37.942121    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-ba67d8-openshift-default-54b4468b87-hstcb"]
Oct 30 07:19:39.503785 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:19:39.503750    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eda91969-699f-446c-96ab-d49d54c97d26" path="/var/lib/kubelet/pods/eda91969-699f-446c-96ab-d49d54c97d26/volumes"
Oct 30 07:19:46.451744 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:19:46.451696    6224 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-125.ec2.internal" podUID="4ba2b683794afdaba9ca4ed884bfac10" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded"
Oct 30 07:20:01.858826 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:20:01.858771    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:20:15.420387 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:15.420309    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2"]
Oct 30 07:20:15.420853 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:15.420545    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2" podUID="5583b983-1645-4e19-acb3-8e2f498f6b5b" containerName="istio-proxy" containerID="cri-o://b0569e021d4e302ca6c6f736e53810818ccb1c88692bd2746bfe157eabab1629" gracePeriod=30
Oct 30 07:20:15.685199 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:15.685130    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:20:15.742311 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:15.742282    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2"]
Oct 30 07:20:15.742509 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:15.742484    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2" podUID="3a24e154-72f7-48d6-9d93-2f9fee0de386" containerName="istio-proxy" containerID="cri-o://462e3e73493d0d3ac4b08e04c48d1a80970a2cbc34da0a76315df502e1dd13f6" gracePeriod=30
Oct 30 07:20:20.692866 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:20.692744    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2"
Oct 30 07:20:20.840142 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:20.840099    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5583b983-1645-4e19-acb3-8e2f498f6b5b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5583b983-1645-4e19-acb3-8e2f498f6b5b-istiod-ca-cert\") pod \"5583b983-1645-4e19-acb3-8e2f498f6b5b\" (UID: \"5583b983-1645-4e19-acb3-8e2f498f6b5b\") "
Oct 30 07:20:20.840333 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:20.840178    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5583b983-1645-4e19-acb3-8e2f498f6b5b-istio-token\" (UniqueName: \"kubernetes.io/projected/5583b983-1645-4e19-acb3-8e2f498f6b5b-istio-token\") pod \"5583b983-1645-4e19-acb3-8e2f498f6b5b\" (UID: \"5583b983-1645-4e19-acb3-8e2f498f6b5b\") "
Oct 30 07:20:20.840333 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:20.840219    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5583b983-1645-4e19-acb3-8e2f498f6b5b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5583b983-1645-4e19-acb3-8e2f498f6b5b-workload-certs\") pod \"5583b983-1645-4e19-acb3-8e2f498f6b5b\" (UID: \"5583b983-1645-4e19-acb3-8e2f498f6b5b\") "
Oct 30 07:20:20.840333 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:20.840272    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5583b983-1645-4e19-acb3-8e2f498f6b5b-kube-api-access-2ljtx\" (UniqueName: \"kubernetes.io/projected/5583b983-1645-4e19-acb3-8e2f498f6b5b-kube-api-access-2ljtx\") pod \"5583b983-1645-4e19-acb3-8e2f498f6b5b\" (UID: \"5583b983-1645-4e19-acb3-8e2f498f6b5b\") "
Oct 30 07:20:20.840333 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:20.840307    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5583b983-1645-4e19-acb3-8e2f498f6b5b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5583b983-1645-4e19-acb3-8e2f498f6b5b-istio-envoy\") pod \"5583b983-1645-4e19-acb3-8e2f498f6b5b\" (UID: \"5583b983-1645-4e19-acb3-8e2f498f6b5b\") "
Oct 30 07:20:20.840543 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:20.840371    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5583b983-1645-4e19-acb3-8e2f498f6b5b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5583b983-1645-4e19-acb3-8e2f498f6b5b-istio-data\") pod \"5583b983-1645-4e19-acb3-8e2f498f6b5b\" (UID: \"5583b983-1645-4e19-acb3-8e2f498f6b5b\") "
Oct 30 07:20:20.840543 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:20.840438    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5583b983-1645-4e19-acb3-8e2f498f6b5b-istiod-ca-cert" pod "5583b983-1645-4e19-acb3-8e2f498f6b5b" (UID: "5583b983-1645-4e19-acb3-8e2f498f6b5b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:20:20.840543 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:20.840495    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5583b983-1645-4e19-acb3-8e2f498f6b5b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5583b983-1645-4e19-acb3-8e2f498f6b5b-credential-socket\") pod \"5583b983-1645-4e19-acb3-8e2f498f6b5b\" (UID: \"5583b983-1645-4e19-acb3-8e2f498f6b5b\") "
Oct 30 07:20:20.840689 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:20.840563    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5583b983-1645-4e19-acb3-8e2f498f6b5b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5583b983-1645-4e19-acb3-8e2f498f6b5b-workload-socket\") pod \"5583b983-1645-4e19-acb3-8e2f498f6b5b\" (UID: \"5583b983-1645-4e19-acb3-8e2f498f6b5b\") "
Oct 30 07:20:20.840689 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:20.840568    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5583b983-1645-4e19-acb3-8e2f498f6b5b-workload-certs" pod "5583b983-1645-4e19-acb3-8e2f498f6b5b" (UID: "5583b983-1645-4e19-acb3-8e2f498f6b5b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:20:20.840689 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:20.840598    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5583b983-1645-4e19-acb3-8e2f498f6b5b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5583b983-1645-4e19-acb3-8e2f498f6b5b-istio-podinfo\") pod \"5583b983-1645-4e19-acb3-8e2f498f6b5b\" (UID: \"5583b983-1645-4e19-acb3-8e2f498f6b5b\") "
Oct 30 07:20:20.840880 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:20.840860    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5583b983-1645-4e19-acb3-8e2f498f6b5b-credential-socket" pod "5583b983-1645-4e19-acb3-8e2f498f6b5b" (UID: "5583b983-1645-4e19-acb3-8e2f498f6b5b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:20:20.841076 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:20.841037    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5583b983-1645-4e19-acb3-8e2f498f6b5b-workload-socket" pod "5583b983-1645-4e19-acb3-8e2f498f6b5b" (UID: "5583b983-1645-4e19-acb3-8e2f498f6b5b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:20:20.841310 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:20.841289    6224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5583b983-1645-4e19-acb3-8e2f498f6b5b-istiod-ca-cert\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:20:20.841310 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:20.841316    6224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5583b983-1645-4e19-acb3-8e2f498f6b5b-workload-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:20:20.841430 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:20.841331    6224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5583b983-1645-4e19-acb3-8e2f498f6b5b-credential-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:20:20.841430 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:20.841347    6224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5583b983-1645-4e19-acb3-8e2f498f6b5b-workload-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:20:20.841566 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:20.841545    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5583b983-1645-4e19-acb3-8e2f498f6b5b-istio-data" pod "5583b983-1645-4e19-acb3-8e2f498f6b5b" (UID: "5583b983-1645-4e19-acb3-8e2f498f6b5b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:20:20.842819 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:20.842776    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5583b983-1645-4e19-acb3-8e2f498f6b5b-istio-envoy" pod "5583b983-1645-4e19-acb3-8e2f498f6b5b" (UID: "5583b983-1645-4e19-acb3-8e2f498f6b5b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:20:20.842938 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:20.842900    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5583b983-1645-4e19-acb3-8e2f498f6b5b-istio-podinfo" pod "5583b983-1645-4e19-acb3-8e2f498f6b5b" (UID: "5583b983-1645-4e19-acb3-8e2f498f6b5b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:20:20.843006 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:20.842985    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5583b983-1645-4e19-acb3-8e2f498f6b5b-kube-api-access-2ljtx" pod "5583b983-1645-4e19-acb3-8e2f498f6b5b" (UID: "5583b983-1645-4e19-acb3-8e2f498f6b5b"). InnerVolumeSpecName "kube-api-access-2ljtx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:20:20.843049 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:20.843016    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5583b983-1645-4e19-acb3-8e2f498f6b5b-istio-token" pod "5583b983-1645-4e19-acb3-8e2f498f6b5b" (UID: "5583b983-1645-4e19-acb3-8e2f498f6b5b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:20:20.944073 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:20.943459    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ljtx\" (UniqueName: \"kubernetes.io/projected/5583b983-1645-4e19-acb3-8e2f498f6b5b-kube-api-access-2ljtx\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:20:20.944073 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:20.943490    6224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5583b983-1645-4e19-acb3-8e2f498f6b5b-istio-envoy\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:20:20.944073 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:20.943508    6224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5583b983-1645-4e19-acb3-8e2f498f6b5b-istio-data\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:20:20.944073 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:20.943522    6224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5583b983-1645-4e19-acb3-8e2f498f6b5b-istio-podinfo\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:20:20.944073 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:20.943535    6224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5583b983-1645-4e19-acb3-8e2f498f6b5b-istio-token\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:20:21.046872 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:21.046385    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2"
Oct 30 07:20:21.165828 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:21.151226    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a24e154-72f7-48d6-9d93-2f9fee0de386-kube-api-access-nrpj8\" (UniqueName: \"kubernetes.io/projected/3a24e154-72f7-48d6-9d93-2f9fee0de386-kube-api-access-nrpj8\") pod \"3a24e154-72f7-48d6-9d93-2f9fee0de386\" (UID: \"3a24e154-72f7-48d6-9d93-2f9fee0de386\") "
Oct 30 07:20:21.165828 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:21.151288    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a24e154-72f7-48d6-9d93-2f9fee0de386-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a24e154-72f7-48d6-9d93-2f9fee0de386-workload-certs\") pod \"3a24e154-72f7-48d6-9d93-2f9fee0de386\" (UID: \"3a24e154-72f7-48d6-9d93-2f9fee0de386\") "
Oct 30 07:20:21.165828 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:21.151338    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a24e154-72f7-48d6-9d93-2f9fee0de386-istio-token\" (UniqueName: \"kubernetes.io/projected/3a24e154-72f7-48d6-9d93-2f9fee0de386-istio-token\") pod \"3a24e154-72f7-48d6-9d93-2f9fee0de386\" (UID: \"3a24e154-72f7-48d6-9d93-2f9fee0de386\") "
Oct 30 07:20:21.165828 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:21.151412    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3a24e154-72f7-48d6-9d93-2f9fee0de386-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a24e154-72f7-48d6-9d93-2f9fee0de386-istio-podinfo\") pod \"3a24e154-72f7-48d6-9d93-2f9fee0de386\" (UID: \"3a24e154-72f7-48d6-9d93-2f9fee0de386\") "
Oct 30 07:20:21.165828 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:21.151454    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a24e154-72f7-48d6-9d93-2f9fee0de386-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a24e154-72f7-48d6-9d93-2f9fee0de386-credential-socket\") pod \"3a24e154-72f7-48d6-9d93-2f9fee0de386\" (UID: \"3a24e154-72f7-48d6-9d93-2f9fee0de386\") "
Oct 30 07:20:21.165828 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:21.151530    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a24e154-72f7-48d6-9d93-2f9fee0de386-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a24e154-72f7-48d6-9d93-2f9fee0de386-workload-socket\") pod \"3a24e154-72f7-48d6-9d93-2f9fee0de386\" (UID: \"3a24e154-72f7-48d6-9d93-2f9fee0de386\") "
Oct 30 07:20:21.165828 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:21.151560    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a24e154-72f7-48d6-9d93-2f9fee0de386-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a24e154-72f7-48d6-9d93-2f9fee0de386-istio-envoy\") pod \"3a24e154-72f7-48d6-9d93-2f9fee0de386\" (UID: \"3a24e154-72f7-48d6-9d93-2f9fee0de386\") "
Oct 30 07:20:21.165828 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:21.151593    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a24e154-72f7-48d6-9d93-2f9fee0de386-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a24e154-72f7-48d6-9d93-2f9fee0de386-istiod-ca-cert\") pod \"3a24e154-72f7-48d6-9d93-2f9fee0de386\" (UID: \"3a24e154-72f7-48d6-9d93-2f9fee0de386\") "
Oct 30 07:20:21.165828 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:21.151699    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a24e154-72f7-48d6-9d93-2f9fee0de386-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a24e154-72f7-48d6-9d93-2f9fee0de386-istio-data\") pod \"3a24e154-72f7-48d6-9d93-2f9fee0de386\" (UID: \"3a24e154-72f7-48d6-9d93-2f9fee0de386\") "
Oct 30 07:20:21.170774 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:21.170463    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a24e154-72f7-48d6-9d93-2f9fee0de386-workload-certs" pod "3a24e154-72f7-48d6-9d93-2f9fee0de386" (UID: "3a24e154-72f7-48d6-9d93-2f9fee0de386"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:20:21.173116 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:21.171119    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a24e154-72f7-48d6-9d93-2f9fee0de386-istio-data" pod "3a24e154-72f7-48d6-9d93-2f9fee0de386" (UID: "3a24e154-72f7-48d6-9d93-2f9fee0de386"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:20:21.173116 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:21.171526    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a24e154-72f7-48d6-9d93-2f9fee0de386-istiod-ca-cert" pod "3a24e154-72f7-48d6-9d93-2f9fee0de386" (UID: "3a24e154-72f7-48d6-9d93-2f9fee0de386"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:20:21.173116 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:21.171781    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a24e154-72f7-48d6-9d93-2f9fee0de386-workload-socket" pod "3a24e154-72f7-48d6-9d93-2f9fee0de386" (UID: "3a24e154-72f7-48d6-9d93-2f9fee0de386"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:20:21.173653 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:21.173617    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a24e154-72f7-48d6-9d93-2f9fee0de386-credential-socket" pod "3a24e154-72f7-48d6-9d93-2f9fee0de386" (UID: "3a24e154-72f7-48d6-9d93-2f9fee0de386"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:20:21.180741 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:21.180705    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a24e154-72f7-48d6-9d93-2f9fee0de386-istio-token" pod "3a24e154-72f7-48d6-9d93-2f9fee0de386" (UID: "3a24e154-72f7-48d6-9d93-2f9fee0de386"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:20:21.184991 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:21.184953    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a24e154-72f7-48d6-9d93-2f9fee0de386-istio-envoy" pod "3a24e154-72f7-48d6-9d93-2f9fee0de386" (UID: "3a24e154-72f7-48d6-9d93-2f9fee0de386"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:20:21.196032 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:21.195949    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a24e154-72f7-48d6-9d93-2f9fee0de386-istio-podinfo" pod "3a24e154-72f7-48d6-9d93-2f9fee0de386" (UID: "3a24e154-72f7-48d6-9d93-2f9fee0de386"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:20:21.200015 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:21.199981    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a24e154-72f7-48d6-9d93-2f9fee0de386-kube-api-access-nrpj8" pod "3a24e154-72f7-48d6-9d93-2f9fee0de386" (UID: "3a24e154-72f7-48d6-9d93-2f9fee0de386"). InnerVolumeSpecName "kube-api-access-nrpj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:20:21.202106 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:21.202071    6224 generic.go:356] "Generic (PLEG): container finished" podID="5583b983-1645-4e19-acb3-8e2f498f6b5b" containerID="b0569e021d4e302ca6c6f736e53810818ccb1c88692bd2746bfe157eabab1629" exitCode=0
Oct 30 07:20:21.202236 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:21.202134    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2"
Oct 30 07:20:21.202236 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:21.202129    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2" event={"ID":"5583b983-1645-4e19-acb3-8e2f498f6b5b","Type":"ContainerDied","Data":"b0569e021d4e302ca6c6f736e53810818ccb1c88692bd2746bfe157eabab1629"}
Oct 30 07:20:21.202351 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:21.202293    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2" event={"ID":"5583b983-1645-4e19-acb3-8e2f498f6b5b","Type":"ContainerDied","Data":"844fb8ffa43b5a45161e1c4ecb230b5a148abfcaad60f043d26f304c76c37087"}
Oct 30 07:20:21.202351 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:21.202329    6224 scope.go:122] "RemoveContainer" containerID="b0569e021d4e302ca6c6f736e53810818ccb1c88692bd2746bfe157eabab1629"
Oct 30 07:20:21.204385 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:21.204361    6224 generic.go:356] "Generic (PLEG): container finished" podID="3a24e154-72f7-48d6-9d93-2f9fee0de386" containerID="462e3e73493d0d3ac4b08e04c48d1a80970a2cbc34da0a76315df502e1dd13f6" exitCode=0
Oct 30 07:20:21.204526 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:21.204430    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2" event={"ID":"3a24e154-72f7-48d6-9d93-2f9fee0de386","Type":"ContainerDied","Data":"462e3e73493d0d3ac4b08e04c48d1a80970a2cbc34da0a76315df502e1dd13f6"}
Oct 30 07:20:21.204526 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:21.204473    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2" event={"ID":"3a24e154-72f7-48d6-9d93-2f9fee0de386","Type":"ContainerDied","Data":"76544539c080b8ec1bb08371125eb3229090d02d6a87153cce22c32f7cf006df"}
Oct 30 07:20:21.204688 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:21.204550    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2"
Oct 30 07:20:21.231974 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:21.231948    6224 scope.go:122] "RemoveContainer" containerID="b0569e021d4e302ca6c6f736e53810818ccb1c88692bd2746bfe157eabab1629"
Oct 30 07:20:21.232347 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:20:21.232322    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0569e021d4e302ca6c6f736e53810818ccb1c88692bd2746bfe157eabab1629\": container with ID starting with b0569e021d4e302ca6c6f736e53810818ccb1c88692bd2746bfe157eabab1629 not found: ID does not exist" containerID="b0569e021d4e302ca6c6f736e53810818ccb1c88692bd2746bfe157eabab1629"
Oct 30 07:20:21.232394 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:21.232362    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0569e021d4e302ca6c6f736e53810818ccb1c88692bd2746bfe157eabab1629"} err="failed to get container status \"b0569e021d4e302ca6c6f736e53810818ccb1c88692bd2746bfe157eabab1629\": rpc error: code = NotFound desc = could not find container \"b0569e021d4e302ca6c6f736e53810818ccb1c88692bd2746bfe157eabab1629\": container with ID starting with b0569e021d4e302ca6c6f736e53810818ccb1c88692bd2746bfe157eabab1629 not found: ID does not exist"
Oct 30 07:20:21.232394 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:21.232387    6224 scope.go:122] "RemoveContainer" containerID="462e3e73493d0d3ac4b08e04c48d1a80970a2cbc34da0a76315df502e1dd13f6"
Oct 30 07:20:21.243366 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:21.243346    6224 scope.go:122] "RemoveContainer" containerID="462e3e73493d0d3ac4b08e04c48d1a80970a2cbc34da0a76315df502e1dd13f6"
Oct 30 07:20:21.243658 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:20:21.243636    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"462e3e73493d0d3ac4b08e04c48d1a80970a2cbc34da0a76315df502e1dd13f6\": container with ID starting with 462e3e73493d0d3ac4b08e04c48d1a80970a2cbc34da0a76315df502e1dd13f6 not found: ID does not exist" containerID="462e3e73493d0d3ac4b08e04c48d1a80970a2cbc34da0a76315df502e1dd13f6"
Oct 30 07:20:21.243752 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:21.243667    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"462e3e73493d0d3ac4b08e04c48d1a80970a2cbc34da0a76315df502e1dd13f6"} err="failed to get container status \"462e3e73493d0d3ac4b08e04c48d1a80970a2cbc34da0a76315df502e1dd13f6\": rpc error: code = NotFound desc = could not find container \"462e3e73493d0d3ac4b08e04c48d1a80970a2cbc34da0a76315df502e1dd13f6\": container with ID starting with 462e3e73493d0d3ac4b08e04c48d1a80970a2cbc34da0a76315df502e1dd13f6 not found: ID does not exist"
Oct 30 07:20:21.256682 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:21.256657    6224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a24e154-72f7-48d6-9d93-2f9fee0de386-istio-data\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:20:21.256682 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:21.256682    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrpj8\" (UniqueName: \"kubernetes.io/projected/3a24e154-72f7-48d6-9d93-2f9fee0de386-kube-api-access-nrpj8\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:20:21.256866 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:21.256697    6224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a24e154-72f7-48d6-9d93-2f9fee0de386-workload-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:20:21.256866 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:21.256710    6224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a24e154-72f7-48d6-9d93-2f9fee0de386-istio-token\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:20:21.256866 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:21.256721    6224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a24e154-72f7-48d6-9d93-2f9fee0de386-istio-podinfo\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:20:21.256866 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:21.256737    6224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a24e154-72f7-48d6-9d93-2f9fee0de386-credential-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:20:21.256866 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:21.256752    6224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a24e154-72f7-48d6-9d93-2f9fee0de386-workload-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:20:21.256866 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:21.256767    6224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a24e154-72f7-48d6-9d93-2f9fee0de386-istio-envoy\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:20:21.256866 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:21.256780    6224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a24e154-72f7-48d6-9d93-2f9fee0de386-istiod-ca-cert\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:20:21.308008 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:21.307975    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2"]
Oct 30 07:20:21.310648 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:21.310628    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mfxm2"]
Oct 30 07:20:21.349128 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:21.349102    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2"]
Oct 30 07:20:21.351776 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:21.351751    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-wdqf2"]
Oct 30 07:20:21.504984 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:21.504950    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a24e154-72f7-48d6-9d93-2f9fee0de386" path="/var/lib/kubelet/pods/3a24e154-72f7-48d6-9d93-2f9fee0de386/volumes"
Oct 30 07:20:21.505512 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:21.505495    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5583b983-1645-4e19-acb3-8e2f498f6b5b" path="/var/lib/kubelet/pods/5583b983-1645-4e19-acb3-8e2f498f6b5b/volumes"
Oct 30 07:20:31.859393 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:20:31.859352    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:20:42.016218 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:42.016180    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm"]
Oct 30 07:20:42.016831 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:42.016491    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm" podUID="16b1f8ac-8fe8-4dd6-a843-2470652b69ab" containerName="istio-proxy" containerID="cri-o://a4635e456f5d90f57ef4b1b2bb0357d5d4ed8bed6f88bbb580fb26500e5598d0" gracePeriod=30
Oct 30 07:20:42.334858 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:42.320289    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:20:43.883277 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:43.883202    6224 patch_prober.go:29] interesting pod/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 07:20:43.883645 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:43.883270    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm" podUID="16b1f8ac-8fe8-4dd6-a843-2470652b69ab" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 07:20:47.295291 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:47.295262    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm"
Oct 30 07:20:47.367699 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:47.367661    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16b1f8ac-8fe8-4dd6-a843-2470652b69ab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b1f8ac-8fe8-4dd6-a843-2470652b69ab-credential-socket\") pod \"16b1f8ac-8fe8-4dd6-a843-2470652b69ab\" (UID: \"16b1f8ac-8fe8-4dd6-a843-2470652b69ab\") "
Oct 30 07:20:47.372429 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:47.370889    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16b1f8ac-8fe8-4dd6-a843-2470652b69ab-credential-socket" pod "16b1f8ac-8fe8-4dd6-a843-2470652b69ab" (UID: "16b1f8ac-8fe8-4dd6-a843-2470652b69ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:20:47.373221 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:47.372777    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16b1f8ac-8fe8-4dd6-a843-2470652b69ab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16b1f8ac-8fe8-4dd6-a843-2470652b69ab-istio-envoy\") pod \"16b1f8ac-8fe8-4dd6-a843-2470652b69ab\" (UID: \"16b1f8ac-8fe8-4dd6-a843-2470652b69ab\") "
Oct 30 07:20:47.373465 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:47.373445    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16b1f8ac-8fe8-4dd6-a843-2470652b69ab-istio-token\" (UniqueName: \"kubernetes.io/projected/16b1f8ac-8fe8-4dd6-a843-2470652b69ab-istio-token\") pod \"16b1f8ac-8fe8-4dd6-a843-2470652b69ab\" (UID: \"16b1f8ac-8fe8-4dd6-a843-2470652b69ab\") "
Oct 30 07:20:47.374057 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:47.374033    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/16b1f8ac-8fe8-4dd6-a843-2470652b69ab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16b1f8ac-8fe8-4dd6-a843-2470652b69ab-istio-podinfo\") pod \"16b1f8ac-8fe8-4dd6-a843-2470652b69ab\" (UID: \"16b1f8ac-8fe8-4dd6-a843-2470652b69ab\") "
Oct 30 07:20:47.374424 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:47.374402    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16b1f8ac-8fe8-4dd6-a843-2470652b69ab-kube-api-access-rnsbh\" (UniqueName: \"kubernetes.io/projected/16b1f8ac-8fe8-4dd6-a843-2470652b69ab-kube-api-access-rnsbh\") pod \"16b1f8ac-8fe8-4dd6-a843-2470652b69ab\" (UID: \"16b1f8ac-8fe8-4dd6-a843-2470652b69ab\") "
Oct 30 07:20:47.374533 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:47.374460    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16b1f8ac-8fe8-4dd6-a843-2470652b69ab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16b1f8ac-8fe8-4dd6-a843-2470652b69ab-istiod-ca-cert\") pod \"16b1f8ac-8fe8-4dd6-a843-2470652b69ab\" (UID: \"16b1f8ac-8fe8-4dd6-a843-2470652b69ab\") "
Oct 30 07:20:47.374533 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:47.374508    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16b1f8ac-8fe8-4dd6-a843-2470652b69ab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b1f8ac-8fe8-4dd6-a843-2470652b69ab-workload-socket\") pod \"16b1f8ac-8fe8-4dd6-a843-2470652b69ab\" (UID: \"16b1f8ac-8fe8-4dd6-a843-2470652b69ab\") "
Oct 30 07:20:47.374705 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:47.374683    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16b1f8ac-8fe8-4dd6-a843-2470652b69ab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16b1f8ac-8fe8-4dd6-a843-2470652b69ab-istio-data\") pod \"16b1f8ac-8fe8-4dd6-a843-2470652b69ab\" (UID: \"16b1f8ac-8fe8-4dd6-a843-2470652b69ab\") "
Oct 30 07:20:47.375032 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:47.375011    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16b1f8ac-8fe8-4dd6-a843-2470652b69ab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16b1f8ac-8fe8-4dd6-a843-2470652b69ab-workload-certs\") pod \"16b1f8ac-8fe8-4dd6-a843-2470652b69ab\" (UID: \"16b1f8ac-8fe8-4dd6-a843-2470652b69ab\") "
Oct 30 07:20:47.375343 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:47.374991    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16b1f8ac-8fe8-4dd6-a843-2470652b69ab-istio-data" pod "16b1f8ac-8fe8-4dd6-a843-2470652b69ab" (UID: "16b1f8ac-8fe8-4dd6-a843-2470652b69ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:20:47.375655 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:47.375095    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16b1f8ac-8fe8-4dd6-a843-2470652b69ab-workload-socket" pod "16b1f8ac-8fe8-4dd6-a843-2470652b69ab" (UID: "16b1f8ac-8fe8-4dd6-a843-2470652b69ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:20:47.375655 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:47.375457    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16b1f8ac-8fe8-4dd6-a843-2470652b69ab-istiod-ca-cert" pod "16b1f8ac-8fe8-4dd6-a843-2470652b69ab" (UID: "16b1f8ac-8fe8-4dd6-a843-2470652b69ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:20:47.375958 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:47.375480    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16b1f8ac-8fe8-4dd6-a843-2470652b69ab-workload-certs" pod "16b1f8ac-8fe8-4dd6-a843-2470652b69ab" (UID: "16b1f8ac-8fe8-4dd6-a843-2470652b69ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:20:47.376180 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:47.376159    6224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b1f8ac-8fe8-4dd6-a843-2470652b69ab-credential-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:20:47.378353 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:47.376187    6224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16b1f8ac-8fe8-4dd6-a843-2470652b69ab-istiod-ca-cert\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:20:47.378353 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:47.376203    6224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b1f8ac-8fe8-4dd6-a843-2470652b69ab-workload-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:20:47.378353 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:47.376217    6224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16b1f8ac-8fe8-4dd6-a843-2470652b69ab-istio-data\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:20:47.378353 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:47.376232    6224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16b1f8ac-8fe8-4dd6-a843-2470652b69ab-workload-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:20:47.378353 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:47.376625    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16b1f8ac-8fe8-4dd6-a843-2470652b69ab-istio-podinfo" pod "16b1f8ac-8fe8-4dd6-a843-2470652b69ab" (UID: "16b1f8ac-8fe8-4dd6-a843-2470652b69ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:20:47.381209 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:47.378708    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16b1f8ac-8fe8-4dd6-a843-2470652b69ab-istio-token" pod "16b1f8ac-8fe8-4dd6-a843-2470652b69ab" (UID: "16b1f8ac-8fe8-4dd6-a843-2470652b69ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:20:47.381209 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:47.379865    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16b1f8ac-8fe8-4dd6-a843-2470652b69ab-istio-envoy" pod "16b1f8ac-8fe8-4dd6-a843-2470652b69ab" (UID: "16b1f8ac-8fe8-4dd6-a843-2470652b69ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:20:47.381456 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:47.381432    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16b1f8ac-8fe8-4dd6-a843-2470652b69ab-kube-api-access-rnsbh" pod "16b1f8ac-8fe8-4dd6-a843-2470652b69ab" (UID: "16b1f8ac-8fe8-4dd6-a843-2470652b69ab"). InnerVolumeSpecName "kube-api-access-rnsbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:20:47.424205 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:47.424169    6224 generic.go:356] "Generic (PLEG): container finished" podID="16b1f8ac-8fe8-4dd6-a843-2470652b69ab" containerID="a4635e456f5d90f57ef4b1b2bb0357d5d4ed8bed6f88bbb580fb26500e5598d0" exitCode=0
Oct 30 07:20:47.424382 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:47.424233    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm"
Oct 30 07:20:47.424382 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:47.424249    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm" event={"ID":"16b1f8ac-8fe8-4dd6-a843-2470652b69ab","Type":"ContainerDied","Data":"a4635e456f5d90f57ef4b1b2bb0357d5d4ed8bed6f88bbb580fb26500e5598d0"}
Oct 30 07:20:47.424382 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:47.424347    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm" event={"ID":"16b1f8ac-8fe8-4dd6-a843-2470652b69ab","Type":"ContainerDied","Data":"965fff3da67538dd43684796f63aa8ac1ccaa1c9a9947c33c22598fe98739a1e"}
Oct 30 07:20:47.424382 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:47.424379    6224 scope.go:122] "RemoveContainer" containerID="a4635e456f5d90f57ef4b1b2bb0357d5d4ed8bed6f88bbb580fb26500e5598d0"
Oct 30 07:20:47.435829 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:47.435786    6224 scope.go:122] "RemoveContainer" containerID="1167e354b39eceac6c07062ada570d558bb29a3790e412303c9ad6b7fb0ded5c"
Oct 30 07:20:47.447683 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:47.447661    6224 scope.go:122] "RemoveContainer" containerID="a4635e456f5d90f57ef4b1b2bb0357d5d4ed8bed6f88bbb580fb26500e5598d0"
Oct 30 07:20:47.448103 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:20:47.448073    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4635e456f5d90f57ef4b1b2bb0357d5d4ed8bed6f88bbb580fb26500e5598d0\": container with ID starting with a4635e456f5d90f57ef4b1b2bb0357d5d4ed8bed6f88bbb580fb26500e5598d0 not found: ID does not exist" containerID="a4635e456f5d90f57ef4b1b2bb0357d5d4ed8bed6f88bbb580fb26500e5598d0"
Oct 30 07:20:47.448202 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:47.448118    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4635e456f5d90f57ef4b1b2bb0357d5d4ed8bed6f88bbb580fb26500e5598d0"} err="failed to get container status \"a4635e456f5d90f57ef4b1b2bb0357d5d4ed8bed6f88bbb580fb26500e5598d0\": rpc error: code = NotFound desc = could not find container \"a4635e456f5d90f57ef4b1b2bb0357d5d4ed8bed6f88bbb580fb26500e5598d0\": container with ID starting with a4635e456f5d90f57ef4b1b2bb0357d5d4ed8bed6f88bbb580fb26500e5598d0 not found: ID does not exist"
Oct 30 07:20:47.448202 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:47.448140    6224 scope.go:122] "RemoveContainer" containerID="1167e354b39eceac6c07062ada570d558bb29a3790e412303c9ad6b7fb0ded5c"
Oct 30 07:20:47.448493 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:20:47.448453    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1167e354b39eceac6c07062ada570d558bb29a3790e412303c9ad6b7fb0ded5c\": container with ID starting with 1167e354b39eceac6c07062ada570d558bb29a3790e412303c9ad6b7fb0ded5c not found: ID does not exist" containerID="1167e354b39eceac6c07062ada570d558bb29a3790e412303c9ad6b7fb0ded5c"
Oct 30 07:20:47.448604 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:47.448502    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1167e354b39eceac6c07062ada570d558bb29a3790e412303c9ad6b7fb0ded5c"} err="failed to get container status \"1167e354b39eceac6c07062ada570d558bb29a3790e412303c9ad6b7fb0ded5c\": rpc error: code = NotFound desc = could not find container \"1167e354b39eceac6c07062ada570d558bb29a3790e412303c9ad6b7fb0ded5c\": container with ID starting with 1167e354b39eceac6c07062ada570d558bb29a3790e412303c9ad6b7fb0ded5c not found: ID does not exist"
Oct 30 07:20:47.478053 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:47.478017    6224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16b1f8ac-8fe8-4dd6-a843-2470652b69ab-istio-podinfo\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:20:47.478053 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:47.478045    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnsbh\" (UniqueName: \"kubernetes.io/projected/16b1f8ac-8fe8-4dd6-a843-2470652b69ab-kube-api-access-rnsbh\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:20:47.478258 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:47.478061    6224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16b1f8ac-8fe8-4dd6-a843-2470652b69ab-istio-envoy\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:20:47.478258 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:47.478073    6224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16b1f8ac-8fe8-4dd6-a843-2470652b69ab-istio-token\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:20:47.483408 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:47.483383    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm"]
Oct 30 07:20:47.490312 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:47.490286    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-a273fd-openshift-default-fd48fc9df-x26hm"]
Oct 30 07:20:47.504032 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:20:47.504005    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16b1f8ac-8fe8-4dd6-a843-2470652b69ab" path="/var/lib/kubelet/pods/16b1f8ac-8fe8-4dd6-a843-2470652b69ab/volumes"
Oct 30 07:21:01.856836 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:21:01.856785    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:21:06.487090 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:06.487054    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:21:22.256631 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:22.256598    6224 scope.go:122] "RemoveContainer" containerID="e470451d24cb0d55996889804f23dcc2ace9bb8b7b5bd2c58760657d9057d00b"
Oct 30 07:21:29.618831 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:29.618311    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:21:31.857404 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:21:31.857367    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:21:56.849623 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:56.849588    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54"]
Oct 30 07:21:56.850374 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:56.850352    6224 cpu_manager.go:414] "containerMap: removing container" podUID="bd9eda1c-1c26-4617-8f10-886e51c2b3e9" containerName="payload-pre-processing"
Oct 30 07:21:56.850422 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:56.850378    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd9eda1c-1c26-4617-8f10-886e51c2b3e9" containerName="payload-pre-processing"
Oct 30 07:21:56.850457 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:56.850430    6224 cpu_manager.go:414] "containerMap: removing container" podUID="3a24e154-72f7-48d6-9d93-2f9fee0de386" containerName="istio-proxy"
Oct 30 07:21:56.850457 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:56.850439    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a24e154-72f7-48d6-9d93-2f9fee0de386" containerName="istio-proxy"
Oct 30 07:21:56.850457 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:56.850449    6224 cpu_manager.go:414] "containerMap: removing container" podUID="eda91969-699f-446c-96ab-d49d54c97d26" containerName="istio-proxy"
Oct 30 07:21:56.850553 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:56.850457    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eda91969-699f-446c-96ab-d49d54c97d26" containerName="istio-proxy"
Oct 30 07:21:56.850553 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:56.850475    6224 cpu_manager.go:414] "containerMap: removing container" podUID="16b1f8ac-8fe8-4dd6-a843-2470652b69ab" containerName="istio-proxy"
Oct 30 07:21:56.850553 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:56.850483    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16b1f8ac-8fe8-4dd6-a843-2470652b69ab" containerName="istio-proxy"
Oct 30 07:21:56.850553 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:56.850492    6224 cpu_manager.go:414] "containerMap: removing container" podUID="a583c68f-99d3-4e7f-ab2c-5b7343cf44b8" containerName="payload-processing"
Oct 30 07:21:56.850553 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:56.850499    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a583c68f-99d3-4e7f-ab2c-5b7343cf44b8" containerName="payload-processing"
Oct 30 07:21:56.850553 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:56.850518    6224 cpu_manager.go:414] "containerMap: removing container" podUID="5583b983-1645-4e19-acb3-8e2f498f6b5b" containerName="istio-proxy"
Oct 30 07:21:56.850553 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:56.850526    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5583b983-1645-4e19-acb3-8e2f498f6b5b" containerName="istio-proxy"
Oct 30 07:21:56.850553 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:56.850546    6224 cpu_manager.go:414] "containerMap: removing container" podUID="16b1f8ac-8fe8-4dd6-a843-2470652b69ab" containerName="istio-proxy"
Oct 30 07:21:56.850553 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:56.850553    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16b1f8ac-8fe8-4dd6-a843-2470652b69ab" containerName="istio-proxy"
Oct 30 07:21:56.850842 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:56.850569    6224 cpu_manager.go:414] "containerMap: removing container" podUID="12729552-3e12-4b64-8779-a1584734c16e" containerName="istio-proxy"
Oct 30 07:21:56.850842 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:56.850576    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12729552-3e12-4b64-8779-a1584734c16e" containerName="istio-proxy"
Oct 30 07:21:56.850842 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:56.850693    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="eda91969-699f-446c-96ab-d49d54c97d26" containerName="istio-proxy"
Oct 30 07:21:56.850842 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:56.850706    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="16b1f8ac-8fe8-4dd6-a843-2470652b69ab" containerName="istio-proxy"
Oct 30 07:21:56.850842 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:56.850714    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="a583c68f-99d3-4e7f-ab2c-5b7343cf44b8" containerName="payload-processing"
Oct 30 07:21:56.850842 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:56.850724    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="5583b983-1645-4e19-acb3-8e2f498f6b5b" containerName="istio-proxy"
Oct 30 07:21:56.850842 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:56.850734    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="16b1f8ac-8fe8-4dd6-a843-2470652b69ab" containerName="istio-proxy"
Oct 30 07:21:56.850842 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:56.850743    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="12729552-3e12-4b64-8779-a1584734c16e" containerName="istio-proxy"
Oct 30 07:21:56.850842 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:56.850754    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd9eda1c-1c26-4617-8f10-886e51c2b3e9" containerName="payload-pre-processing"
Oct 30 07:21:56.850842 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:56.850768    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a24e154-72f7-48d6-9d93-2f9fee0de386" containerName="istio-proxy"
Oct 30 07:21:56.855969 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:56.855950    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54"
Oct 30 07:21:56.881334 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:56.881304    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54"]
Oct 30 07:21:56.957964 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:56.957936    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-704bf6-openshift-default-dockercfg-dtdjx\""
Oct 30 07:21:56.990199 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:56.990166    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff3d5b5d-94fe-40f6-baea-d732683fc981-istio-data\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54\" (UID: \"ff3d5b5d-94fe-40f6-baea-d732683fc981\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54"
Oct 30 07:21:56.990341 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:56.990227    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3d5b5d-94fe-40f6-baea-d732683fc981-workload-socket\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54\" (UID: \"ff3d5b5d-94fe-40f6-baea-d732683fc981\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54"
Oct 30 07:21:56.990341 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:56.990326    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff3d5b5d-94fe-40f6-baea-d732683fc981-istio-podinfo\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54\" (UID: \"ff3d5b5d-94fe-40f6-baea-d732683fc981\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54"
Oct 30 07:21:56.990411 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:56.990356    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3d5b5d-94fe-40f6-baea-d732683fc981-credential-socket\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54\" (UID: \"ff3d5b5d-94fe-40f6-baea-d732683fc981\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54"
Oct 30 07:21:56.990411 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:56.990374    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff3d5b5d-94fe-40f6-baea-d732683fc981-workload-certs\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54\" (UID: \"ff3d5b5d-94fe-40f6-baea-d732683fc981\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54"
Oct 30 07:21:56.990509 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:56.990497    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf58f\" (UniqueName: \"kubernetes.io/projected/ff3d5b5d-94fe-40f6-baea-d732683fc981-kube-api-access-pf58f\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54\" (UID: \"ff3d5b5d-94fe-40f6-baea-d732683fc981\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54"
Oct 30 07:21:56.990660 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:56.990614    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff3d5b5d-94fe-40f6-baea-d732683fc981-istio-envoy\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54\" (UID: \"ff3d5b5d-94fe-40f6-baea-d732683fc981\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54"
Oct 30 07:21:56.990699 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:56.990670    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff3d5b5d-94fe-40f6-baea-d732683fc981-istiod-ca-cert\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54\" (UID: \"ff3d5b5d-94fe-40f6-baea-d732683fc981\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54"
Oct 30 07:21:56.990779 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:56.990758    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff3d5b5d-94fe-40f6-baea-d732683fc981-istio-token\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54\" (UID: \"ff3d5b5d-94fe-40f6-baea-d732683fc981\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54"
Oct 30 07:21:57.091560 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:57.091522    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf58f\" (UniqueName: \"kubernetes.io/projected/ff3d5b5d-94fe-40f6-baea-d732683fc981-kube-api-access-pf58f\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54\" (UID: \"ff3d5b5d-94fe-40f6-baea-d732683fc981\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54"
Oct 30 07:21:57.091764 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:57.091628    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff3d5b5d-94fe-40f6-baea-d732683fc981-istio-envoy\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54\" (UID: \"ff3d5b5d-94fe-40f6-baea-d732683fc981\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54"
Oct 30 07:21:57.091764 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:57.091662    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff3d5b5d-94fe-40f6-baea-d732683fc981-istiod-ca-cert\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54\" (UID: \"ff3d5b5d-94fe-40f6-baea-d732683fc981\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54"
Oct 30 07:21:57.091764 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:57.091708    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff3d5b5d-94fe-40f6-baea-d732683fc981-istio-token\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54\" (UID: \"ff3d5b5d-94fe-40f6-baea-d732683fc981\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54"
Oct 30 07:21:57.092004 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:57.091763    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff3d5b5d-94fe-40f6-baea-d732683fc981-istio-data\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54\" (UID: \"ff3d5b5d-94fe-40f6-baea-d732683fc981\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54"
Oct 30 07:21:57.092004 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:57.091868    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3d5b5d-94fe-40f6-baea-d732683fc981-workload-socket\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54\" (UID: \"ff3d5b5d-94fe-40f6-baea-d732683fc981\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54"
Oct 30 07:21:57.092004 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:57.091928    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff3d5b5d-94fe-40f6-baea-d732683fc981-istio-podinfo\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54\" (UID: \"ff3d5b5d-94fe-40f6-baea-d732683fc981\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54"
Oct 30 07:21:57.092004 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:57.091959    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3d5b5d-94fe-40f6-baea-d732683fc981-credential-socket\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54\" (UID: \"ff3d5b5d-94fe-40f6-baea-d732683fc981\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54"
Oct 30 07:21:57.092004 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:57.091985    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff3d5b5d-94fe-40f6-baea-d732683fc981-workload-certs\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54\" (UID: \"ff3d5b5d-94fe-40f6-baea-d732683fc981\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54"
Oct 30 07:21:57.092438 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:57.092415    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff3d5b5d-94fe-40f6-baea-d732683fc981-workload-certs\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54\" (UID: \"ff3d5b5d-94fe-40f6-baea-d732683fc981\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54"
Oct 30 07:21:57.093148 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:57.093123    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff3d5b5d-94fe-40f6-baea-d732683fc981-istio-data\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54\" (UID: \"ff3d5b5d-94fe-40f6-baea-d732683fc981\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54"
Oct 30 07:21:57.093987 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:57.093964    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3d5b5d-94fe-40f6-baea-d732683fc981-workload-socket\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54\" (UID: \"ff3d5b5d-94fe-40f6-baea-d732683fc981\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54"
Oct 30 07:21:57.094301 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:57.094270    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff3d5b5d-94fe-40f6-baea-d732683fc981-istiod-ca-cert\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54\" (UID: \"ff3d5b5d-94fe-40f6-baea-d732683fc981\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54"
Oct 30 07:21:57.094459 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:57.094357    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3d5b5d-94fe-40f6-baea-d732683fc981-credential-socket\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54\" (UID: \"ff3d5b5d-94fe-40f6-baea-d732683fc981\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54"
Oct 30 07:21:57.095665 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:57.095641    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff3d5b5d-94fe-40f6-baea-d732683fc981-istio-envoy\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54\" (UID: \"ff3d5b5d-94fe-40f6-baea-d732683fc981\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54"
Oct 30 07:21:57.096754 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:57.096734    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff3d5b5d-94fe-40f6-baea-d732683fc981-istio-podinfo\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54\" (UID: \"ff3d5b5d-94fe-40f6-baea-d732683fc981\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54"
Oct 30 07:21:57.118295 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:57.118213    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf58f\" (UniqueName: \"kubernetes.io/projected/ff3d5b5d-94fe-40f6-baea-d732683fc981-kube-api-access-pf58f\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54\" (UID: \"ff3d5b5d-94fe-40f6-baea-d732683fc981\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54"
Oct 30 07:21:57.118858 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:57.118827    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff3d5b5d-94fe-40f6-baea-d732683fc981-istio-token\") pod \"e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54\" (UID: \"ff3d5b5d-94fe-40f6-baea-d732683fc981\") " pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54"
Oct 30 07:21:57.170720 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:57.170690    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54"
Oct 30 07:21:57.497706 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:57.497673    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54"]
Oct 30 07:21:57.501109 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:21:57.501081    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff3d5b5d_94fe_40f6_baea_d732683fc981.slice/crio-cc3144efcef98eafae45ff838515c0d0a3e211f8db5c0e9f218a64d5126787d1 WatchSource:0}: Error finding container cc3144efcef98eafae45ff838515c0d0a3e211f8db5c0e9f218a64d5126787d1: Status 404 returned error can't find the container with id cc3144efcef98eafae45ff838515c0d0a3e211f8db5c0e9f218a64d5126787d1
Oct 30 07:21:57.503625 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:57.503592    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:21:57.503732 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:57.503675    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:21:57.503732 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:57.503721    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:21:57.948044 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:57.947999    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54" event={"ID":"ff3d5b5d-94fe-40f6-baea-d732683fc981","Type":"ContainerStarted","Data":"1acc4b74d968fdfb295a0b3741a076b27e4c75eb8162e796b7a859d6c02b93a9"}
Oct 30 07:21:57.948445 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:57.948068    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54" event={"ID":"ff3d5b5d-94fe-40f6-baea-d732683fc981","Type":"ContainerStarted","Data":"cc3144efcef98eafae45ff838515c0d0a3e211f8db5c0e9f218a64d5126787d1"}
Oct 30 07:21:58.012601 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:58.012553    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54" podStartSLOduration=2.012539024 podStartE2EDuration="2.012539024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:21:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:21:58.010231946 +0000 UTC m=+1476.619638942" watchObservedRunningTime="2026-10-30 07:21:58.012539024 +0000 UTC m=+1476.621945994"
Oct 30 07:21:58.174307 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:58.173867    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54"
Oct 30 07:21:58.174307 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:58.173997    6224 patch_prober.go:29] interesting pod/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 30 07:21:58.174307 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:58.174056    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54" podUID="ff3d5b5d-94fe-40f6-baea-d732683fc981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 30 07:21:59.177170 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:59.177133    6224 patch_prober.go:29] interesting pod/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 30 07:21:59.178835 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:21:59.178786    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54" podUID="ff3d5b5d-94fe-40f6-baea-d732683fc981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 30 07:22:00.183831 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:00.180167    6224 patch_prober.go:29] interesting pod/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 30 07:22:00.183831 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:00.180226    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54" podUID="ff3d5b5d-94fe-40f6-baea-d732683fc981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 30 07:22:01.187871 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:01.186173    6224 patch_prober.go:29] interesting pod/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 30 07:22:01.187871 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:01.186249    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54" podUID="ff3d5b5d-94fe-40f6-baea-d732683fc981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 30 07:22:01.858449 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:22:01.858412    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:22:02.176278 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:02.176189    6224 patch_prober.go:29] interesting pod/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 30 07:22:02.176535 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:02.176506    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54" podUID="ff3d5b5d-94fe-40f6-baea-d732683fc981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 30 07:22:03.171191 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:03.171155    6224 patch_prober.go:29] interesting pod/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 30 07:22:03.172331 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:03.171931    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54" podUID="ff3d5b5d-94fe-40f6-baea-d732683fc981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 30 07:22:04.175500 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:04.175466    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54"
Oct 30 07:22:04.175936 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:04.175916    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54"
Oct 30 07:22:04.176729 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:04.176713    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54"
Oct 30 07:22:06.288370 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:06.277091    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:22:11.468967 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:11.468916    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff"]
Oct 30 07:22:11.469398 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:11.469157    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff" podUID="ada682d6-e6ed-4d78-a3cf-07072b35f4f0" containerName="istio-proxy" containerID="cri-o://eb6841701a7999487e6d9a19348bd93ab96f61d1408b3a845db072a1432b47b9" gracePeriod=30
Oct 30 07:22:16.232445 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:16.232405    6224 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 07:22:16.232883 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:16.232475    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff" podUID="ada682d6-e6ed-4d78-a3cf-07072b35f4f0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 07:22:16.722632 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:16.722605    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff"
Oct 30 07:22:16.811825 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:16.811779    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ada682d6-e6ed-4d78-a3cf-07072b35f4f0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ada682d6-e6ed-4d78-a3cf-07072b35f4f0-istio-podinfo\") pod \"ada682d6-e6ed-4d78-a3cf-07072b35f4f0\" (UID: \"ada682d6-e6ed-4d78-a3cf-07072b35f4f0\") "
Oct 30 07:22:16.812021 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:16.811843    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ada682d6-e6ed-4d78-a3cf-07072b35f4f0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ada682d6-e6ed-4d78-a3cf-07072b35f4f0-workload-socket\") pod \"ada682d6-e6ed-4d78-a3cf-07072b35f4f0\" (UID: \"ada682d6-e6ed-4d78-a3cf-07072b35f4f0\") "
Oct 30 07:22:16.812021 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:16.811872    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ada682d6-e6ed-4d78-a3cf-07072b35f4f0-kube-api-access-jwxnx\" (UniqueName: \"kubernetes.io/projected/ada682d6-e6ed-4d78-a3cf-07072b35f4f0-kube-api-access-jwxnx\") pod \"ada682d6-e6ed-4d78-a3cf-07072b35f4f0\" (UID: \"ada682d6-e6ed-4d78-a3cf-07072b35f4f0\") "
Oct 30 07:22:16.812021 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:16.811897    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ada682d6-e6ed-4d78-a3cf-07072b35f4f0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ada682d6-e6ed-4d78-a3cf-07072b35f4f0-istio-data\") pod \"ada682d6-e6ed-4d78-a3cf-07072b35f4f0\" (UID: \"ada682d6-e6ed-4d78-a3cf-07072b35f4f0\") "
Oct 30 07:22:16.812218 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:16.812065    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ada682d6-e6ed-4d78-a3cf-07072b35f4f0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ada682d6-e6ed-4d78-a3cf-07072b35f4f0-credential-socket\") pod \"ada682d6-e6ed-4d78-a3cf-07072b35f4f0\" (UID: \"ada682d6-e6ed-4d78-a3cf-07072b35f4f0\") "
Oct 30 07:22:16.812218 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:16.812124    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ada682d6-e6ed-4d78-a3cf-07072b35f4f0-istio-token\" (UniqueName: \"kubernetes.io/projected/ada682d6-e6ed-4d78-a3cf-07072b35f4f0-istio-token\") pod \"ada682d6-e6ed-4d78-a3cf-07072b35f4f0\" (UID: \"ada682d6-e6ed-4d78-a3cf-07072b35f4f0\") "
Oct 30 07:22:16.812218 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:16.812206    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ada682d6-e6ed-4d78-a3cf-07072b35f4f0-workload-socket" pod "ada682d6-e6ed-4d78-a3cf-07072b35f4f0" (UID: "ada682d6-e6ed-4d78-a3cf-07072b35f4f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:22:16.812382 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:16.812213    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ada682d6-e6ed-4d78-a3cf-07072b35f4f0-istio-data" pod "ada682d6-e6ed-4d78-a3cf-07072b35f4f0" (UID: "ada682d6-e6ed-4d78-a3cf-07072b35f4f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:22:16.812382 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:16.812253    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ada682d6-e6ed-4d78-a3cf-07072b35f4f0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ada682d6-e6ed-4d78-a3cf-07072b35f4f0-istiod-ca-cert\") pod \"ada682d6-e6ed-4d78-a3cf-07072b35f4f0\" (UID: \"ada682d6-e6ed-4d78-a3cf-07072b35f4f0\") "
Oct 30 07:22:16.812382 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:16.812371    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ada682d6-e6ed-4d78-a3cf-07072b35f4f0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ada682d6-e6ed-4d78-a3cf-07072b35f4f0-workload-certs\") pod \"ada682d6-e6ed-4d78-a3cf-07072b35f4f0\" (UID: \"ada682d6-e6ed-4d78-a3cf-07072b35f4f0\") "
Oct 30 07:22:16.812382 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:16.812376    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ada682d6-e6ed-4d78-a3cf-07072b35f4f0-credential-socket" pod "ada682d6-e6ed-4d78-a3cf-07072b35f4f0" (UID: "ada682d6-e6ed-4d78-a3cf-07072b35f4f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:22:16.812595 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:16.812443    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ada682d6-e6ed-4d78-a3cf-07072b35f4f0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ada682d6-e6ed-4d78-a3cf-07072b35f4f0-istio-envoy\") pod \"ada682d6-e6ed-4d78-a3cf-07072b35f4f0\" (UID: \"ada682d6-e6ed-4d78-a3cf-07072b35f4f0\") "
Oct 30 07:22:16.812681 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:16.812641    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ada682d6-e6ed-4d78-a3cf-07072b35f4f0-workload-certs" pod "ada682d6-e6ed-4d78-a3cf-07072b35f4f0" (UID: "ada682d6-e6ed-4d78-a3cf-07072b35f4f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:22:16.812744 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:16.812687    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ada682d6-e6ed-4d78-a3cf-07072b35f4f0-istiod-ca-cert" pod "ada682d6-e6ed-4d78-a3cf-07072b35f4f0" (UID: "ada682d6-e6ed-4d78-a3cf-07072b35f4f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:22:16.813205 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:16.813075    6224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ada682d6-e6ed-4d78-a3cf-07072b35f4f0-workload-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:22:16.813205 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:16.813103    6224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ada682d6-e6ed-4d78-a3cf-07072b35f4f0-workload-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:22:16.813205 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:16.813117    6224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ada682d6-e6ed-4d78-a3cf-07072b35f4f0-istio-data\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:22:16.813205 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:16.813132    6224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ada682d6-e6ed-4d78-a3cf-07072b35f4f0-credential-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:22:16.813205 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:16.813146    6224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ada682d6-e6ed-4d78-a3cf-07072b35f4f0-istiod-ca-cert\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:22:16.814117 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:16.814093    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ada682d6-e6ed-4d78-a3cf-07072b35f4f0-istio-podinfo" pod "ada682d6-e6ed-4d78-a3cf-07072b35f4f0" (UID: "ada682d6-e6ed-4d78-a3cf-07072b35f4f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:22:16.814730 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:16.814706    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ada682d6-e6ed-4d78-a3cf-07072b35f4f0-istio-token" pod "ada682d6-e6ed-4d78-a3cf-07072b35f4f0" (UID: "ada682d6-e6ed-4d78-a3cf-07072b35f4f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:22:16.814861 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:16.814783    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ada682d6-e6ed-4d78-a3cf-07072b35f4f0-kube-api-access-jwxnx" pod "ada682d6-e6ed-4d78-a3cf-07072b35f4f0" (UID: "ada682d6-e6ed-4d78-a3cf-07072b35f4f0"). InnerVolumeSpecName "kube-api-access-jwxnx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:22:16.814998 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:16.814982    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ada682d6-e6ed-4d78-a3cf-07072b35f4f0-istio-envoy" pod "ada682d6-e6ed-4d78-a3cf-07072b35f4f0" (UID: "ada682d6-e6ed-4d78-a3cf-07072b35f4f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:22:16.914520 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:16.914491    6224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ada682d6-e6ed-4d78-a3cf-07072b35f4f0-istio-envoy\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:22:16.914520 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:16.914518    6224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ada682d6-e6ed-4d78-a3cf-07072b35f4f0-istio-podinfo\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:22:16.914520 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:16.914528    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwxnx\" (UniqueName: \"kubernetes.io/projected/ada682d6-e6ed-4d78-a3cf-07072b35f4f0-kube-api-access-jwxnx\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:22:16.914761 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:16.914537    6224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ada682d6-e6ed-4d78-a3cf-07072b35f4f0-istio-token\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:22:17.124989 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:17.124856    6224 generic.go:356] "Generic (PLEG): container finished" podID="ada682d6-e6ed-4d78-a3cf-07072b35f4f0" containerID="eb6841701a7999487e6d9a19348bd93ab96f61d1408b3a845db072a1432b47b9" exitCode=0
Oct 30 07:22:17.124989 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:17.124910    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff"
Oct 30 07:22:17.124989 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:17.124952    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff" event={"ID":"ada682d6-e6ed-4d78-a3cf-07072b35f4f0","Type":"ContainerDied","Data":"eb6841701a7999487e6d9a19348bd93ab96f61d1408b3a845db072a1432b47b9"}
Oct 30 07:22:17.125298 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:17.125024    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff" event={"ID":"ada682d6-e6ed-4d78-a3cf-07072b35f4f0","Type":"ContainerDied","Data":"a7b678ebcf716ae709c5e480e0d392307606b92031864849ceddbfb43ea465e7"}
Oct 30 07:22:17.125298 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:17.125061    6224 scope.go:122] "RemoveContainer" containerID="eb6841701a7999487e6d9a19348bd93ab96f61d1408b3a845db072a1432b47b9"
Oct 30 07:22:17.148206 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:17.148119    6224 scope.go:122] "RemoveContainer" containerID="eb6841701a7999487e6d9a19348bd93ab96f61d1408b3a845db072a1432b47b9"
Oct 30 07:22:17.148614 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:22:17.148581    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb6841701a7999487e6d9a19348bd93ab96f61d1408b3a845db072a1432b47b9\": container with ID starting with eb6841701a7999487e6d9a19348bd93ab96f61d1408b3a845db072a1432b47b9 not found: ID does not exist" containerID="eb6841701a7999487e6d9a19348bd93ab96f61d1408b3a845db072a1432b47b9"
Oct 30 07:22:17.148721 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:17.148626    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb6841701a7999487e6d9a19348bd93ab96f61d1408b3a845db072a1432b47b9"} err="failed to get container status \"eb6841701a7999487e6d9a19348bd93ab96f61d1408b3a845db072a1432b47b9\": rpc error: code = NotFound desc = could not find container \"eb6841701a7999487e6d9a19348bd93ab96f61d1408b3a845db072a1432b47b9\": container with ID starting with eb6841701a7999487e6d9a19348bd93ab96f61d1408b3a845db072a1432b47b9 not found: ID does not exist"
Oct 30 07:22:17.189354 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:17.189323    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff"]
Oct 30 07:22:17.196407 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:17.196376    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-r8vff"]
Oct 30 07:22:17.505247 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:17.505163    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ada682d6-e6ed-4d78-a3cf-07072b35f4f0" path="/var/lib/kubelet/pods/ada682d6-e6ed-4d78-a3cf-07072b35f4f0/volumes"
Oct 30 07:22:30.272630 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:30.272593    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9"]
Oct 30 07:22:30.273076 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:30.272821    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9" podUID="44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7" containerName="istio-proxy" containerID="cri-o://e8fa1e378c61c7febe12a7df2a4d245e543c7f570a095ae4007565ffe764ed56" gracePeriod=30
Oct 30 07:22:30.911655 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:30.911622    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq"]
Oct 30 07:22:30.911957 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:30.911928    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq" podUID="6fd8ca77-b283-4d93-87e7-640397564926" containerName="istio-proxy" containerID="cri-o://5cf20a309dcf4192b17baccb1b3a184451f1bdb69d4d9e8cba8b494512feda25" gracePeriod=30
Oct 30 07:22:31.865295 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:22:31.865252    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:22:35.538857 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:35.538734    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9"
Oct 30 07:22:35.730376 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:35.730329    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7-kube-api-access-lxlxg\" (UniqueName: \"kubernetes.io/projected/44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7-kube-api-access-lxlxg\") pod \"44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7\" (UID: \"44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7\") "
Oct 30 07:22:35.730581 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:35.730421    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7-istiod-ca-cert\") pod \"44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7\" (UID: \"44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7\") "
Oct 30 07:22:35.730581 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:35.730467    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7-istio-token\" (UniqueName: \"kubernetes.io/projected/44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7-istio-token\") pod \"44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7\" (UID: \"44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7\") "
Oct 30 07:22:35.730581 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:35.730494    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7-credential-socket\") pod \"44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7\" (UID: \"44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7\") "
Oct 30 07:22:35.730581 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:35.730518    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7-workload-socket\") pod \"44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7\" (UID: \"44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7\") "
Oct 30 07:22:35.730581 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:35.730583    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7-istio-podinfo\") pod \"44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7\" (UID: \"44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7\") "
Oct 30 07:22:35.730899 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:35.730612    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7-istio-envoy\") pod \"44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7\" (UID: \"44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7\") "
Oct 30 07:22:35.730899 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:35.730649    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7-istio-data\") pod \"44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7\" (UID: \"44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7\") "
Oct 30 07:22:35.730899 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:35.730683    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7-workload-certs\") pod \"44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7\" (UID: \"44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7\") "
Oct 30 07:22:35.730899 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:35.730799    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7-credential-socket" pod "44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7" (UID: "44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:22:35.731107 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:35.730905    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7-istiod-ca-cert" pod "44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7" (UID: "44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:22:35.731107 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:35.730912    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7-workload-socket" pod "44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7" (UID: "44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:22:35.731107 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:35.731078    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7-workload-certs" pod "44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7" (UID: "44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:22:35.731269 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:35.731161    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7-istio-data" pod "44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7" (UID: "44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:22:35.731391 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:35.731371    6224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7-istiod-ca-cert\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:22:35.731391 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:35.731391    6224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7-credential-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:22:35.731549 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:35.731404    6224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7-workload-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:22:35.731549 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:35.731417    6224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7-istio-data\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:22:35.731549 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:35.731428    6224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7-workload-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:22:35.732760 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:35.732733    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7-istio-podinfo" pod "44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7" (UID: "44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:22:35.732949 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:35.732930    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7-kube-api-access-lxlxg" pod "44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7" (UID: "44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7"). InnerVolumeSpecName "kube-api-access-lxlxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:22:35.733246 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:35.733218    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7-istio-envoy" pod "44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7" (UID: "44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:22:35.733246 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:35.733224    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7-istio-token" pod "44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7" (UID: "44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:22:35.833420 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:35.833184    6224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7-istio-envoy\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:22:35.833420 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:35.833283    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxlxg\" (UniqueName: \"kubernetes.io/projected/44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7-kube-api-access-lxlxg\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:22:35.833420 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:35.833301    6224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7-istio-token\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:22:35.833420 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:35.833315    6224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7-istio-podinfo\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:22:36.166784 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:36.166754    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq"
Oct 30 07:22:36.238127 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:36.238097    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fd8ca77-b283-4d93-87e7-640397564926-istio-token\" (UniqueName: \"kubernetes.io/projected/6fd8ca77-b283-4d93-87e7-640397564926-istio-token\") pod \"6fd8ca77-b283-4d93-87e7-640397564926\" (UID: \"6fd8ca77-b283-4d93-87e7-640397564926\") "
Oct 30 07:22:36.238300 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:36.238157    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fd8ca77-b283-4d93-87e7-640397564926-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fd8ca77-b283-4d93-87e7-640397564926-istio-envoy\") pod \"6fd8ca77-b283-4d93-87e7-640397564926\" (UID: \"6fd8ca77-b283-4d93-87e7-640397564926\") "
Oct 30 07:22:36.238300 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:36.238210    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fd8ca77-b283-4d93-87e7-640397564926-kube-api-access-97nfs\" (UniqueName: \"kubernetes.io/projected/6fd8ca77-b283-4d93-87e7-640397564926-kube-api-access-97nfs\") pod \"6fd8ca77-b283-4d93-87e7-640397564926\" (UID: \"6fd8ca77-b283-4d93-87e7-640397564926\") "
Oct 30 07:22:36.238300 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:36.238258    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fd8ca77-b283-4d93-87e7-640397564926-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fd8ca77-b283-4d93-87e7-640397564926-credential-socket\") pod \"6fd8ca77-b283-4d93-87e7-640397564926\" (UID: \"6fd8ca77-b283-4d93-87e7-640397564926\") "
Oct 30 07:22:36.238300 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:36.238294    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fd8ca77-b283-4d93-87e7-640397564926-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fd8ca77-b283-4d93-87e7-640397564926-istio-data\") pod \"6fd8ca77-b283-4d93-87e7-640397564926\" (UID: \"6fd8ca77-b283-4d93-87e7-640397564926\") "
Oct 30 07:22:36.239511 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:36.238333    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6fd8ca77-b283-4d93-87e7-640397564926-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fd8ca77-b283-4d93-87e7-640397564926-istio-podinfo\") pod \"6fd8ca77-b283-4d93-87e7-640397564926\" (UID: \"6fd8ca77-b283-4d93-87e7-640397564926\") "
Oct 30 07:22:36.239511 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:36.238384    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fd8ca77-b283-4d93-87e7-640397564926-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fd8ca77-b283-4d93-87e7-640397564926-workload-certs\") pod \"6fd8ca77-b283-4d93-87e7-640397564926\" (UID: \"6fd8ca77-b283-4d93-87e7-640397564926\") "
Oct 30 07:22:36.239511 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:36.238434    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fd8ca77-b283-4d93-87e7-640397564926-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fd8ca77-b283-4d93-87e7-640397564926-istiod-ca-cert\") pod \"6fd8ca77-b283-4d93-87e7-640397564926\" (UID: \"6fd8ca77-b283-4d93-87e7-640397564926\") "
Oct 30 07:22:36.239511 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:36.238474    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fd8ca77-b283-4d93-87e7-640397564926-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fd8ca77-b283-4d93-87e7-640397564926-workload-socket\") pod \"6fd8ca77-b283-4d93-87e7-640397564926\" (UID: \"6fd8ca77-b283-4d93-87e7-640397564926\") "
Oct 30 07:22:36.239511 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:36.238611    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fd8ca77-b283-4d93-87e7-640397564926-credential-socket" pod "6fd8ca77-b283-4d93-87e7-640397564926" (UID: "6fd8ca77-b283-4d93-87e7-640397564926"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:22:36.239511 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:36.238728    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fd8ca77-b283-4d93-87e7-640397564926-workload-certs" pod "6fd8ca77-b283-4d93-87e7-640397564926" (UID: "6fd8ca77-b283-4d93-87e7-640397564926"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:22:36.239511 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:36.238838    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fd8ca77-b283-4d93-87e7-640397564926-istiod-ca-cert" pod "6fd8ca77-b283-4d93-87e7-640397564926" (UID: "6fd8ca77-b283-4d93-87e7-640397564926"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:22:36.239511 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:36.238931    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fd8ca77-b283-4d93-87e7-640397564926-workload-socket" pod "6fd8ca77-b283-4d93-87e7-640397564926" (UID: "6fd8ca77-b283-4d93-87e7-640397564926"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:22:36.239511 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:36.239122    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fd8ca77-b283-4d93-87e7-640397564926-istio-data" pod "6fd8ca77-b283-4d93-87e7-640397564926" (UID: "6fd8ca77-b283-4d93-87e7-640397564926"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:22:36.240095 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:36.239707    6224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fd8ca77-b283-4d93-87e7-640397564926-credential-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:22:36.240095 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:36.239725    6224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fd8ca77-b283-4d93-87e7-640397564926-istio-data\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:22:36.240095 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:36.239738    6224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fd8ca77-b283-4d93-87e7-640397564926-workload-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:22:36.240095 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:36.239746    6224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fd8ca77-b283-4d93-87e7-640397564926-istiod-ca-cert\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:22:36.240095 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:36.239755    6224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fd8ca77-b283-4d93-87e7-640397564926-workload-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:22:36.241289 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:36.241246    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6fd8ca77-b283-4d93-87e7-640397564926-istio-podinfo" pod "6fd8ca77-b283-4d93-87e7-640397564926" (UID: "6fd8ca77-b283-4d93-87e7-640397564926"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:22:36.242606 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:36.242584    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fd8ca77-b283-4d93-87e7-640397564926-istio-token" pod "6fd8ca77-b283-4d93-87e7-640397564926" (UID: "6fd8ca77-b283-4d93-87e7-640397564926"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:22:36.251853 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:36.245563    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fd8ca77-b283-4d93-87e7-640397564926-kube-api-access-97nfs" pod "6fd8ca77-b283-4d93-87e7-640397564926" (UID: "6fd8ca77-b283-4d93-87e7-640397564926"). InnerVolumeSpecName "kube-api-access-97nfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:22:36.251853 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:36.247497    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fd8ca77-b283-4d93-87e7-640397564926-istio-envoy" pod "6fd8ca77-b283-4d93-87e7-640397564926" (UID: "6fd8ca77-b283-4d93-87e7-640397564926"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:22:36.251853 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:36.247996    6224 generic.go:356] "Generic (PLEG): container finished" podID="6fd8ca77-b283-4d93-87e7-640397564926" containerID="5cf20a309dcf4192b17baccb1b3a184451f1bdb69d4d9e8cba8b494512feda25" exitCode=0
Oct 30 07:22:36.251853 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:36.248090    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq"
Oct 30 07:22:36.251853 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:36.248121    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq" event={"ID":"6fd8ca77-b283-4d93-87e7-640397564926","Type":"ContainerDied","Data":"5cf20a309dcf4192b17baccb1b3a184451f1bdb69d4d9e8cba8b494512feda25"}
Oct 30 07:22:36.251853 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:36.248182    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq" event={"ID":"6fd8ca77-b283-4d93-87e7-640397564926","Type":"ContainerDied","Data":"5567ca82082345865da633f31106792bc0e11fc5d5fe03b986ec1e81bbb89b73"}
Oct 30 07:22:36.251853 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:36.248213    6224 scope.go:122] "RemoveContainer" containerID="5cf20a309dcf4192b17baccb1b3a184451f1bdb69d4d9e8cba8b494512feda25"
Oct 30 07:22:36.251853 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:36.250122    6224 generic.go:356] "Generic (PLEG): container finished" podID="44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7" containerID="e8fa1e378c61c7febe12a7df2a4d245e543c7f570a095ae4007565ffe764ed56" exitCode=0
Oct 30 07:22:36.251853 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:36.250233    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9" event={"ID":"44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7","Type":"ContainerDied","Data":"e8fa1e378c61c7febe12a7df2a4d245e543c7f570a095ae4007565ffe764ed56"}
Oct 30 07:22:36.251853 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:36.250274    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9" event={"ID":"44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7","Type":"ContainerDied","Data":"c1a03e6201d4cb9b6aaadf70b2cca37c0b78e21f238eeba49e677e39dc29a3ce"}
Oct 30 07:22:36.251853 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:36.250344    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9"
Oct 30 07:22:36.260407 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:36.260388    6224 scope.go:122] "RemoveContainer" containerID="5cf20a309dcf4192b17baccb1b3a184451f1bdb69d4d9e8cba8b494512feda25"
Oct 30 07:22:36.260673 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:22:36.260653    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cf20a309dcf4192b17baccb1b3a184451f1bdb69d4d9e8cba8b494512feda25\": container with ID starting with 5cf20a309dcf4192b17baccb1b3a184451f1bdb69d4d9e8cba8b494512feda25 not found: ID does not exist" containerID="5cf20a309dcf4192b17baccb1b3a184451f1bdb69d4d9e8cba8b494512feda25"
Oct 30 07:22:36.260753 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:36.260683    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cf20a309dcf4192b17baccb1b3a184451f1bdb69d4d9e8cba8b494512feda25"} err="failed to get container status \"5cf20a309dcf4192b17baccb1b3a184451f1bdb69d4d9e8cba8b494512feda25\": rpc error: code = NotFound desc = could not find container \"5cf20a309dcf4192b17baccb1b3a184451f1bdb69d4d9e8cba8b494512feda25\": container with ID starting with 5cf20a309dcf4192b17baccb1b3a184451f1bdb69d4d9e8cba8b494512feda25 not found: ID does not exist"
Oct 30 07:22:36.260753 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:36.260699    6224 scope.go:122] "RemoveContainer" containerID="e8fa1e378c61c7febe12a7df2a4d245e543c7f570a095ae4007565ffe764ed56"
Oct 30 07:22:36.271652 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:36.271632    6224 scope.go:122] "RemoveContainer" containerID="e8fa1e378c61c7febe12a7df2a4d245e543c7f570a095ae4007565ffe764ed56"
Oct 30 07:22:36.271953 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:22:36.271934    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8fa1e378c61c7febe12a7df2a4d245e543c7f570a095ae4007565ffe764ed56\": container with ID starting with e8fa1e378c61c7febe12a7df2a4d245e543c7f570a095ae4007565ffe764ed56 not found: ID does not exist" containerID="e8fa1e378c61c7febe12a7df2a4d245e543c7f570a095ae4007565ffe764ed56"
Oct 30 07:22:36.272023 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:36.271961    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8fa1e378c61c7febe12a7df2a4d245e543c7f570a095ae4007565ffe764ed56"} err="failed to get container status \"e8fa1e378c61c7febe12a7df2a4d245e543c7f570a095ae4007565ffe764ed56\": rpc error: code = NotFound desc = could not find container \"e8fa1e378c61c7febe12a7df2a4d245e543c7f570a095ae4007565ffe764ed56\": container with ID starting with e8fa1e378c61c7febe12a7df2a4d245e543c7f570a095ae4007565ffe764ed56 not found: ID does not exist"
Oct 30 07:22:36.318364 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:36.318340    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq"]
Oct 30 07:22:36.329492 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:36.329469    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-mm4sq"]
Oct 30 07:22:36.340719 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:36.340663    6224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fd8ca77-b283-4d93-87e7-640397564926-istio-token\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:22:36.340719 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:36.340693    6224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fd8ca77-b283-4d93-87e7-640397564926-istio-envoy\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:22:36.340719 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:36.340705    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97nfs\" (UniqueName: \"kubernetes.io/projected/6fd8ca77-b283-4d93-87e7-640397564926-kube-api-access-97nfs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:22:36.340719 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:36.340717    6224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fd8ca77-b283-4d93-87e7-640397564926-istio-podinfo\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:22:36.465832 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:36.465793    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9"]
Oct 30 07:22:36.483198 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:36.483170    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-h8hz9"]
Oct 30 07:22:37.249831 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:37.249768    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:22:37.389397 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:37.389365    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth"]
Oct 30 07:22:37.389613 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:37.389586    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth" podUID="9ec378e4-d80e-4306-b1b8-8556b80ce58d" containerName="istio-proxy" containerID="cri-o://4b48742d8237c38f3e906322f7cc80c9b2e8555465da374f092a99f67af591b4" gracePeriod=30
Oct 30 07:22:37.508858 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:37.508743    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7" path="/var/lib/kubelet/pods/44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7/volumes"
Oct 30 07:22:37.509352 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:37.509335    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fd8ca77-b283-4d93-87e7-640397564926" path="/var/lib/kubelet/pods/6fd8ca77-b283-4d93-87e7-640397564926/volumes"
Oct 30 07:22:42.659470 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:42.659412    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth"
Oct 30 07:22:42.727366 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:42.727333    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ec378e4-d80e-4306-b1b8-8556b80ce58d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ec378e4-d80e-4306-b1b8-8556b80ce58d-istio-envoy\") pod \"9ec378e4-d80e-4306-b1b8-8556b80ce58d\" (UID: \"9ec378e4-d80e-4306-b1b8-8556b80ce58d\") "
Oct 30 07:22:42.727519 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:42.727376    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9ec378e4-d80e-4306-b1b8-8556b80ce58d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ec378e4-d80e-4306-b1b8-8556b80ce58d-istio-podinfo\") pod \"9ec378e4-d80e-4306-b1b8-8556b80ce58d\" (UID: \"9ec378e4-d80e-4306-b1b8-8556b80ce58d\") "
Oct 30 07:22:42.727585 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:42.727540    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ec378e4-d80e-4306-b1b8-8556b80ce58d-kube-api-access-48chc\" (UniqueName: \"kubernetes.io/projected/9ec378e4-d80e-4306-b1b8-8556b80ce58d-kube-api-access-48chc\") pod \"9ec378e4-d80e-4306-b1b8-8556b80ce58d\" (UID: \"9ec378e4-d80e-4306-b1b8-8556b80ce58d\") "
Oct 30 07:22:42.727656 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:42.727617    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ec378e4-d80e-4306-b1b8-8556b80ce58d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec378e4-d80e-4306-b1b8-8556b80ce58d-workload-socket\") pod \"9ec378e4-d80e-4306-b1b8-8556b80ce58d\" (UID: \"9ec378e4-d80e-4306-b1b8-8556b80ce58d\") "
Oct 30 07:22:42.727709 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:42.727658    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ec378e4-d80e-4306-b1b8-8556b80ce58d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec378e4-d80e-4306-b1b8-8556b80ce58d-credential-socket\") pod \"9ec378e4-d80e-4306-b1b8-8556b80ce58d\" (UID: \"9ec378e4-d80e-4306-b1b8-8556b80ce58d\") "
Oct 30 07:22:42.727709 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:42.727690    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ec378e4-d80e-4306-b1b8-8556b80ce58d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ec378e4-d80e-4306-b1b8-8556b80ce58d-istio-data\") pod \"9ec378e4-d80e-4306-b1b8-8556b80ce58d\" (UID: \"9ec378e4-d80e-4306-b1b8-8556b80ce58d\") "
Oct 30 07:22:42.727835 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:42.727740    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ec378e4-d80e-4306-b1b8-8556b80ce58d-istio-token\" (UniqueName: \"kubernetes.io/projected/9ec378e4-d80e-4306-b1b8-8556b80ce58d-istio-token\") pod \"9ec378e4-d80e-4306-b1b8-8556b80ce58d\" (UID: \"9ec378e4-d80e-4306-b1b8-8556b80ce58d\") "
Oct 30 07:22:42.727835 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:42.727791    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ec378e4-d80e-4306-b1b8-8556b80ce58d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ec378e4-d80e-4306-b1b8-8556b80ce58d-workload-certs\") pod \"9ec378e4-d80e-4306-b1b8-8556b80ce58d\" (UID: \"9ec378e4-d80e-4306-b1b8-8556b80ce58d\") "
Oct 30 07:22:42.727953 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:42.727856    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ec378e4-d80e-4306-b1b8-8556b80ce58d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ec378e4-d80e-4306-b1b8-8556b80ce58d-istiod-ca-cert\") pod \"9ec378e4-d80e-4306-b1b8-8556b80ce58d\" (UID: \"9ec378e4-d80e-4306-b1b8-8556b80ce58d\") "
Oct 30 07:22:42.728076 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:42.728047    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ec378e4-d80e-4306-b1b8-8556b80ce58d-workload-socket" pod "9ec378e4-d80e-4306-b1b8-8556b80ce58d" (UID: "9ec378e4-d80e-4306-b1b8-8556b80ce58d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:22:42.728301 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:42.728274    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ec378e4-d80e-4306-b1b8-8556b80ce58d-credential-socket" pod "9ec378e4-d80e-4306-b1b8-8556b80ce58d" (UID: "9ec378e4-d80e-4306-b1b8-8556b80ce58d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:22:42.728395 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:42.728293    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ec378e4-d80e-4306-b1b8-8556b80ce58d-workload-certs" pod "9ec378e4-d80e-4306-b1b8-8556b80ce58d" (UID: "9ec378e4-d80e-4306-b1b8-8556b80ce58d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:22:42.728395 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:42.728385    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ec378e4-d80e-4306-b1b8-8556b80ce58d-istiod-ca-cert" pod "9ec378e4-d80e-4306-b1b8-8556b80ce58d" (UID: "9ec378e4-d80e-4306-b1b8-8556b80ce58d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:22:42.728666 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:42.728622    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ec378e4-d80e-4306-b1b8-8556b80ce58d-istio-data" pod "9ec378e4-d80e-4306-b1b8-8556b80ce58d" (UID: "9ec378e4-d80e-4306-b1b8-8556b80ce58d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:22:42.728864 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:42.728843    6224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec378e4-d80e-4306-b1b8-8556b80ce58d-workload-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:22:42.728944 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:42.728871    6224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec378e4-d80e-4306-b1b8-8556b80ce58d-credential-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:22:42.728944 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:42.728885    6224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ec378e4-d80e-4306-b1b8-8556b80ce58d-istio-data\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:22:42.728944 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:42.728898    6224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ec378e4-d80e-4306-b1b8-8556b80ce58d-workload-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:22:42.728944 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:42.728911    6224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ec378e4-d80e-4306-b1b8-8556b80ce58d-istiod-ca-cert\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:22:42.730127 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:42.730099    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ec378e4-d80e-4306-b1b8-8556b80ce58d-istio-envoy" pod "9ec378e4-d80e-4306-b1b8-8556b80ce58d" (UID: "9ec378e4-d80e-4306-b1b8-8556b80ce58d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:22:42.730252 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:42.730121    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ec378e4-d80e-4306-b1b8-8556b80ce58d-istio-podinfo" pod "9ec378e4-d80e-4306-b1b8-8556b80ce58d" (UID: "9ec378e4-d80e-4306-b1b8-8556b80ce58d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:22:42.730526 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:42.730502    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ec378e4-d80e-4306-b1b8-8556b80ce58d-kube-api-access-48chc" pod "9ec378e4-d80e-4306-b1b8-8556b80ce58d" (UID: "9ec378e4-d80e-4306-b1b8-8556b80ce58d"). InnerVolumeSpecName "kube-api-access-48chc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:22:42.730725 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:42.730703    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ec378e4-d80e-4306-b1b8-8556b80ce58d-istio-token" pod "9ec378e4-d80e-4306-b1b8-8556b80ce58d" (UID: "9ec378e4-d80e-4306-b1b8-8556b80ce58d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:22:42.830655 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:42.830606    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48chc\" (UniqueName: \"kubernetes.io/projected/9ec378e4-d80e-4306-b1b8-8556b80ce58d-kube-api-access-48chc\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:22:42.830655 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:42.830645    6224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ec378e4-d80e-4306-b1b8-8556b80ce58d-istio-token\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:22:42.830655 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:42.830659    6224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ec378e4-d80e-4306-b1b8-8556b80ce58d-istio-envoy\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:22:42.830932 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:42.830670    6224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ec378e4-d80e-4306-b1b8-8556b80ce58d-istio-podinfo\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:22:43.322116 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:43.322082    6224 generic.go:356] "Generic (PLEG): container finished" podID="9ec378e4-d80e-4306-b1b8-8556b80ce58d" containerID="4b48742d8237c38f3e906322f7cc80c9b2e8555465da374f092a99f67af591b4" exitCode=0
Oct 30 07:22:43.322320 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:43.322128    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth"
Oct 30 07:22:43.322320 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:43.322162    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth" event={"ID":"9ec378e4-d80e-4306-b1b8-8556b80ce58d","Type":"ContainerDied","Data":"4b48742d8237c38f3e906322f7cc80c9b2e8555465da374f092a99f67af591b4"}
Oct 30 07:22:43.322320 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:43.322216    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth" event={"ID":"9ec378e4-d80e-4306-b1b8-8556b80ce58d","Type":"ContainerDied","Data":"aa825ccfbb09388ab42e096ec47df174f4b1bb3f14fb16ceac90bd38b6cb118e"}
Oct 30 07:22:43.322320 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:43.322239    6224 scope.go:122] "RemoveContainer" containerID="4b48742d8237c38f3e906322f7cc80c9b2e8555465da374f092a99f67af591b4"
Oct 30 07:22:43.332933 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:43.332914    6224 scope.go:122] "RemoveContainer" containerID="4b48742d8237c38f3e906322f7cc80c9b2e8555465da374f092a99f67af591b4"
Oct 30 07:22:43.333216 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:22:43.333197    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b48742d8237c38f3e906322f7cc80c9b2e8555465da374f092a99f67af591b4\": container with ID starting with 4b48742d8237c38f3e906322f7cc80c9b2e8555465da374f092a99f67af591b4 not found: ID does not exist" containerID="4b48742d8237c38f3e906322f7cc80c9b2e8555465da374f092a99f67af591b4"
Oct 30 07:22:43.333283 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:43.333226    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b48742d8237c38f3e906322f7cc80c9b2e8555465da374f092a99f67af591b4"} err="failed to get container status \"4b48742d8237c38f3e906322f7cc80c9b2e8555465da374f092a99f67af591b4\": rpc error: code = NotFound desc = could not find container \"4b48742d8237c38f3e906322f7cc80c9b2e8555465da374f092a99f67af591b4\": container with ID starting with 4b48742d8237c38f3e906322f7cc80c9b2e8555465da374f092a99f67af591b4 not found: ID does not exist"
Oct 30 07:22:43.394097 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:43.394072    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth"]
Oct 30 07:22:43.408428 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:43.408399    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-d7zth"]
Oct 30 07:22:43.503285 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:22:43.503263    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ec378e4-d80e-4306-b1b8-8556b80ce58d" path="/var/lib/kubelet/pods/9ec378e4-d80e-4306-b1b8-8556b80ce58d/volumes"
Oct 30 07:23:01.857772 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:23:01.857726    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:23:02.177887 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:02.177104    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54"]
Oct 30 07:23:02.177887 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:02.177341    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54" podUID="ff3d5b5d-94fe-40f6-baea-d732683fc981" containerName="istio-proxy" containerID="cri-o://1acc4b74d968fdfb295a0b3741a076b27e4c75eb8162e796b7a859d6c02b93a9" gracePeriod=30
Oct 30 07:23:04.177712 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:04.177628    6224 patch_prober.go:29] interesting pod/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 07:23:04.177712 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:04.177695    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54" podUID="ff3d5b5d-94fe-40f6-baea-d732683fc981" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 07:23:07.455190 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:07.455130    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54"
Oct 30 07:23:07.506270 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:07.506239    6224 generic.go:356] "Generic (PLEG): container finished" podID="ff3d5b5d-94fe-40f6-baea-d732683fc981" containerID="1acc4b74d968fdfb295a0b3741a076b27e4c75eb8162e796b7a859d6c02b93a9" exitCode=0
Oct 30 07:23:07.506425 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:07.506280    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54"
Oct 30 07:23:07.506425 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:07.506309    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54" event={"ID":"ff3d5b5d-94fe-40f6-baea-d732683fc981","Type":"ContainerDied","Data":"1acc4b74d968fdfb295a0b3741a076b27e4c75eb8162e796b7a859d6c02b93a9"}
Oct 30 07:23:07.506425 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:07.506351    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54" event={"ID":"ff3d5b5d-94fe-40f6-baea-d732683fc981","Type":"ContainerDied","Data":"cc3144efcef98eafae45ff838515c0d0a3e211f8db5c0e9f218a64d5126787d1"}
Oct 30 07:23:07.506425 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:07.506380    6224 scope.go:122] "RemoveContainer" containerID="1acc4b74d968fdfb295a0b3741a076b27e4c75eb8162e796b7a859d6c02b93a9"
Oct 30 07:23:07.516357 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:07.516322    6224 scope.go:122] "RemoveContainer" containerID="1acc4b74d968fdfb295a0b3741a076b27e4c75eb8162e796b7a859d6c02b93a9"
Oct 30 07:23:07.516945 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:23:07.516914    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1acc4b74d968fdfb295a0b3741a076b27e4c75eb8162e796b7a859d6c02b93a9\": container with ID starting with 1acc4b74d968fdfb295a0b3741a076b27e4c75eb8162e796b7a859d6c02b93a9 not found: ID does not exist" containerID="1acc4b74d968fdfb295a0b3741a076b27e4c75eb8162e796b7a859d6c02b93a9"
Oct 30 07:23:07.516945 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:07.516955    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1acc4b74d968fdfb295a0b3741a076b27e4c75eb8162e796b7a859d6c02b93a9"} err="failed to get container status \"1acc4b74d968fdfb295a0b3741a076b27e4c75eb8162e796b7a859d6c02b93a9\": rpc error: code = NotFound desc = could not find container \"1acc4b74d968fdfb295a0b3741a076b27e4c75eb8162e796b7a859d6c02b93a9\": container with ID starting with 1acc4b74d968fdfb295a0b3741a076b27e4c75eb8162e796b7a859d6c02b93a9 not found: ID does not exist"
Oct 30 07:23:07.594760 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:07.594714    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff3d5b5d-94fe-40f6-baea-d732683fc981-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3d5b5d-94fe-40f6-baea-d732683fc981-credential-socket\") pod \"ff3d5b5d-94fe-40f6-baea-d732683fc981\" (UID: \"ff3d5b5d-94fe-40f6-baea-d732683fc981\") "
Oct 30 07:23:07.594977 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:07.594780    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff3d5b5d-94fe-40f6-baea-d732683fc981-istio-token\" (UniqueName: \"kubernetes.io/projected/ff3d5b5d-94fe-40f6-baea-d732683fc981-istio-token\") pod \"ff3d5b5d-94fe-40f6-baea-d732683fc981\" (UID: \"ff3d5b5d-94fe-40f6-baea-d732683fc981\") "
Oct 30 07:23:07.594977 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:07.594883    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff3d5b5d-94fe-40f6-baea-d732683fc981-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff3d5b5d-94fe-40f6-baea-d732683fc981-workload-certs\") pod \"ff3d5b5d-94fe-40f6-baea-d732683fc981\" (UID: \"ff3d5b5d-94fe-40f6-baea-d732683fc981\") "
Oct 30 07:23:07.594977 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:07.594952    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff3d5b5d-94fe-40f6-baea-d732683fc981-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff3d5b5d-94fe-40f6-baea-d732683fc981-istio-envoy\") pod \"ff3d5b5d-94fe-40f6-baea-d732683fc981\" (UID: \"ff3d5b5d-94fe-40f6-baea-d732683fc981\") "
Oct 30 07:23:07.595151 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:07.595029    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff3d5b5d-94fe-40f6-baea-d732683fc981-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff3d5b5d-94fe-40f6-baea-d732683fc981-istio-podinfo\") pod \"ff3d5b5d-94fe-40f6-baea-d732683fc981\" (UID: \"ff3d5b5d-94fe-40f6-baea-d732683fc981\") "
Oct 30 07:23:07.595151 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:07.595036    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff3d5b5d-94fe-40f6-baea-d732683fc981-credential-socket" pod "ff3d5b5d-94fe-40f6-baea-d732683fc981" (UID: "ff3d5b5d-94fe-40f6-baea-d732683fc981"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:23:07.595151 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:07.595078    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff3d5b5d-94fe-40f6-baea-d732683fc981-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff3d5b5d-94fe-40f6-baea-d732683fc981-istio-data\") pod \"ff3d5b5d-94fe-40f6-baea-d732683fc981\" (UID: \"ff3d5b5d-94fe-40f6-baea-d732683fc981\") "
Oct 30 07:23:07.595151 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:07.595129    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff3d5b5d-94fe-40f6-baea-d732683fc981-kube-api-access-pf58f\" (UniqueName: \"kubernetes.io/projected/ff3d5b5d-94fe-40f6-baea-d732683fc981-kube-api-access-pf58f\") pod \"ff3d5b5d-94fe-40f6-baea-d732683fc981\" (UID: \"ff3d5b5d-94fe-40f6-baea-d732683fc981\") "
Oct 30 07:23:07.595345 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:07.595177    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff3d5b5d-94fe-40f6-baea-d732683fc981-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff3d5b5d-94fe-40f6-baea-d732683fc981-istiod-ca-cert\") pod \"ff3d5b5d-94fe-40f6-baea-d732683fc981\" (UID: \"ff3d5b5d-94fe-40f6-baea-d732683fc981\") "
Oct 30 07:23:07.595345 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:07.595176    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff3d5b5d-94fe-40f6-baea-d732683fc981-workload-certs" pod "ff3d5b5d-94fe-40f6-baea-d732683fc981" (UID: "ff3d5b5d-94fe-40f6-baea-d732683fc981"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:23:07.595345 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:07.595251    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff3d5b5d-94fe-40f6-baea-d732683fc981-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3d5b5d-94fe-40f6-baea-d732683fc981-workload-socket\") pod \"ff3d5b5d-94fe-40f6-baea-d732683fc981\" (UID: \"ff3d5b5d-94fe-40f6-baea-d732683fc981\") "
Oct 30 07:23:07.595508 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:07.595376    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff3d5b5d-94fe-40f6-baea-d732683fc981-istio-data" pod "ff3d5b5d-94fe-40f6-baea-d732683fc981" (UID: "ff3d5b5d-94fe-40f6-baea-d732683fc981"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:23:07.595900 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:07.595593    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff3d5b5d-94fe-40f6-baea-d732683fc981-workload-socket" pod "ff3d5b5d-94fe-40f6-baea-d732683fc981" (UID: "ff3d5b5d-94fe-40f6-baea-d732683fc981"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:23:07.595900 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:07.595869    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff3d5b5d-94fe-40f6-baea-d732683fc981-istiod-ca-cert" pod "ff3d5b5d-94fe-40f6-baea-d732683fc981" (UID: "ff3d5b5d-94fe-40f6-baea-d732683fc981"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:23:07.595900 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:07.595893    6224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff3d5b5d-94fe-40f6-baea-d732683fc981-istio-data\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:23:07.596127 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:07.595911    6224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3d5b5d-94fe-40f6-baea-d732683fc981-workload-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:23:07.596127 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:07.595924    6224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3d5b5d-94fe-40f6-baea-d732683fc981-credential-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:23:07.596127 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:07.595937    6224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff3d5b5d-94fe-40f6-baea-d732683fc981-workload-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:23:07.597739 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:07.597692    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff3d5b5d-94fe-40f6-baea-d732683fc981-istio-podinfo" pod "ff3d5b5d-94fe-40f6-baea-d732683fc981" (UID: "ff3d5b5d-94fe-40f6-baea-d732683fc981"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:23:07.597901 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:07.597784    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff3d5b5d-94fe-40f6-baea-d732683fc981-istio-envoy" pod "ff3d5b5d-94fe-40f6-baea-d732683fc981" (UID: "ff3d5b5d-94fe-40f6-baea-d732683fc981"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:23:07.598016 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:07.597994    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff3d5b5d-94fe-40f6-baea-d732683fc981-kube-api-access-pf58f" pod "ff3d5b5d-94fe-40f6-baea-d732683fc981" (UID: "ff3d5b5d-94fe-40f6-baea-d732683fc981"). InnerVolumeSpecName "kube-api-access-pf58f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:23:07.598016 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:07.598008    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff3d5b5d-94fe-40f6-baea-d732683fc981-istio-token" pod "ff3d5b5d-94fe-40f6-baea-d732683fc981" (UID: "ff3d5b5d-94fe-40f6-baea-d732683fc981"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:23:07.696638 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:07.696601    6224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff3d5b5d-94fe-40f6-baea-d732683fc981-istio-token\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:23:07.696638 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:07.696630    6224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff3d5b5d-94fe-40f6-baea-d732683fc981-istio-envoy\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:23:07.696638 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:07.696642    6224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff3d5b5d-94fe-40f6-baea-d732683fc981-istio-podinfo\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:23:07.696887 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:07.696655    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pf58f\" (UniqueName: \"kubernetes.io/projected/ff3d5b5d-94fe-40f6-baea-d732683fc981-kube-api-access-pf58f\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:23:07.696887 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:07.696667    6224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff3d5b5d-94fe-40f6-baea-d732683fc981-istiod-ca-cert\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:23:07.862229 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:07.862196    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54"]
Oct 30 07:23:07.875441 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:07.875410    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-704bf6-openshift-default-7b79bc878f-mll54"]
Oct 30 07:23:09.502931 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:09.502896    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff3d5b5d-94fe-40f6-baea-d732683fc981" path="/var/lib/kubelet/pods/ff3d5b5d-94fe-40f6-baea-d732683fc981/volumes"
Oct 30 07:23:20.270942 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:20.270892    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dpzbb"]
Oct 30 07:23:20.271430 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:20.271108    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dpzbb" podUID="bf2e54e1-8437-47eb-bfa7-04bde103ebf8" containerName="discovery" containerID="cri-o://725955b0bb733b6239ae04fcbed2ac3bce21ec0884d929e0ee2321c890d4eebb" gracePeriod=30
Oct 30 07:23:20.539070 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:20.539047    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dpzbb"
Oct 30 07:23:20.554406 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:20.554380    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf2e54e1-8437-47eb-bfa7-04bde103ebf8-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bf2e54e1-8437-47eb-bfa7-04bde103ebf8-istio-kubeconfig\") pod \"bf2e54e1-8437-47eb-bfa7-04bde103ebf8\" (UID: \"bf2e54e1-8437-47eb-bfa7-04bde103ebf8\") "
Oct 30 07:23:20.554520 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:20.554476    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf2e54e1-8437-47eb-bfa7-04bde103ebf8-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf2e54e1-8437-47eb-bfa7-04bde103ebf8-local-certs\") pod \"bf2e54e1-8437-47eb-bfa7-04bde103ebf8\" (UID: \"bf2e54e1-8437-47eb-bfa7-04bde103ebf8\") "
Oct 30 07:23:20.554585 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:20.554567    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf2e54e1-8437-47eb-bfa7-04bde103ebf8-istio-token\" (UniqueName: \"kubernetes.io/projected/bf2e54e1-8437-47eb-bfa7-04bde103ebf8-istio-token\") pod \"bf2e54e1-8437-47eb-bfa7-04bde103ebf8\" (UID: \"bf2e54e1-8437-47eb-bfa7-04bde103ebf8\") "
Oct 30 07:23:20.554660 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:20.554644    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf2e54e1-8437-47eb-bfa7-04bde103ebf8-kube-api-access-kmtpp\" (UniqueName: \"kubernetes.io/projected/bf2e54e1-8437-47eb-bfa7-04bde103ebf8-kube-api-access-kmtpp\") pod \"bf2e54e1-8437-47eb-bfa7-04bde103ebf8\" (UID: \"bf2e54e1-8437-47eb-bfa7-04bde103ebf8\") "
Oct 30 07:23:20.554719 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:20.554676    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf2e54e1-8437-47eb-bfa7-04bde103ebf8-cacerts\" (UniqueName: \"kubernetes.io/secret/bf2e54e1-8437-47eb-bfa7-04bde103ebf8-cacerts\") pod \"bf2e54e1-8437-47eb-bfa7-04bde103ebf8\" (UID: \"bf2e54e1-8437-47eb-bfa7-04bde103ebf8\") "
Oct 30 07:23:20.554819 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:20.554788    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf2e54e1-8437-47eb-bfa7-04bde103ebf8-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bf2e54e1-8437-47eb-bfa7-04bde103ebf8-istio-csr-ca-configmap\") pod \"bf2e54e1-8437-47eb-bfa7-04bde103ebf8\" (UID: \"bf2e54e1-8437-47eb-bfa7-04bde103ebf8\") "
Oct 30 07:23:20.554889 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:20.554858    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf2e54e1-8437-47eb-bfa7-04bde103ebf8-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bf2e54e1-8437-47eb-bfa7-04bde103ebf8-istio-csr-dns-cert\") pod \"bf2e54e1-8437-47eb-bfa7-04bde103ebf8\" (UID: \"bf2e54e1-8437-47eb-bfa7-04bde103ebf8\") "
Oct 30 07:23:20.555322 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:20.555210    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf2e54e1-8437-47eb-bfa7-04bde103ebf8-istio-csr-ca-configmap" pod "bf2e54e1-8437-47eb-bfa7-04bde103ebf8" (UID: "bf2e54e1-8437-47eb-bfa7-04bde103ebf8"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:23:20.555433 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:20.555410    6224 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bf2e54e1-8437-47eb-bfa7-04bde103ebf8-istio-csr-ca-configmap\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:23:20.556948 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:20.556925    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf2e54e1-8437-47eb-bfa7-04bde103ebf8-cacerts" pod "bf2e54e1-8437-47eb-bfa7-04bde103ebf8" (UID: "bf2e54e1-8437-47eb-bfa7-04bde103ebf8"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:23:20.557156 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:20.557058    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf2e54e1-8437-47eb-bfa7-04bde103ebf8-istio-kubeconfig" pod "bf2e54e1-8437-47eb-bfa7-04bde103ebf8" (UID: "bf2e54e1-8437-47eb-bfa7-04bde103ebf8"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:23:20.557156 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:20.557140    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf2e54e1-8437-47eb-bfa7-04bde103ebf8-istio-token" pod "bf2e54e1-8437-47eb-bfa7-04bde103ebf8" (UID: "bf2e54e1-8437-47eb-bfa7-04bde103ebf8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:23:20.557270 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:20.557104    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf2e54e1-8437-47eb-bfa7-04bde103ebf8-kube-api-access-kmtpp" pod "bf2e54e1-8437-47eb-bfa7-04bde103ebf8" (UID: "bf2e54e1-8437-47eb-bfa7-04bde103ebf8"). InnerVolumeSpecName "kube-api-access-kmtpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:23:20.557270 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:20.557108    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf2e54e1-8437-47eb-bfa7-04bde103ebf8-local-certs" pod "bf2e54e1-8437-47eb-bfa7-04bde103ebf8" (UID: "bf2e54e1-8437-47eb-bfa7-04bde103ebf8"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:23:20.557381 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:20.557366    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf2e54e1-8437-47eb-bfa7-04bde103ebf8-istio-csr-dns-cert" pod "bf2e54e1-8437-47eb-bfa7-04bde103ebf8" (UID: "bf2e54e1-8437-47eb-bfa7-04bde103ebf8"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:23:20.585671 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:20.585647    6224 generic.go:356] "Generic (PLEG): container finished" podID="bf2e54e1-8437-47eb-bfa7-04bde103ebf8" containerID="725955b0bb733b6239ae04fcbed2ac3bce21ec0884d929e0ee2321c890d4eebb" exitCode=0
Oct 30 07:23:20.585767 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:20.585692    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dpzbb"
Oct 30 07:23:20.585767 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:20.585722    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dpzbb" event={"ID":"bf2e54e1-8437-47eb-bfa7-04bde103ebf8","Type":"ContainerDied","Data":"725955b0bb733b6239ae04fcbed2ac3bce21ec0884d929e0ee2321c890d4eebb"}
Oct 30 07:23:20.585767 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:20.585758    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dpzbb" event={"ID":"bf2e54e1-8437-47eb-bfa7-04bde103ebf8","Type":"ContainerDied","Data":"9e1303123eaee352393fcb5f69aa7c345a593beb874f5bf0b11961ecb32949ca"}
Oct 30 07:23:20.585915 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:20.585775    6224 scope.go:122] "RemoveContainer" containerID="725955b0bb733b6239ae04fcbed2ac3bce21ec0884d929e0ee2321c890d4eebb"
Oct 30 07:23:20.598435 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:20.598413    6224 scope.go:122] "RemoveContainer" containerID="725955b0bb733b6239ae04fcbed2ac3bce21ec0884d929e0ee2321c890d4eebb"
Oct 30 07:23:20.598742 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:23:20.598721    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"725955b0bb733b6239ae04fcbed2ac3bce21ec0884d929e0ee2321c890d4eebb\": container with ID starting with 725955b0bb733b6239ae04fcbed2ac3bce21ec0884d929e0ee2321c890d4eebb not found: ID does not exist" containerID="725955b0bb733b6239ae04fcbed2ac3bce21ec0884d929e0ee2321c890d4eebb"
Oct 30 07:23:20.598850 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:20.598749    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"725955b0bb733b6239ae04fcbed2ac3bce21ec0884d929e0ee2321c890d4eebb"} err="failed to get container status \"725955b0bb733b6239ae04fcbed2ac3bce21ec0884d929e0ee2321c890d4eebb\": rpc error: code = NotFound desc = could not find container \"725955b0bb733b6239ae04fcbed2ac3bce21ec0884d929e0ee2321c890d4eebb\": container with ID starting with 725955b0bb733b6239ae04fcbed2ac3bce21ec0884d929e0ee2321c890d4eebb not found: ID does not exist"
Oct 30 07:23:20.652862 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:20.652770    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dpzbb"]
Oct 30 07:23:20.656573 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:20.656534    6224 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf2e54e1-8437-47eb-bfa7-04bde103ebf8-local-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:23:20.656573 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:20.656569    6224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf2e54e1-8437-47eb-bfa7-04bde103ebf8-istio-token\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:23:20.656573 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:20.656578    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmtpp\" (UniqueName: \"kubernetes.io/projected/bf2e54e1-8437-47eb-bfa7-04bde103ebf8-kube-api-access-kmtpp\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:23:20.656749 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:20.656587    6224 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bf2e54e1-8437-47eb-bfa7-04bde103ebf8-cacerts\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:23:20.656749 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:20.656597    6224 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bf2e54e1-8437-47eb-bfa7-04bde103ebf8-istio-csr-dns-cert\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:23:20.656749 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:20.656608    6224 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bf2e54e1-8437-47eb-bfa7-04bde103ebf8-istio-kubeconfig\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:23:20.656749 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:20.656709    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dpzbb"]
Oct 30 07:23:21.504034 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:21.503999    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf2e54e1-8437-47eb-bfa7-04bde103ebf8" path="/var/lib/kubelet/pods/bf2e54e1-8437-47eb-bfa7-04bde103ebf8/volumes"
Oct 30 07:23:23.966009 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:23.965969    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b387aa31-pzrjz"]
Oct 30 07:23:23.966467 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:23.966410    6224 cpu_manager.go:414] "containerMap: removing container" podUID="ff3d5b5d-94fe-40f6-baea-d732683fc981" containerName="istio-proxy"
Oct 30 07:23:23.966467 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:23.966421    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff3d5b5d-94fe-40f6-baea-d732683fc981" containerName="istio-proxy"
Oct 30 07:23:23.966467 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:23.966441    6224 cpu_manager.go:414] "containerMap: removing container" podUID="ada682d6-e6ed-4d78-a3cf-07072b35f4f0" containerName="istio-proxy"
Oct 30 07:23:23.966467 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:23.966446    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ada682d6-e6ed-4d78-a3cf-07072b35f4f0" containerName="istio-proxy"
Oct 30 07:23:23.966467 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:23.966459    6224 cpu_manager.go:414] "containerMap: removing container" podUID="bf2e54e1-8437-47eb-bfa7-04bde103ebf8" containerName="discovery"
Oct 30 07:23:23.966467 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:23.966464    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf2e54e1-8437-47eb-bfa7-04bde103ebf8" containerName="discovery"
Oct 30 07:23:23.966652 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:23.966472    6224 cpu_manager.go:414] "containerMap: removing container" podUID="44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7" containerName="istio-proxy"
Oct 30 07:23:23.966652 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:23.966477    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7" containerName="istio-proxy"
Oct 30 07:23:23.966652 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:23.966483    6224 cpu_manager.go:414] "containerMap: removing container" podUID="6fd8ca77-b283-4d93-87e7-640397564926" containerName="istio-proxy"
Oct 30 07:23:23.966652 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:23.966487    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fd8ca77-b283-4d93-87e7-640397564926" containerName="istio-proxy"
Oct 30 07:23:23.966652 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:23.966507    6224 cpu_manager.go:414] "containerMap: removing container" podUID="9ec378e4-d80e-4306-b1b8-8556b80ce58d" containerName="istio-proxy"
Oct 30 07:23:23.966652 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:23.966511    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ec378e4-d80e-4306-b1b8-8556b80ce58d" containerName="istio-proxy"
Oct 30 07:23:23.966652 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:23.966602    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff3d5b5d-94fe-40f6-baea-d732683fc981" containerName="istio-proxy"
Oct 30 07:23:23.966652 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:23.966611    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="ada682d6-e6ed-4d78-a3cf-07072b35f4f0" containerName="istio-proxy"
Oct 30 07:23:23.966652 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:23.966618    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf2e54e1-8437-47eb-bfa7-04bde103ebf8" containerName="discovery"
Oct 30 07:23:23.966652 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:23.966625    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="44d3fa93-d6ab-40f7-a8f7-ee15e4ee0ca7" containerName="istio-proxy"
Oct 30 07:23:23.966652 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:23.966632    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fd8ca77-b283-4d93-87e7-640397564926" containerName="istio-proxy"
Oct 30 07:23:23.966652 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:23.966641    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ec378e4-d80e-4306-b1b8-8556b80ce58d" containerName="istio-proxy"
Oct 30 07:23:23.971375 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:23.971357    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b387aa31-pzrjz"
Oct 30 07:23:23.988788 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:23.988762    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltnqb\" (UniqueName: \"kubernetes.io/projected/0846f9fd-b067-4762-8590-944ecd142425-kube-api-access-ltnqb\") pod \"maas-api-revoke-sub-b387aa31-pzrjz\" (UID: \"0846f9fd-b067-4762-8590-944ecd142425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b387aa31-pzrjz"
Oct 30 07:23:23.989130 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:23.988854    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0846f9fd-b067-4762-8590-944ecd142425-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b387aa31-pzrjz\" (UID: \"0846f9fd-b067-4762-8590-944ecd142425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b387aa31-pzrjz"
Oct 30 07:23:23.989130 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:23.988948    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0846f9fd-b067-4762-8590-944ecd142425-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b387aa31-pzrjz\" (UID: \"0846f9fd-b067-4762-8590-944ecd142425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b387aa31-pzrjz"
Oct 30 07:23:23.989555 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:23.989536    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b387aa31-pzrjz"]
Oct 30 07:23:24.000311 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:24.000283    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:23:24.000419 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:24.000369    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:23:24.090300 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:24.090270    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltnqb\" (UniqueName: \"kubernetes.io/projected/0846f9fd-b067-4762-8590-944ecd142425-kube-api-access-ltnqb\") pod \"maas-api-revoke-sub-b387aa31-pzrjz\" (UID: \"0846f9fd-b067-4762-8590-944ecd142425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b387aa31-pzrjz"
Oct 30 07:23:24.090448 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:24.090315    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0846f9fd-b067-4762-8590-944ecd142425-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b387aa31-pzrjz\" (UID: \"0846f9fd-b067-4762-8590-944ecd142425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b387aa31-pzrjz"
Oct 30 07:23:24.090505 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:24.090438    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0846f9fd-b067-4762-8590-944ecd142425-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b387aa31-pzrjz\" (UID: \"0846f9fd-b067-4762-8590-944ecd142425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b387aa31-pzrjz"
Oct 30 07:23:24.091024 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:24.091003    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0846f9fd-b067-4762-8590-944ecd142425-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b387aa31-pzrjz\" (UID: \"0846f9fd-b067-4762-8590-944ecd142425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b387aa31-pzrjz"
Oct 30 07:23:24.092761 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:24.092740    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0846f9fd-b067-4762-8590-944ecd142425-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b387aa31-pzrjz\" (UID: \"0846f9fd-b067-4762-8590-944ecd142425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b387aa31-pzrjz"
Oct 30 07:23:24.115105 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:24.115083    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltnqb\" (UniqueName: \"kubernetes.io/projected/0846f9fd-b067-4762-8590-944ecd142425-kube-api-access-ltnqb\") pod \"maas-api-revoke-sub-b387aa31-pzrjz\" (UID: \"0846f9fd-b067-4762-8590-944ecd142425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b387aa31-pzrjz"
Oct 30 07:23:24.284864 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:24.284836    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b387aa31-pzrjz"
Oct 30 07:23:24.588297 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:24.588264    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b387aa31-pzrjz"]
Oct 30 07:23:24.588534 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:23:24.588513    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0846f9fd_b067_4762_8590_944ecd142425.slice/crio-5689a5e3790f604423f47628dc0f5aa29d9e701414f77bcc1cd9108623b8c3bf WatchSource:0}: Error finding container 5689a5e3790f604423f47628dc0f5aa29d9e701414f77bcc1cd9108623b8c3bf: Status 404 returned error can't find the container with id 5689a5e3790f604423f47628dc0f5aa29d9e701414f77bcc1cd9108623b8c3bf
Oct 30 07:23:24.611929 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:24.611902    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b387aa31-pzrjz" event={"ID":"0846f9fd-b067-4762-8590-944ecd142425","Type":"ContainerStarted","Data":"5689a5e3790f604423f47628dc0f5aa29d9e701414f77bcc1cd9108623b8c3bf"}
Oct 30 07:23:25.618791 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:25.618702    6224 generic.go:356] "Generic (PLEG): container finished" podID="0846f9fd-b067-4762-8590-944ecd142425" containerID="1b5a3b65466b8431bc64a51f957c30600081d3855126a26224d60e6d0d9319f1" exitCode=0
Oct 30 07:23:25.619196 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:25.618784    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b387aa31-pzrjz" event={"ID":"0846f9fd-b067-4762-8590-944ecd142425","Type":"ContainerDied","Data":"1b5a3b65466b8431bc64a51f957c30600081d3855126a26224d60e6d0d9319f1"}
Oct 30 07:23:26.774759 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:26.774730    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b387aa31-pzrjz"
Oct 30 07:23:26.821724 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:26.821691    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0846f9fd-b067-4762-8590-944ecd142425-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0846f9fd-b067-4762-8590-944ecd142425-maas-api-openshift-ca\") pod \"0846f9fd-b067-4762-8590-944ecd142425\" (UID: \"0846f9fd-b067-4762-8590-944ecd142425\") "
Oct 30 07:23:26.821911 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:26.821860    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0846f9fd-b067-4762-8590-944ecd142425-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0846f9fd-b067-4762-8590-944ecd142425-maas-api-xks-ca\") pod \"0846f9fd-b067-4762-8590-944ecd142425\" (UID: \"0846f9fd-b067-4762-8590-944ecd142425\") "
Oct 30 07:23:26.821993 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:26.821923    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0846f9fd-b067-4762-8590-944ecd142425-kube-api-access-ltnqb\" (UniqueName: \"kubernetes.io/projected/0846f9fd-b067-4762-8590-944ecd142425-kube-api-access-ltnqb\") pod \"0846f9fd-b067-4762-8590-944ecd142425\" (UID: \"0846f9fd-b067-4762-8590-944ecd142425\") "
Oct 30 07:23:26.822817 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:26.822773    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0846f9fd-b067-4762-8590-944ecd142425-maas-api-openshift-ca" pod "0846f9fd-b067-4762-8590-944ecd142425" (UID: "0846f9fd-b067-4762-8590-944ecd142425"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:23:26.824616 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:26.824585    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0846f9fd-b067-4762-8590-944ecd142425-maas-api-xks-ca" pod "0846f9fd-b067-4762-8590-944ecd142425" (UID: "0846f9fd-b067-4762-8590-944ecd142425"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:23:26.824887 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:26.824857    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0846f9fd-b067-4762-8590-944ecd142425-kube-api-access-ltnqb" pod "0846f9fd-b067-4762-8590-944ecd142425" (UID: "0846f9fd-b067-4762-8590-944ecd142425"). InnerVolumeSpecName "kube-api-access-ltnqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:23:26.923521 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:26.923450    6224 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0846f9fd-b067-4762-8590-944ecd142425-maas-api-xks-ca\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:23:26.923521 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:26.923477    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltnqb\" (UniqueName: \"kubernetes.io/projected/0846f9fd-b067-4762-8590-944ecd142425-kube-api-access-ltnqb\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:23:26.923521 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:26.923488    6224 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0846f9fd-b067-4762-8590-944ecd142425-maas-api-openshift-ca\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:23:27.632745 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:27.632716    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b387aa31-pzrjz"
Oct 30 07:23:27.632921 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:27.632744    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b387aa31-pzrjz" event={"ID":"0846f9fd-b067-4762-8590-944ecd142425","Type":"ContainerDied","Data":"5689a5e3790f604423f47628dc0f5aa29d9e701414f77bcc1cd9108623b8c3bf"}
Oct 30 07:23:27.632921 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:27.632793    6224 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5689a5e3790f604423f47628dc0f5aa29d9e701414f77bcc1cd9108623b8c3bf"
Oct 30 07:23:28.600232 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:28.600179    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:23:31.858211 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:23:31.858173    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:23:46.381532 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:46.381499    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc"]
Oct 30 07:23:46.381963 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:46.381727    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc" podUID="79fa648e-4151-4998-b954-fc46d9841b97" containerName="istio-proxy" containerID="cri-o://7c571c755f2c14a1b556679c7c9b250883d3784dd73c8aa9d214ab917cf63b71" gracePeriod=30
Oct 30 07:23:47.818112 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:47.818079    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6eee8b4c-f28rj"]
Oct 30 07:23:47.818561 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:47.818547    6224 cpu_manager.go:414] "containerMap: removing container" podUID="0846f9fd-b067-4762-8590-944ecd142425" containerName="revoke-keys"
Oct 30 07:23:47.818561 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:47.818561    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0846f9fd-b067-4762-8590-944ecd142425" containerName="revoke-keys"
Oct 30 07:23:47.818646 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:47.818637    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="0846f9fd-b067-4762-8590-944ecd142425" containerName="revoke-keys"
Oct 30 07:23:47.821955 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:47.821934    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eee8b4c-f28rj"
Oct 30 07:23:47.840889 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:47.840863    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6eee8b4c-f28rj"]
Oct 30 07:23:47.851396 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:47.851374    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:23:47.851506 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:47.851458    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:23:47.894214 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:47.894187    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59vxr\" (UniqueName: \"kubernetes.io/projected/0e7ee039-3a41-4271-8335-9cd1c1796949-kube-api-access-59vxr\") pod \"maas-api-revoke-sub-6eee8b4c-f28rj\" (UID: \"0e7ee039-3a41-4271-8335-9cd1c1796949\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eee8b4c-f28rj"
Oct 30 07:23:47.894353 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:47.894226    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e7ee039-3a41-4271-8335-9cd1c1796949-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6eee8b4c-f28rj\" (UID: \"0e7ee039-3a41-4271-8335-9cd1c1796949\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eee8b4c-f28rj"
Oct 30 07:23:47.894398 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:47.894370    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e7ee039-3a41-4271-8335-9cd1c1796949-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6eee8b4c-f28rj\" (UID: \"0e7ee039-3a41-4271-8335-9cd1c1796949\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eee8b4c-f28rj"
Oct 30 07:23:47.995914 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:47.995883    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59vxr\" (UniqueName: \"kubernetes.io/projected/0e7ee039-3a41-4271-8335-9cd1c1796949-kube-api-access-59vxr\") pod \"maas-api-revoke-sub-6eee8b4c-f28rj\" (UID: \"0e7ee039-3a41-4271-8335-9cd1c1796949\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eee8b4c-f28rj"
Oct 30 07:23:47.996064 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:47.995923    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e7ee039-3a41-4271-8335-9cd1c1796949-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6eee8b4c-f28rj\" (UID: \"0e7ee039-3a41-4271-8335-9cd1c1796949\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eee8b4c-f28rj"
Oct 30 07:23:47.996064 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:47.995975    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e7ee039-3a41-4271-8335-9cd1c1796949-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6eee8b4c-f28rj\" (UID: \"0e7ee039-3a41-4271-8335-9cd1c1796949\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eee8b4c-f28rj"
Oct 30 07:23:47.996611 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:47.996581    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e7ee039-3a41-4271-8335-9cd1c1796949-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6eee8b4c-f28rj\" (UID: \"0e7ee039-3a41-4271-8335-9cd1c1796949\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eee8b4c-f28rj"
Oct 30 07:23:47.998253 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:47.998233    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e7ee039-3a41-4271-8335-9cd1c1796949-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6eee8b4c-f28rj\" (UID: \"0e7ee039-3a41-4271-8335-9cd1c1796949\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eee8b4c-f28rj"
Oct 30 07:23:48.021205 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:48.021181    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59vxr\" (UniqueName: \"kubernetes.io/projected/0e7ee039-3a41-4271-8335-9cd1c1796949-kube-api-access-59vxr\") pod \"maas-api-revoke-sub-6eee8b4c-f28rj\" (UID: \"0e7ee039-3a41-4271-8335-9cd1c1796949\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eee8b4c-f28rj"
Oct 30 07:23:48.134890 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:48.134826    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eee8b4c-f28rj"
Oct 30 07:23:48.431695 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:23:48.431664    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e7ee039_3a41_4271_8335_9cd1c1796949.slice/crio-874ccd50cb13627cc76ffc737e0a20becac26e43df350684b6a6f41e61084fae WatchSource:0}: Error finding container 874ccd50cb13627cc76ffc737e0a20becac26e43df350684b6a6f41e61084fae: Status 404 returned error can't find the container with id 874ccd50cb13627cc76ffc737e0a20becac26e43df350684b6a6f41e61084fae
Oct 30 07:23:48.432047 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:48.432024    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6eee8b4c-f28rj"]
Oct 30 07:23:48.781170 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:48.781143    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eee8b4c-f28rj" event={"ID":"0e7ee039-3a41-4271-8335-9cd1c1796949","Type":"ContainerStarted","Data":"874ccd50cb13627cc76ffc737e0a20becac26e43df350684b6a6f41e61084fae"}
Oct 30 07:23:49.869692 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:49.869652    6224 generic.go:356] "Generic (PLEG): container finished" podID="0e7ee039-3a41-4271-8335-9cd1c1796949" containerID="154a053b99a8d8d4b86b177e9ec1268b1c5cbb1a2295a7867dcebcf40ee92506" exitCode=0
Oct 30 07:23:49.870187 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:49.869692    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eee8b4c-f28rj" event={"ID":"0e7ee039-3a41-4271-8335-9cd1c1796949","Type":"ContainerDied","Data":"154a053b99a8d8d4b86b177e9ec1268b1c5cbb1a2295a7867dcebcf40ee92506"}
Oct 30 07:23:51.025752 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:51.025723    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eee8b4c-f28rj"
Oct 30 07:23:51.071574 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:51.071514    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e7ee039-3a41-4271-8335-9cd1c1796949-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e7ee039-3a41-4271-8335-9cd1c1796949-maas-api-openshift-ca\") pod \"0e7ee039-3a41-4271-8335-9cd1c1796949\" (UID: \"0e7ee039-3a41-4271-8335-9cd1c1796949\") "
Oct 30 07:23:51.071786 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:51.071599    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e7ee039-3a41-4271-8335-9cd1c1796949-kube-api-access-59vxr\" (UniqueName: \"kubernetes.io/projected/0e7ee039-3a41-4271-8335-9cd1c1796949-kube-api-access-59vxr\") pod \"0e7ee039-3a41-4271-8335-9cd1c1796949\" (UID: \"0e7ee039-3a41-4271-8335-9cd1c1796949\") "
Oct 30 07:23:51.072124 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:51.072095    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e7ee039-3a41-4271-8335-9cd1c1796949-maas-api-openshift-ca" pod "0e7ee039-3a41-4271-8335-9cd1c1796949" (UID: "0e7ee039-3a41-4271-8335-9cd1c1796949"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:23:51.072407 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:51.072383    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e7ee039-3a41-4271-8335-9cd1c1796949-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e7ee039-3a41-4271-8335-9cd1c1796949-maas-api-xks-ca\") pod \"0e7ee039-3a41-4271-8335-9cd1c1796949\" (UID: \"0e7ee039-3a41-4271-8335-9cd1c1796949\") "
Oct 30 07:23:51.073507 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:51.073484    6224 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e7ee039-3a41-4271-8335-9cd1c1796949-maas-api-openshift-ca\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:23:51.074165 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:51.074139    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e7ee039-3a41-4271-8335-9cd1c1796949-kube-api-access-59vxr" pod "0e7ee039-3a41-4271-8335-9cd1c1796949" (UID: "0e7ee039-3a41-4271-8335-9cd1c1796949"). InnerVolumeSpecName "kube-api-access-59vxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:23:51.074986 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:51.074956    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e7ee039-3a41-4271-8335-9cd1c1796949-maas-api-xks-ca" pod "0e7ee039-3a41-4271-8335-9cd1c1796949" (UID: "0e7ee039-3a41-4271-8335-9cd1c1796949"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:23:51.174964 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:51.174895    6224 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e7ee039-3a41-4271-8335-9cd1c1796949-maas-api-xks-ca\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:23:51.174964 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:51.174922    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59vxr\" (UniqueName: \"kubernetes.io/projected/0e7ee039-3a41-4271-8335-9cd1c1796949-kube-api-access-59vxr\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:23:51.641063 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:51.641032    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc"
Oct 30 07:23:51.679609 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:51.679585    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79fa648e-4151-4998-b954-fc46d9841b97-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79fa648e-4151-4998-b954-fc46d9841b97-istio-envoy\") pod \"79fa648e-4151-4998-b954-fc46d9841b97\" (UID: \"79fa648e-4151-4998-b954-fc46d9841b97\") "
Oct 30 07:23:51.679749 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:51.679618    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79fa648e-4151-4998-b954-fc46d9841b97-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79fa648e-4151-4998-b954-fc46d9841b97-credential-socket\") pod \"79fa648e-4151-4998-b954-fc46d9841b97\" (UID: \"79fa648e-4151-4998-b954-fc46d9841b97\") "
Oct 30 07:23:51.679749 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:51.679649    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79fa648e-4151-4998-b954-fc46d9841b97-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79fa648e-4151-4998-b954-fc46d9841b97-istio-data\") pod \"79fa648e-4151-4998-b954-fc46d9841b97\" (UID: \"79fa648e-4151-4998-b954-fc46d9841b97\") "
Oct 30 07:23:51.679749 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:51.679706    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79fa648e-4151-4998-b954-fc46d9841b97-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79fa648e-4151-4998-b954-fc46d9841b97-workload-socket\") pod \"79fa648e-4151-4998-b954-fc46d9841b97\" (UID: \"79fa648e-4151-4998-b954-fc46d9841b97\") "
Oct 30 07:23:51.679915 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:51.679785    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79fa648e-4151-4998-b954-fc46d9841b97-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79fa648e-4151-4998-b954-fc46d9841b97-workload-certs\") pod \"79fa648e-4151-4998-b954-fc46d9841b97\" (UID: \"79fa648e-4151-4998-b954-fc46d9841b97\") "
Oct 30 07:23:51.679915 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:51.679843    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79fa648e-4151-4998-b954-fc46d9841b97-istio-token\" (UniqueName: \"kubernetes.io/projected/79fa648e-4151-4998-b954-fc46d9841b97-istio-token\") pod \"79fa648e-4151-4998-b954-fc46d9841b97\" (UID: \"79fa648e-4151-4998-b954-fc46d9841b97\") "
Oct 30 07:23:51.680028 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:51.679926    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79fa648e-4151-4998-b954-fc46d9841b97-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79fa648e-4151-4998-b954-fc46d9841b97-istiod-ca-cert\") pod \"79fa648e-4151-4998-b954-fc46d9841b97\" (UID: \"79fa648e-4151-4998-b954-fc46d9841b97\") "
Oct 30 07:23:51.680028 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:51.679926    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79fa648e-4151-4998-b954-fc46d9841b97-credential-socket" pod "79fa648e-4151-4998-b954-fc46d9841b97" (UID: "79fa648e-4151-4998-b954-fc46d9841b97"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:23:51.680028 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:51.679964    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79fa648e-4151-4998-b954-fc46d9841b97-istio-data" pod "79fa648e-4151-4998-b954-fc46d9841b97" (UID: "79fa648e-4151-4998-b954-fc46d9841b97"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:23:51.680187 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:51.680031    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79fa648e-4151-4998-b954-fc46d9841b97-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79fa648e-4151-4998-b954-fc46d9841b97-istio-podinfo\") pod \"79fa648e-4151-4998-b954-fc46d9841b97\" (UID: \"79fa648e-4151-4998-b954-fc46d9841b97\") "
Oct 30 07:23:51.680187 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:51.680040    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79fa648e-4151-4998-b954-fc46d9841b97-workload-socket" pod "79fa648e-4151-4998-b954-fc46d9841b97" (UID: "79fa648e-4151-4998-b954-fc46d9841b97"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:23:51.680187 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:51.680068    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79fa648e-4151-4998-b954-fc46d9841b97-workload-certs" pod "79fa648e-4151-4998-b954-fc46d9841b97" (UID: "79fa648e-4151-4998-b954-fc46d9841b97"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:23:51.680381 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:51.680186    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79fa648e-4151-4998-b954-fc46d9841b97-kube-api-access-t9pvn\" (UniqueName: \"kubernetes.io/projected/79fa648e-4151-4998-b954-fc46d9841b97-kube-api-access-t9pvn\") pod \"79fa648e-4151-4998-b954-fc46d9841b97\" (UID: \"79fa648e-4151-4998-b954-fc46d9841b97\") "
Oct 30 07:23:51.680381 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:51.680237    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79fa648e-4151-4998-b954-fc46d9841b97-istiod-ca-cert" pod "79fa648e-4151-4998-b954-fc46d9841b97" (UID: "79fa648e-4151-4998-b954-fc46d9841b97"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:23:51.680673 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:51.680651    6224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79fa648e-4151-4998-b954-fc46d9841b97-credential-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:23:51.680754 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:51.680677    6224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79fa648e-4151-4998-b954-fc46d9841b97-istio-data\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:23:51.680754 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:51.680691    6224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79fa648e-4151-4998-b954-fc46d9841b97-workload-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:23:51.680754 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:51.680704    6224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79fa648e-4151-4998-b954-fc46d9841b97-workload-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:23:51.680754 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:51.680717    6224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79fa648e-4151-4998-b954-fc46d9841b97-istiod-ca-cert\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:23:51.682020 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:51.681998    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79fa648e-4151-4998-b954-fc46d9841b97-istio-envoy" pod "79fa648e-4151-4998-b954-fc46d9841b97" (UID: "79fa648e-4151-4998-b954-fc46d9841b97"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:23:51.682125 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:51.682038    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79fa648e-4151-4998-b954-fc46d9841b97-istio-token" pod "79fa648e-4151-4998-b954-fc46d9841b97" (UID: "79fa648e-4151-4998-b954-fc46d9841b97"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:23:51.682401 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:51.682388    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79fa648e-4151-4998-b954-fc46d9841b97-istio-podinfo" pod "79fa648e-4151-4998-b954-fc46d9841b97" (UID: "79fa648e-4151-4998-b954-fc46d9841b97"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:23:51.682442 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:51.682400    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79fa648e-4151-4998-b954-fc46d9841b97-kube-api-access-t9pvn" pod "79fa648e-4151-4998-b954-fc46d9841b97" (UID: "79fa648e-4151-4998-b954-fc46d9841b97"). InnerVolumeSpecName "kube-api-access-t9pvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:23:51.781567 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:51.781536    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9pvn\" (UniqueName: \"kubernetes.io/projected/79fa648e-4151-4998-b954-fc46d9841b97-kube-api-access-t9pvn\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:23:51.781567 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:51.781565    6224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79fa648e-4151-4998-b954-fc46d9841b97-istio-envoy\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:23:51.781745 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:51.781577    6224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79fa648e-4151-4998-b954-fc46d9841b97-istio-token\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:23:51.781745 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:51.781589    6224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79fa648e-4151-4998-b954-fc46d9841b97-istio-podinfo\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:23:51.882526 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:51.882495    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eee8b4c-f28rj"
Oct 30 07:23:51.882691 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:51.882499    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eee8b4c-f28rj" event={"ID":"0e7ee039-3a41-4271-8335-9cd1c1796949","Type":"ContainerDied","Data":"874ccd50cb13627cc76ffc737e0a20becac26e43df350684b6a6f41e61084fae"}
Oct 30 07:23:51.882691 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:51.882642    6224 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="874ccd50cb13627cc76ffc737e0a20becac26e43df350684b6a6f41e61084fae"
Oct 30 07:23:51.883978 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:51.883951    6224 generic.go:356] "Generic (PLEG): container finished" podID="79fa648e-4151-4998-b954-fc46d9841b97" containerID="7c571c755f2c14a1b556679c7c9b250883d3784dd73c8aa9d214ab917cf63b71" exitCode=0
Oct 30 07:23:51.884124 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:51.883991    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc" event={"ID":"79fa648e-4151-4998-b954-fc46d9841b97","Type":"ContainerDied","Data":"7c571c755f2c14a1b556679c7c9b250883d3784dd73c8aa9d214ab917cf63b71"}
Oct 30 07:23:51.884124 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:51.884004    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc"
Oct 30 07:23:51.884124 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:51.884022    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc" event={"ID":"79fa648e-4151-4998-b954-fc46d9841b97","Type":"ContainerDied","Data":"7b7c8fa38bcf14d6573412161b5985d292fe3b810a4a92cb7b27d4a4a93c7bbb"}
Oct 30 07:23:51.884124 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:51.884041    6224 scope.go:122] "RemoveContainer" containerID="7c571c755f2c14a1b556679c7c9b250883d3784dd73c8aa9d214ab917cf63b71"
Oct 30 07:23:51.894236 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:51.894212    6224 scope.go:122] "RemoveContainer" containerID="7c571c755f2c14a1b556679c7c9b250883d3784dd73c8aa9d214ab917cf63b71"
Oct 30 07:23:51.894535 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:23:51.894508    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c571c755f2c14a1b556679c7c9b250883d3784dd73c8aa9d214ab917cf63b71\": container with ID starting with 7c571c755f2c14a1b556679c7c9b250883d3784dd73c8aa9d214ab917cf63b71 not found: ID does not exist" containerID="7c571c755f2c14a1b556679c7c9b250883d3784dd73c8aa9d214ab917cf63b71"
Oct 30 07:23:51.894615 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:51.894543    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c571c755f2c14a1b556679c7c9b250883d3784dd73c8aa9d214ab917cf63b71"} err="failed to get container status \"7c571c755f2c14a1b556679c7c9b250883d3784dd73c8aa9d214ab917cf63b71\": rpc error: code = NotFound desc = could not find container \"7c571c755f2c14a1b556679c7c9b250883d3784dd73c8aa9d214ab917cf63b71\": container with ID starting with 7c571c755f2c14a1b556679c7c9b250883d3784dd73c8aa9d214ab917cf63b71 not found: ID does not exist"
Oct 30 07:23:51.956951 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:51.956925    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc"]
Oct 30 07:23:51.961069 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:51.961032    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-46phc"]
Oct 30 07:23:53.518863 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:53.518823    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79fa648e-4151-4998-b954-fc46d9841b97" path="/var/lib/kubelet/pods/79fa648e-4151-4998-b954-fc46d9841b97/volumes"
Oct 30 07:23:58.196067 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:58.196037    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-490cf3a0-96jsg"]
Oct 30 07:23:58.196518 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:58.196510    6224 cpu_manager.go:414] "containerMap: removing container" podUID="0e7ee039-3a41-4271-8335-9cd1c1796949" containerName="revoke-keys"
Oct 30 07:23:58.196560 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:58.196522    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e7ee039-3a41-4271-8335-9cd1c1796949" containerName="revoke-keys"
Oct 30 07:23:58.196560 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:58.196537    6224 cpu_manager.go:414] "containerMap: removing container" podUID="79fa648e-4151-4998-b954-fc46d9841b97" containerName="istio-proxy"
Oct 30 07:23:58.196560 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:58.196543    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79fa648e-4151-4998-b954-fc46d9841b97" containerName="istio-proxy"
Oct 30 07:23:58.196652 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:58.196611    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="79fa648e-4151-4998-b954-fc46d9841b97" containerName="istio-proxy"
Oct 30 07:23:58.196652 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:58.196625    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e7ee039-3a41-4271-8335-9cd1c1796949" containerName="revoke-keys"
Oct 30 07:23:58.201426 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:58.201406    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-490cf3a0-96jsg"
Oct 30 07:23:58.221980 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:58.218676    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:23:58.229634 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:58.229601    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-490cf3a0-96jsg"]
Oct 30 07:23:58.231567 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:58.231539    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:23:58.262187 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:58.262153    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k4h8\" (UniqueName: \"kubernetes.io/projected/3b952e3e-75e2-42f1-9dbc-55b5a46dbc16-kube-api-access-6k4h8\") pod \"maas-api-revoke-sub-490cf3a0-96jsg\" (UID: \"3b952e3e-75e2-42f1-9dbc-55b5a46dbc16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-490cf3a0-96jsg"
Oct 30 07:23:58.262303 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:58.262282    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b952e3e-75e2-42f1-9dbc-55b5a46dbc16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-490cf3a0-96jsg\" (UID: \"3b952e3e-75e2-42f1-9dbc-55b5a46dbc16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-490cf3a0-96jsg"
Oct 30 07:23:58.262351 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:58.262335    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b952e3e-75e2-42f1-9dbc-55b5a46dbc16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-490cf3a0-96jsg\" (UID: \"3b952e3e-75e2-42f1-9dbc-55b5a46dbc16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-490cf3a0-96jsg"
Oct 30 07:23:58.363618 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:58.363590    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k4h8\" (UniqueName: \"kubernetes.io/projected/3b952e3e-75e2-42f1-9dbc-55b5a46dbc16-kube-api-access-6k4h8\") pod \"maas-api-revoke-sub-490cf3a0-96jsg\" (UID: \"3b952e3e-75e2-42f1-9dbc-55b5a46dbc16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-490cf3a0-96jsg"
Oct 30 07:23:58.363768 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:58.363680    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b952e3e-75e2-42f1-9dbc-55b5a46dbc16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-490cf3a0-96jsg\" (UID: \"3b952e3e-75e2-42f1-9dbc-55b5a46dbc16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-490cf3a0-96jsg"
Oct 30 07:23:58.363768 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:58.363697    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b952e3e-75e2-42f1-9dbc-55b5a46dbc16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-490cf3a0-96jsg\" (UID: \"3b952e3e-75e2-42f1-9dbc-55b5a46dbc16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-490cf3a0-96jsg"
Oct 30 07:23:58.364433 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:58.364414    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b952e3e-75e2-42f1-9dbc-55b5a46dbc16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-490cf3a0-96jsg\" (UID: \"3b952e3e-75e2-42f1-9dbc-55b5a46dbc16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-490cf3a0-96jsg"
Oct 30 07:23:58.366061 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:58.366042    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b952e3e-75e2-42f1-9dbc-55b5a46dbc16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-490cf3a0-96jsg\" (UID: \"3b952e3e-75e2-42f1-9dbc-55b5a46dbc16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-490cf3a0-96jsg"
Oct 30 07:23:58.403411 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:58.403383    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k4h8\" (UniqueName: \"kubernetes.io/projected/3b952e3e-75e2-42f1-9dbc-55b5a46dbc16-kube-api-access-6k4h8\") pod \"maas-api-revoke-sub-490cf3a0-96jsg\" (UID: \"3b952e3e-75e2-42f1-9dbc-55b5a46dbc16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-490cf3a0-96jsg"
Oct 30 07:23:58.514996 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:58.514920    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-490cf3a0-96jsg"
Oct 30 07:23:58.852502 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:58.852471    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-490cf3a0-96jsg"]
Oct 30 07:23:58.938488 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:58.938448    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-490cf3a0-96jsg" event={"ID":"3b952e3e-75e2-42f1-9dbc-55b5a46dbc16","Type":"ContainerStarted","Data":"aaf17b61182942e6189aaf0001a247b057da0ec6a2b72a9cac7994f7d7c9c8d1"}
Oct 30 07:23:59.945642 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:59.945567    6224 generic.go:356] "Generic (PLEG): container finished" podID="3b952e3e-75e2-42f1-9dbc-55b5a46dbc16" containerID="943bbdd422f55c8aa8f5e48542ceb9b64e1724e6345d1c8bee0593a3691d98d2" exitCode=0
Oct 30 07:23:59.945642 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:23:59.945608    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-490cf3a0-96jsg" event={"ID":"3b952e3e-75e2-42f1-9dbc-55b5a46dbc16","Type":"ContainerDied","Data":"943bbdd422f55c8aa8f5e48542ceb9b64e1724e6345d1c8bee0593a3691d98d2"}
Oct 30 07:24:01.101876 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:01.101838    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-490cf3a0-96jsg"
Oct 30 07:24:01.202194 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:01.202125    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b952e3e-75e2-42f1-9dbc-55b5a46dbc16-kube-api-access-6k4h8\" (UniqueName: \"kubernetes.io/projected/3b952e3e-75e2-42f1-9dbc-55b5a46dbc16-kube-api-access-6k4h8\") pod \"3b952e3e-75e2-42f1-9dbc-55b5a46dbc16\" (UID: \"3b952e3e-75e2-42f1-9dbc-55b5a46dbc16\") "
Oct 30 07:24:01.202337 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:01.202303    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b952e3e-75e2-42f1-9dbc-55b5a46dbc16-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b952e3e-75e2-42f1-9dbc-55b5a46dbc16-maas-api-xks-ca\") pod \"3b952e3e-75e2-42f1-9dbc-55b5a46dbc16\" (UID: \"3b952e3e-75e2-42f1-9dbc-55b5a46dbc16\") "
Oct 30 07:24:01.202386 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:01.202366    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b952e3e-75e2-42f1-9dbc-55b5a46dbc16-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b952e3e-75e2-42f1-9dbc-55b5a46dbc16-maas-api-openshift-ca\") pod \"3b952e3e-75e2-42f1-9dbc-55b5a46dbc16\" (UID: \"3b952e3e-75e2-42f1-9dbc-55b5a46dbc16\") "
Oct 30 07:24:01.202779 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:01.202751    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b952e3e-75e2-42f1-9dbc-55b5a46dbc16-maas-api-openshift-ca" pod "3b952e3e-75e2-42f1-9dbc-55b5a46dbc16" (UID: "3b952e3e-75e2-42f1-9dbc-55b5a46dbc16"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:24:01.204270 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:01.204247    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b952e3e-75e2-42f1-9dbc-55b5a46dbc16-kube-api-access-6k4h8" pod "3b952e3e-75e2-42f1-9dbc-55b5a46dbc16" (UID: "3b952e3e-75e2-42f1-9dbc-55b5a46dbc16"). InnerVolumeSpecName "kube-api-access-6k4h8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:24:01.204722 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:01.204695    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b952e3e-75e2-42f1-9dbc-55b5a46dbc16-maas-api-xks-ca" pod "3b952e3e-75e2-42f1-9dbc-55b5a46dbc16" (UID: "3b952e3e-75e2-42f1-9dbc-55b5a46dbc16"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:24:01.303381 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:01.303355    6224 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b952e3e-75e2-42f1-9dbc-55b5a46dbc16-maas-api-xks-ca\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:24:01.303381 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:01.303380    6224 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b952e3e-75e2-42f1-9dbc-55b5a46dbc16-maas-api-openshift-ca\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:24:01.303545 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:01.303390    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6k4h8\" (UniqueName: \"kubernetes.io/projected/3b952e3e-75e2-42f1-9dbc-55b5a46dbc16-kube-api-access-6k4h8\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:24:01.858661 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:24:01.858628    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:24:01.958612 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:01.958582    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-490cf3a0-96jsg"
Oct 30 07:24:01.958757 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:01.958581    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-490cf3a0-96jsg" event={"ID":"3b952e3e-75e2-42f1-9dbc-55b5a46dbc16","Type":"ContainerDied","Data":"aaf17b61182942e6189aaf0001a247b057da0ec6a2b72a9cac7994f7d7c9c8d1"}
Oct 30 07:24:01.958757 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:01.958713    6224 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aaf17b61182942e6189aaf0001a247b057da0ec6a2b72a9cac7994f7d7c9c8d1"
Oct 30 07:24:16.057477 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:16.057436    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:24:25.509376 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:25.509343    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vzfkb"]
Oct 30 07:24:25.509862 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:25.509568    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vzfkb" podUID="233f4cde-e81e-4c82-8144-7bf56a77c876" containerName="istio-proxy" containerID="cri-o://9bcacd625bef8e858de20e58316b6ff6856458affc8d1ef833e34dada225e9a6" gracePeriod=30
Oct 30 07:24:25.511506 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:25.511478    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lkxfq"]
Oct 30 07:24:25.511745 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:25.511703    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lkxfq" podUID="0495c412-7a58-46b2-9ccc-c6abc6611c97" containerName="istio-proxy" containerID="cri-o://bf9cdb7c8c3fa411d77858ef931ee15818c0e1eee3e798b15572302eabd9ae2a" gracePeriod=30
Oct 30 07:24:30.773986 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:30.773953    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vzfkb"
Oct 30 07:24:30.814506 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:30.814484    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lkxfq"
Oct 30 07:24:30.919418 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:30.919379    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0495c412-7a58-46b2-9ccc-c6abc6611c97-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0495c412-7a58-46b2-9ccc-c6abc6611c97-credential-socket\") pod \"0495c412-7a58-46b2-9ccc-c6abc6611c97\" (UID: \"0495c412-7a58-46b2-9ccc-c6abc6611c97\") "
Oct 30 07:24:30.919418 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:30.919417    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0495c412-7a58-46b2-9ccc-c6abc6611c97-kube-api-access-tppff\" (UniqueName: \"kubernetes.io/projected/0495c412-7a58-46b2-9ccc-c6abc6611c97-kube-api-access-tppff\") pod \"0495c412-7a58-46b2-9ccc-c6abc6611c97\" (UID: \"0495c412-7a58-46b2-9ccc-c6abc6611c97\") "
Oct 30 07:24:30.919657 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:30.919466    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/233f4cde-e81e-4c82-8144-7bf56a77c876-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/233f4cde-e81e-4c82-8144-7bf56a77c876-credential-socket\") pod \"233f4cde-e81e-4c82-8144-7bf56a77c876\" (UID: \"233f4cde-e81e-4c82-8144-7bf56a77c876\") "
Oct 30 07:24:30.919657 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:30.919542    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0495c412-7a58-46b2-9ccc-c6abc6611c97-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0495c412-7a58-46b2-9ccc-c6abc6611c97-istio-podinfo\") pod \"0495c412-7a58-46b2-9ccc-c6abc6611c97\" (UID: \"0495c412-7a58-46b2-9ccc-c6abc6611c97\") "
Oct 30 07:24:30.919657 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:30.919581    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/233f4cde-e81e-4c82-8144-7bf56a77c876-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/233f4cde-e81e-4c82-8144-7bf56a77c876-workload-certs\") pod \"233f4cde-e81e-4c82-8144-7bf56a77c876\" (UID: \"233f4cde-e81e-4c82-8144-7bf56a77c876\") "
Oct 30 07:24:30.919657 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:30.919645    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/233f4cde-e81e-4c82-8144-7bf56a77c876-kube-api-access-xgnhd\" (UniqueName: \"kubernetes.io/projected/233f4cde-e81e-4c82-8144-7bf56a77c876-kube-api-access-xgnhd\") pod \"233f4cde-e81e-4c82-8144-7bf56a77c876\" (UID: \"233f4cde-e81e-4c82-8144-7bf56a77c876\") "
Oct 30 07:24:30.919892 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:30.919697    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/233f4cde-e81e-4c82-8144-7bf56a77c876-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/233f4cde-e81e-4c82-8144-7bf56a77c876-istio-envoy\") pod \"233f4cde-e81e-4c82-8144-7bf56a77c876\" (UID: \"233f4cde-e81e-4c82-8144-7bf56a77c876\") "
Oct 30 07:24:30.919892 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:30.919706    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0495c412-7a58-46b2-9ccc-c6abc6611c97-credential-socket" pod "0495c412-7a58-46b2-9ccc-c6abc6611c97" (UID: "0495c412-7a58-46b2-9ccc-c6abc6611c97"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:24:30.919892 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:30.919761    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/233f4cde-e81e-4c82-8144-7bf56a77c876-credential-socket" pod "233f4cde-e81e-4c82-8144-7bf56a77c876" (UID: "233f4cde-e81e-4c82-8144-7bf56a77c876"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:24:30.920063 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:30.919893    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0495c412-7a58-46b2-9ccc-c6abc6611c97-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0495c412-7a58-46b2-9ccc-c6abc6611c97-istio-envoy\") pod \"0495c412-7a58-46b2-9ccc-c6abc6611c97\" (UID: \"0495c412-7a58-46b2-9ccc-c6abc6611c97\") "
Oct 30 07:24:30.920063 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:30.919933    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/233f4cde-e81e-4c82-8144-7bf56a77c876-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/233f4cde-e81e-4c82-8144-7bf56a77c876-workload-socket\") pod \"233f4cde-e81e-4c82-8144-7bf56a77c876\" (UID: \"233f4cde-e81e-4c82-8144-7bf56a77c876\") "
Oct 30 07:24:30.920063 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:30.919962    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0495c412-7a58-46b2-9ccc-c6abc6611c97-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0495c412-7a58-46b2-9ccc-c6abc6611c97-istio-data\") pod \"0495c412-7a58-46b2-9ccc-c6abc6611c97\" (UID: \"0495c412-7a58-46b2-9ccc-c6abc6611c97\") "
Oct 30 07:24:30.920063 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:30.919987    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0495c412-7a58-46b2-9ccc-c6abc6611c97-istio-token\" (UniqueName: \"kubernetes.io/projected/0495c412-7a58-46b2-9ccc-c6abc6611c97-istio-token\") pod \"0495c412-7a58-46b2-9ccc-c6abc6611c97\" (UID: \"0495c412-7a58-46b2-9ccc-c6abc6611c97\") "
Oct 30 07:24:30.920063 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:30.920052    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0495c412-7a58-46b2-9ccc-c6abc6611c97-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0495c412-7a58-46b2-9ccc-c6abc6611c97-workload-certs\") pod \"0495c412-7a58-46b2-9ccc-c6abc6611c97\" (UID: \"0495c412-7a58-46b2-9ccc-c6abc6611c97\") "
Oct 30 07:24:30.920319 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:30.920087    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0495c412-7a58-46b2-9ccc-c6abc6611c97-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0495c412-7a58-46b2-9ccc-c6abc6611c97-workload-socket\") pod \"0495c412-7a58-46b2-9ccc-c6abc6611c97\" (UID: \"0495c412-7a58-46b2-9ccc-c6abc6611c97\") "
Oct 30 07:24:30.920319 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:30.920136    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/233f4cde-e81e-4c82-8144-7bf56a77c876-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/233f4cde-e81e-4c82-8144-7bf56a77c876-istio-podinfo\") pod \"233f4cde-e81e-4c82-8144-7bf56a77c876\" (UID: \"233f4cde-e81e-4c82-8144-7bf56a77c876\") "
Oct 30 07:24:30.920319 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:30.920191    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/233f4cde-e81e-4c82-8144-7bf56a77c876-workload-certs" pod "233f4cde-e81e-4c82-8144-7bf56a77c876" (UID: "233f4cde-e81e-4c82-8144-7bf56a77c876"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:24:30.920319 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:30.920220    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/233f4cde-e81e-4c82-8144-7bf56a77c876-istio-token\" (UniqueName: \"kubernetes.io/projected/233f4cde-e81e-4c82-8144-7bf56a77c876-istio-token\") pod \"233f4cde-e81e-4c82-8144-7bf56a77c876\" (UID: \"233f4cde-e81e-4c82-8144-7bf56a77c876\") "
Oct 30 07:24:30.920319 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:30.920269    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0495c412-7a58-46b2-9ccc-c6abc6611c97-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0495c412-7a58-46b2-9ccc-c6abc6611c97-istiod-ca-cert\") pod \"0495c412-7a58-46b2-9ccc-c6abc6611c97\" (UID: \"0495c412-7a58-46b2-9ccc-c6abc6611c97\") "
Oct 30 07:24:30.920319 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:30.920315    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/233f4cde-e81e-4c82-8144-7bf56a77c876-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/233f4cde-e81e-4c82-8144-7bf56a77c876-istio-data\") pod \"233f4cde-e81e-4c82-8144-7bf56a77c876\" (UID: \"233f4cde-e81e-4c82-8144-7bf56a77c876\") "
Oct 30 07:24:30.920623 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:30.920369    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/233f4cde-e81e-4c82-8144-7bf56a77c876-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/233f4cde-e81e-4c82-8144-7bf56a77c876-istiod-ca-cert\") pod \"233f4cde-e81e-4c82-8144-7bf56a77c876\" (UID: \"233f4cde-e81e-4c82-8144-7bf56a77c876\") "
Oct 30 07:24:30.920623 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:30.920562    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/233f4cde-e81e-4c82-8144-7bf56a77c876-workload-socket" pod "233f4cde-e81e-4c82-8144-7bf56a77c876" (UID: "233f4cde-e81e-4c82-8144-7bf56a77c876"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:24:30.921155 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:30.921131    6224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/233f4cde-e81e-4c82-8144-7bf56a77c876-workload-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:24:30.921711 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:30.921693    6224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/233f4cde-e81e-4c82-8144-7bf56a77c876-workload-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:24:30.921917 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:30.921898    6224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0495c412-7a58-46b2-9ccc-c6abc6611c97-credential-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:24:30.922163 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:30.922147    6224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/233f4cde-e81e-4c82-8144-7bf56a77c876-credential-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:24:30.922309 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:30.921208    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0495c412-7a58-46b2-9ccc-c6abc6611c97-istio-data" pod "0495c412-7a58-46b2-9ccc-c6abc6611c97" (UID: "0495c412-7a58-46b2-9ccc-c6abc6611c97"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:24:30.922309 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:30.921351    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0495c412-7a58-46b2-9ccc-c6abc6611c97-workload-certs" pod "0495c412-7a58-46b2-9ccc-c6abc6611c97" (UID: "0495c412-7a58-46b2-9ccc-c6abc6611c97"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:24:30.922491 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:30.921629    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0495c412-7a58-46b2-9ccc-c6abc6611c97-istiod-ca-cert" pod "0495c412-7a58-46b2-9ccc-c6abc6611c97" (UID: "0495c412-7a58-46b2-9ccc-c6abc6611c97"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:24:30.922491 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:30.921695    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/233f4cde-e81e-4c82-8144-7bf56a77c876-istio-data" pod "233f4cde-e81e-4c82-8144-7bf56a77c876" (UID: "233f4cde-e81e-4c82-8144-7bf56a77c876"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:24:30.922491 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:30.921865    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/233f4cde-e81e-4c82-8144-7bf56a77c876-istiod-ca-cert" pod "233f4cde-e81e-4c82-8144-7bf56a77c876" (UID: "233f4cde-e81e-4c82-8144-7bf56a77c876"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:24:30.922491 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:30.922096    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0495c412-7a58-46b2-9ccc-c6abc6611c97-workload-socket" pod "0495c412-7a58-46b2-9ccc-c6abc6611c97" (UID: "0495c412-7a58-46b2-9ccc-c6abc6611c97"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:24:30.923736 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:30.923704    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0495c412-7a58-46b2-9ccc-c6abc6611c97-istio-token" pod "0495c412-7a58-46b2-9ccc-c6abc6611c97" (UID: "0495c412-7a58-46b2-9ccc-c6abc6611c97"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:24:30.923736 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:30.923713    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/233f4cde-e81e-4c82-8144-7bf56a77c876-istio-envoy" pod "233f4cde-e81e-4c82-8144-7bf56a77c876" (UID: "233f4cde-e81e-4c82-8144-7bf56a77c876"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:24:30.923932 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:30.923736    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/233f4cde-e81e-4c82-8144-7bf56a77c876-kube-api-access-xgnhd" pod "233f4cde-e81e-4c82-8144-7bf56a77c876" (UID: "233f4cde-e81e-4c82-8144-7bf56a77c876"). InnerVolumeSpecName "kube-api-access-xgnhd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:24:30.923932 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:30.923785    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0495c412-7a58-46b2-9ccc-c6abc6611c97-istio-envoy" pod "0495c412-7a58-46b2-9ccc-c6abc6611c97" (UID: "0495c412-7a58-46b2-9ccc-c6abc6611c97"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:24:30.923932 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:30.923798    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0495c412-7a58-46b2-9ccc-c6abc6611c97-kube-api-access-tppff" pod "0495c412-7a58-46b2-9ccc-c6abc6611c97" (UID: "0495c412-7a58-46b2-9ccc-c6abc6611c97"). InnerVolumeSpecName "kube-api-access-tppff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:24:30.923932 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:30.923856    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/233f4cde-e81e-4c82-8144-7bf56a77c876-istio-podinfo" pod "233f4cde-e81e-4c82-8144-7bf56a77c876" (UID: "233f4cde-e81e-4c82-8144-7bf56a77c876"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:24:30.924086 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:30.924036    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/233f4cde-e81e-4c82-8144-7bf56a77c876-istio-token" pod "233f4cde-e81e-4c82-8144-7bf56a77c876" (UID: "233f4cde-e81e-4c82-8144-7bf56a77c876"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:24:30.924936 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:30.924914    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0495c412-7a58-46b2-9ccc-c6abc6611c97-istio-podinfo" pod "0495c412-7a58-46b2-9ccc-c6abc6611c97" (UID: "0495c412-7a58-46b2-9ccc-c6abc6611c97"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:24:31.023549 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:31.023512    6224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/233f4cde-e81e-4c82-8144-7bf56a77c876-istio-data\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:24:31.023549 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:31.023543    6224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/233f4cde-e81e-4c82-8144-7bf56a77c876-istiod-ca-cert\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:24:31.023549 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:31.023552    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tppff\" (UniqueName: \"kubernetes.io/projected/0495c412-7a58-46b2-9ccc-c6abc6611c97-kube-api-access-tppff\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:24:31.023784 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:31.023566    6224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0495c412-7a58-46b2-9ccc-c6abc6611c97-istio-podinfo\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:24:31.023784 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:31.023575    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgnhd\" (UniqueName: \"kubernetes.io/projected/233f4cde-e81e-4c82-8144-7bf56a77c876-kube-api-access-xgnhd\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:24:31.023784 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:31.023583    6224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/233f4cde-e81e-4c82-8144-7bf56a77c876-istio-envoy\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:24:31.023784 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:31.023593    6224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0495c412-7a58-46b2-9ccc-c6abc6611c97-istio-envoy\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:24:31.023784 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:31.023601    6224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0495c412-7a58-46b2-9ccc-c6abc6611c97-istio-data\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:24:31.023784 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:31.023612    6224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0495c412-7a58-46b2-9ccc-c6abc6611c97-istio-token\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:24:31.023784 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:31.023619    6224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0495c412-7a58-46b2-9ccc-c6abc6611c97-workload-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:24:31.023784 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:31.023626    6224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0495c412-7a58-46b2-9ccc-c6abc6611c97-workload-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:24:31.023784 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:31.023634    6224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/233f4cde-e81e-4c82-8144-7bf56a77c876-istio-podinfo\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:24:31.023784 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:31.023641    6224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/233f4cde-e81e-4c82-8144-7bf56a77c876-istio-token\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:24:31.023784 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:31.023649    6224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0495c412-7a58-46b2-9ccc-c6abc6611c97-istiod-ca-cert\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:24:31.444485 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:31.444449    6224 generic.go:356] "Generic (PLEG): container finished" podID="233f4cde-e81e-4c82-8144-7bf56a77c876" containerID="9bcacd625bef8e858de20e58316b6ff6856458affc8d1ef833e34dada225e9a6" exitCode=0
Oct 30 07:24:31.444662 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:31.444496    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vzfkb"
Oct 30 07:24:31.444662 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:31.444542    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vzfkb" event={"ID":"233f4cde-e81e-4c82-8144-7bf56a77c876","Type":"ContainerDied","Data":"9bcacd625bef8e858de20e58316b6ff6856458affc8d1ef833e34dada225e9a6"}
Oct 30 07:24:31.444662 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:31.444601    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vzfkb" event={"ID":"233f4cde-e81e-4c82-8144-7bf56a77c876","Type":"ContainerDied","Data":"59cd68cace3f580384f4b80fadd9549a7a4899d366b54004e7c7c1e373a5fda4"}
Oct 30 07:24:31.444662 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:31.444629    6224 scope.go:122] "RemoveContainer" containerID="9bcacd625bef8e858de20e58316b6ff6856458affc8d1ef833e34dada225e9a6"
Oct 30 07:24:31.446033 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:31.446010    6224 generic.go:356] "Generic (PLEG): container finished" podID="0495c412-7a58-46b2-9ccc-c6abc6611c97" containerID="bf9cdb7c8c3fa411d77858ef931ee15818c0e1eee3e798b15572302eabd9ae2a" exitCode=0
Oct 30 07:24:31.446148 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:31.446035    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lkxfq" event={"ID":"0495c412-7a58-46b2-9ccc-c6abc6611c97","Type":"ContainerDied","Data":"bf9cdb7c8c3fa411d77858ef931ee15818c0e1eee3e798b15572302eabd9ae2a"}
Oct 30 07:24:31.446148 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:31.446049    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lkxfq"
Oct 30 07:24:31.446148 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:31.446087    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lkxfq" event={"ID":"0495c412-7a58-46b2-9ccc-c6abc6611c97","Type":"ContainerDied","Data":"dcece42a068a286cae899ea09473f1eda3f65e08829ea49b72f5898d3bc14abb"}
Oct 30 07:24:31.454990 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:31.454971    6224 scope.go:122] "RemoveContainer" containerID="9bcacd625bef8e858de20e58316b6ff6856458affc8d1ef833e34dada225e9a6"
Oct 30 07:24:31.455343 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:24:31.455253    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bcacd625bef8e858de20e58316b6ff6856458affc8d1ef833e34dada225e9a6\": container with ID starting with 9bcacd625bef8e858de20e58316b6ff6856458affc8d1ef833e34dada225e9a6 not found: ID does not exist" containerID="9bcacd625bef8e858de20e58316b6ff6856458affc8d1ef833e34dada225e9a6"
Oct 30 07:24:31.455343 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:31.455286    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bcacd625bef8e858de20e58316b6ff6856458affc8d1ef833e34dada225e9a6"} err="failed to get container status \"9bcacd625bef8e858de20e58316b6ff6856458affc8d1ef833e34dada225e9a6\": rpc error: code = NotFound desc = could not find container \"9bcacd625bef8e858de20e58316b6ff6856458affc8d1ef833e34dada225e9a6\": container with ID starting with 9bcacd625bef8e858de20e58316b6ff6856458affc8d1ef833e34dada225e9a6 not found: ID does not exist"
Oct 30 07:24:31.455343 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:31.455310    6224 scope.go:122] "RemoveContainer" containerID="bf9cdb7c8c3fa411d77858ef931ee15818c0e1eee3e798b15572302eabd9ae2a"
Oct 30 07:24:31.464006 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:31.463983    6224 scope.go:122] "RemoveContainer" containerID="bf9cdb7c8c3fa411d77858ef931ee15818c0e1eee3e798b15572302eabd9ae2a"
Oct 30 07:24:31.464244 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:24:31.464223    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf9cdb7c8c3fa411d77858ef931ee15818c0e1eee3e798b15572302eabd9ae2a\": container with ID starting with bf9cdb7c8c3fa411d77858ef931ee15818c0e1eee3e798b15572302eabd9ae2a not found: ID does not exist" containerID="bf9cdb7c8c3fa411d77858ef931ee15818c0e1eee3e798b15572302eabd9ae2a"
Oct 30 07:24:31.464298 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:31.464251    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf9cdb7c8c3fa411d77858ef931ee15818c0e1eee3e798b15572302eabd9ae2a"} err="failed to get container status \"bf9cdb7c8c3fa411d77858ef931ee15818c0e1eee3e798b15572302eabd9ae2a\": rpc error: code = NotFound desc = could not find container \"bf9cdb7c8c3fa411d77858ef931ee15818c0e1eee3e798b15572302eabd9ae2a\": container with ID starting with bf9cdb7c8c3fa411d77858ef931ee15818c0e1eee3e798b15572302eabd9ae2a not found: ID does not exist"
Oct 30 07:24:31.503540 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:31.503514    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lkxfq"]
Oct 30 07:24:31.516220 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:31.516198    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lkxfq"]
Oct 30 07:24:31.560776 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:31.560752    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vzfkb"]
Oct 30 07:24:31.571399 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:31.571379    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vzfkb"]
Oct 30 07:24:31.858403 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:24:31.858370    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:24:33.502202 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:33.502130    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0495c412-7a58-46b2-9ccc-c6abc6611c97" path="/var/lib/kubelet/pods/0495c412-7a58-46b2-9ccc-c6abc6611c97/volumes"
Oct 30 07:24:33.502781 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:33.502762    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="233f4cde-e81e-4c82-8144-7bf56a77c876" path="/var/lib/kubelet/pods/233f4cde-e81e-4c82-8144-7bf56a77c876/volumes"
Oct 30 07:24:52.446435 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:52.446403    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v"]
Oct 30 07:24:52.446937 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:52.446921    6224 cpu_manager.go:414] "containerMap: removing container" podUID="0495c412-7a58-46b2-9ccc-c6abc6611c97" containerName="istio-proxy"
Oct 30 07:24:52.446990 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:52.446939    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0495c412-7a58-46b2-9ccc-c6abc6611c97" containerName="istio-proxy"
Oct 30 07:24:52.446990 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:52.446965    6224 cpu_manager.go:414] "containerMap: removing container" podUID="3b952e3e-75e2-42f1-9dbc-55b5a46dbc16" containerName="revoke-keys"
Oct 30 07:24:52.446990 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:52.446970    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b952e3e-75e2-42f1-9dbc-55b5a46dbc16" containerName="revoke-keys"
Oct 30 07:24:52.447088 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:52.446992    6224 cpu_manager.go:414] "containerMap: removing container" podUID="233f4cde-e81e-4c82-8144-7bf56a77c876" containerName="istio-proxy"
Oct 30 07:24:52.447088 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:52.446998    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="233f4cde-e81e-4c82-8144-7bf56a77c876" containerName="istio-proxy"
Oct 30 07:24:52.447088 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:52.447074    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="0495c412-7a58-46b2-9ccc-c6abc6611c97" containerName="istio-proxy"
Oct 30 07:24:52.447088 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:52.447083    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b952e3e-75e2-42f1-9dbc-55b5a46dbc16" containerName="revoke-keys"
Oct 30 07:24:52.447205 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:52.447093    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="233f4cde-e81e-4c82-8144-7bf56a77c876" containerName="istio-proxy"
Oct 30 07:24:52.450529 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:52.450511    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v"
Oct 30 07:24:52.475067 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:52.475021    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v"]
Oct 30 07:24:52.541025 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:52.540990    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea11f788-17b6-4758-945d-fbb0d273e840-istio-podinfo\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v\" (UID: \"ea11f788-17b6-4758-945d-fbb0d273e840\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v"
Oct 30 07:24:52.541161 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:52.541035    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea11f788-17b6-4758-945d-fbb0d273e840-istio-envoy\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v\" (UID: \"ea11f788-17b6-4758-945d-fbb0d273e840\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v"
Oct 30 07:24:52.541161 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:52.541144    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea11f788-17b6-4758-945d-fbb0d273e840-istio-token\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v\" (UID: \"ea11f788-17b6-4758-945d-fbb0d273e840\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v"
Oct 30 07:24:52.541252 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:52.541194    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea11f788-17b6-4758-945d-fbb0d273e840-istiod-ca-cert\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v\" (UID: \"ea11f788-17b6-4758-945d-fbb0d273e840\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v"
Oct 30 07:24:52.541286 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:52.541271    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea11f788-17b6-4758-945d-fbb0d273e840-credential-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v\" (UID: \"ea11f788-17b6-4758-945d-fbb0d273e840\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v"
Oct 30 07:24:52.541323 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:52.541300    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea11f788-17b6-4758-945d-fbb0d273e840-istio-data\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v\" (UID: \"ea11f788-17b6-4758-945d-fbb0d273e840\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v"
Oct 30 07:24:52.541385 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:52.541370    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4pqm\" (UniqueName: \"kubernetes.io/projected/ea11f788-17b6-4758-945d-fbb0d273e840-kube-api-access-h4pqm\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v\" (UID: \"ea11f788-17b6-4758-945d-fbb0d273e840\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v"
Oct 30 07:24:52.541420 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:52.541395    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea11f788-17b6-4758-945d-fbb0d273e840-workload-certs\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v\" (UID: \"ea11f788-17b6-4758-945d-fbb0d273e840\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v"
Oct 30 07:24:52.541511 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:52.541494    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea11f788-17b6-4758-945d-fbb0d273e840-workload-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v\" (UID: \"ea11f788-17b6-4758-945d-fbb0d273e840\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v"
Oct 30 07:24:52.642603 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:52.642572    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea11f788-17b6-4758-945d-fbb0d273e840-istio-envoy\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v\" (UID: \"ea11f788-17b6-4758-945d-fbb0d273e840\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v"
Oct 30 07:24:52.642768 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:52.642650    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea11f788-17b6-4758-945d-fbb0d273e840-istio-token\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v\" (UID: \"ea11f788-17b6-4758-945d-fbb0d273e840\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v"
Oct 30 07:24:52.642768 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:52.642696    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea11f788-17b6-4758-945d-fbb0d273e840-istiod-ca-cert\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v\" (UID: \"ea11f788-17b6-4758-945d-fbb0d273e840\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v"
Oct 30 07:24:52.642768 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:52.642735    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea11f788-17b6-4758-945d-fbb0d273e840-credential-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v\" (UID: \"ea11f788-17b6-4758-945d-fbb0d273e840\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v"
Oct 30 07:24:52.642994 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:52.642766    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea11f788-17b6-4758-945d-fbb0d273e840-istio-data\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v\" (UID: \"ea11f788-17b6-4758-945d-fbb0d273e840\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v"
Oct 30 07:24:52.642994 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:52.642859    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4pqm\" (UniqueName: \"kubernetes.io/projected/ea11f788-17b6-4758-945d-fbb0d273e840-kube-api-access-h4pqm\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v\" (UID: \"ea11f788-17b6-4758-945d-fbb0d273e840\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v"
Oct 30 07:24:52.642994 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:52.642891    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea11f788-17b6-4758-945d-fbb0d273e840-workload-certs\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v\" (UID: \"ea11f788-17b6-4758-945d-fbb0d273e840\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v"
Oct 30 07:24:52.642994 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:52.642967    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea11f788-17b6-4758-945d-fbb0d273e840-workload-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v\" (UID: \"ea11f788-17b6-4758-945d-fbb0d273e840\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v"
Oct 30 07:24:52.643201 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:52.643017    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea11f788-17b6-4758-945d-fbb0d273e840-istio-podinfo\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v\" (UID: \"ea11f788-17b6-4758-945d-fbb0d273e840\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v"
Oct 30 07:24:52.643273 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:52.643247    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea11f788-17b6-4758-945d-fbb0d273e840-istio-data\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v\" (UID: \"ea11f788-17b6-4758-945d-fbb0d273e840\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v"
Oct 30 07:24:52.643379 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:52.643352    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea11f788-17b6-4758-945d-fbb0d273e840-credential-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v\" (UID: \"ea11f788-17b6-4758-945d-fbb0d273e840\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v"
Oct 30 07:24:52.643542 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:52.643521    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea11f788-17b6-4758-945d-fbb0d273e840-workload-certs\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v\" (UID: \"ea11f788-17b6-4758-945d-fbb0d273e840\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v"
Oct 30 07:24:52.643650 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:52.643613    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea11f788-17b6-4758-945d-fbb0d273e840-workload-socket\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v\" (UID: \"ea11f788-17b6-4758-945d-fbb0d273e840\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v"
Oct 30 07:24:52.643759 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:52.643739    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea11f788-17b6-4758-945d-fbb0d273e840-istiod-ca-cert\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v\" (UID: \"ea11f788-17b6-4758-945d-fbb0d273e840\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v"
Oct 30 07:24:52.645135 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:52.645117    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea11f788-17b6-4758-945d-fbb0d273e840-istio-envoy\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v\" (UID: \"ea11f788-17b6-4758-945d-fbb0d273e840\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v"
Oct 30 07:24:52.645629 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:52.645608    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea11f788-17b6-4758-945d-fbb0d273e840-istio-podinfo\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v\" (UID: \"ea11f788-17b6-4758-945d-fbb0d273e840\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v"
Oct 30 07:24:52.667225 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:52.667198    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea11f788-17b6-4758-945d-fbb0d273e840-istio-token\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v\" (UID: \"ea11f788-17b6-4758-945d-fbb0d273e840\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v"
Oct 30 07:24:52.667493 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:52.667474    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4pqm\" (UniqueName: \"kubernetes.io/projected/ea11f788-17b6-4758-945d-fbb0d273e840-kube-api-access-h4pqm\") pod \"e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v\" (UID: \"ea11f788-17b6-4758-945d-fbb0d273e840\") " pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v"
Oct 30 07:24:52.764990 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:52.764927    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v"
Oct 30 07:24:53.076923 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:24:53.076894    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea11f788_17b6_4758_945d_fbb0d273e840.slice/crio-5a71bdb4be531fba6bbd5464f88337fb5b2e77bda7eac1ca08d1ff86c0104002 WatchSource:0}: Error finding container 5a71bdb4be531fba6bbd5464f88337fb5b2e77bda7eac1ca08d1ff86c0104002: Status 404 returned error can't find the container with id 5a71bdb4be531fba6bbd5464f88337fb5b2e77bda7eac1ca08d1ff86c0104002
Oct 30 07:24:53.077064 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:53.076960    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v"]
Oct 30 07:24:53.079240 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:53.079211    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:24:53.079331 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:53.079269    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:24:53.079331 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:53.079311    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:24:53.696118 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:53.696079    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v" event={"ID":"ea11f788-17b6-4758-945d-fbb0d273e840","Type":"ContainerStarted","Data":"a4b50c1feb0e7a59388b908e7a1a4da3d70d33efe27b59b3191567d8655cca46"}
Oct 30 07:24:53.696505 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:53.696171    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v" event={"ID":"ea11f788-17b6-4758-945d-fbb0d273e840","Type":"ContainerStarted","Data":"5a71bdb4be531fba6bbd5464f88337fb5b2e77bda7eac1ca08d1ff86c0104002"}
Oct 30 07:24:53.764435 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:53.764391    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v" podStartSLOduration=1.764376041 podStartE2EDuration="1.764376041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:24:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:24:53.762402131 +0000 UTC m=+1652.371809102" watchObservedRunningTime="2026-10-30 07:24:53.764376041 +0000 UTC m=+1652.373783013"
Oct 30 07:24:53.765180 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:53.765161    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v"
Oct 30 07:24:54.781832 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:54.775921    6224 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 30 07:24:54.781832 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:54.776009    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v" podUID="ea11f788-17b6-4758-945d-fbb0d273e840" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 30 07:24:54.889385 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:54.889335    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:24:55.766196 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:55.766159    6224 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 30 07:24:55.767198 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:55.767164    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v" podUID="ea11f788-17b6-4758-945d-fbb0d273e840" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 30 07:24:56.766222 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:56.766184    6224 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 30 07:24:56.766788 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:56.766757    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v" podUID="ea11f788-17b6-4758-945d-fbb0d273e840" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 30 07:24:57.158942 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:57.158894    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-06b26e/distinct-w0-06b26e-kserve-545cf57898-2fhhd" podUID="71294bfa-ac2c-40b7-b889-0527daa07f63" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:24:57.217834 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:57.209907    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw" podUID="0597696e-26b8-4ef8-8697-64d49b2b9398" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:24:57.226103 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:57.220501    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb6be4/simulator-w2-bb6be4-kserve-8485bbf954-dnbbz" podUID="6370f7e2-d6c8-4fb5-8198-32b38837ca31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Oct 30 07:24:57.258268 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:57.257909    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb6be4/premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t" podUID="7dd5133c-480f-4fa0-9dd1-27662004aae9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:24:57.324740 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:57.324693    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2bab4c/distinct-w4-2bab4c-kserve-6868d767c9-j7wlm" podUID="11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Oct 30 07:24:57.509287 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:57.509184    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2bab4c/simulator-w4-2bab4c-kserve-6db9444549-7nrk8" podUID="c7b09178-9ac7-492f-890b-5ff3da130981" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Oct 30 07:24:57.618937 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:57.618890    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/simulator-w3-2eda2c-kserve-56776b9849-9bqzm" podUID="482e6b7c-2231-423c-b54f-a54e56f16ab7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:24:57.741416 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:57.741371    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2bab4c/premium-w4-2bab4c-kserve-665887fc68-c6dgf" podUID="f9463476-fcf5-4d66-bfd2-b85ce4ed651f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:24:57.774670 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:57.774166    6224 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 30 07:24:57.774670 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:57.774237    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v" podUID="ea11f788-17b6-4758-945d-fbb0d273e840" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 30 07:24:57.842014 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:57.841951    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-06b26e/premium-w0-06b26e-kserve-c9dbd9879-b4jlk" podUID="1230dcae-e1cd-460d-86fc-bd40c8427b5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:24:58.162975 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:58.162926    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-06b26e/distinct-w0-06b26e-kserve-545cf57898-2fhhd" podUID="71294bfa-ac2c-40b7-b889-0527daa07f63" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Oct 30 07:24:58.165064 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:58.164023    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-06b26e/distinct-w0-06b26e-kserve-545cf57898-2fhhd"
Oct 30 07:24:58.230908 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:58.230351    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw" podUID="0597696e-26b8-4ef8-8697-64d49b2b9398" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 30 07:24:58.231161 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:58.231093    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw"
Oct 30 07:24:58.241887 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:58.232146    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb6be4/simulator-w2-bb6be4-kserve-8485bbf954-dnbbz" podUID="6370f7e2-d6c8-4fb5-8198-32b38837ca31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:24:58.241887 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:58.232467    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb6be4/simulator-w2-bb6be4-kserve-8485bbf954-dnbbz"
Oct 30 07:24:58.276482 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:58.276103    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb6be4/premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t" podUID="7dd5133c-480f-4fa0-9dd1-27662004aae9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Oct 30 07:24:58.277538 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:58.276775    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb6be4/premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t"
Oct 30 07:24:58.385831 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:58.311749    6224 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw" podUID="0597696e-26b8-4ef8-8697-64d49b2b9398" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:24:58.389323 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:58.335884    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2bab4c/distinct-w4-2bab4c-kserve-6868d767c9-j7wlm" podUID="11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:24:58.389837 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:58.353078    6224 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-bb6be4/premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t" podUID="7dd5133c-480f-4fa0-9dd1-27662004aae9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:24:58.389974 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:58.353503    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw"
Oct 30 07:24:58.397930 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:58.397908    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2bab4c/distinct-w4-2bab4c-kserve-6868d767c9-j7wlm"
Oct 30 07:24:58.523012 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:58.522194    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2bab4c/simulator-w4-2bab4c-kserve-6db9444549-7nrk8" podUID="c7b09178-9ac7-492f-890b-5ff3da130981" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:24:58.523012 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:58.522876    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2bab4c/simulator-w4-2bab4c-kserve-6db9444549-7nrk8"
Oct 30 07:24:58.623991 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:58.623936    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/simulator-w3-2eda2c-kserve-56776b9849-9bqzm" podUID="482e6b7c-2231-423c-b54f-a54e56f16ab7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:24:58.629353 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:58.629329    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2eda2c/simulator-w3-2eda2c-kserve-56776b9849-9bqzm"
Oct 30 07:24:58.651279 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:58.651236    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb6be4/premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t"
Oct 30 07:24:58.657137 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:58.653857    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a71a224-wsc72"]
Oct 30 07:24:58.658772 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:58.658746    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a71a224-wsc72"]
Oct 30 07:24:58.669828 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:58.659051    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a71a224-wsc72"
Oct 30 07:24:58.698828 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:58.695455    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:24:58.698828 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:58.695790    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 07:24:58.760147 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:58.760083    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2bab4c/premium-w4-2bab4c-kserve-665887fc68-c6dgf" podUID="f9463476-fcf5-4d66-bfd2-b85ce4ed651f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:24:58.791240 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:58.762983    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2bab4c/premium-w4-2bab4c-kserve-665887fc68-c6dgf"
Oct 30 07:24:58.814039 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:58.814004    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2bab4c/simulator-w4-2bab4c-kserve-6db9444549-7nrk8"
Oct 30 07:24:58.840668 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:58.840640    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb6be4/simulator-w2-bb6be4-kserve-8485bbf954-dnbbz"
Oct 30 07:24:58.841493 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:58.841460    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ce27afe-fbcc-4c82-9986-ab4aa4f575ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a71a224-wsc72\" (UID: \"4ce27afe-fbcc-4c82-9986-ab4aa4f575ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a71a224-wsc72"
Oct 30 07:24:58.841677 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:58.841658    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ce27afe-fbcc-4c82-9986-ab4aa4f575ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a71a224-wsc72\" (UID: \"4ce27afe-fbcc-4c82-9986-ab4aa4f575ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a71a224-wsc72"
Oct 30 07:24:58.841758 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:58.841715    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn87c\" (UniqueName: \"kubernetes.io/projected/4ce27afe-fbcc-4c82-9986-ab4aa4f575ee-kube-api-access-hn87c\") pod \"maas-api-revoke-sub-7a71a224-wsc72\" (UID: \"4ce27afe-fbcc-4c82-9986-ab4aa4f575ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a71a224-wsc72"
Oct 30 07:24:58.842503 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:58.842480    6224 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 30 07:24:58.842695 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:58.842659    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v" podUID="ea11f788-17b6-4758-945d-fbb0d273e840" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 30 07:24:58.864658 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:58.856088    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2bab4c/distinct-w4-2bab4c-kserve-6868d767c9-j7wlm"
Oct 30 07:24:58.864658 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:58.856138    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2eda2c/simulator-w3-2eda2c-kserve-56776b9849-9bqzm"
Oct 30 07:24:58.864658 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:58.859627    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-06b26e/distinct-w0-06b26e-kserve-545cf57898-2fhhd"
Oct 30 07:24:58.891390 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:58.891352    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2bab4c/premium-w4-2bab4c-kserve-665887fc68-c6dgf"
Oct 30 07:24:58.952828 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:58.952773    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ce27afe-fbcc-4c82-9986-ab4aa4f575ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a71a224-wsc72\" (UID: \"4ce27afe-fbcc-4c82-9986-ab4aa4f575ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a71a224-wsc72"
Oct 30 07:24:58.952989 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:58.952861    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn87c\" (UniqueName: \"kubernetes.io/projected/4ce27afe-fbcc-4c82-9986-ab4aa4f575ee-kube-api-access-hn87c\") pod \"maas-api-revoke-sub-7a71a224-wsc72\" (UID: \"4ce27afe-fbcc-4c82-9986-ab4aa4f575ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a71a224-wsc72"
Oct 30 07:24:58.952989 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:58.952912    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ce27afe-fbcc-4c82-9986-ab4aa4f575ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a71a224-wsc72\" (UID: \"4ce27afe-fbcc-4c82-9986-ab4aa4f575ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a71a224-wsc72"
Oct 30 07:24:58.955221 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:58.955192    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ce27afe-fbcc-4c82-9986-ab4aa4f575ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a71a224-wsc72\" (UID: \"4ce27afe-fbcc-4c82-9986-ab4aa4f575ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a71a224-wsc72"
Oct 30 07:24:58.956681 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:58.956652    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ce27afe-fbcc-4c82-9986-ab4aa4f575ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a71a224-wsc72\" (UID: \"4ce27afe-fbcc-4c82-9986-ab4aa4f575ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a71a224-wsc72"
Oct 30 07:24:58.998717 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:58.994637    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn87c\" (UniqueName: \"kubernetes.io/projected/4ce27afe-fbcc-4c82-9986-ab4aa4f575ee-kube-api-access-hn87c\") pod \"maas-api-revoke-sub-7a71a224-wsc72\" (UID: \"4ce27afe-fbcc-4c82-9986-ab4aa4f575ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a71a224-wsc72"
Oct 30 07:24:59.215569 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:59.215482    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a71a224-wsc72"
Oct 30 07:24:59.553532 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:59.553498    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a71a224-wsc72"]
Oct 30 07:24:59.553895 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:24:59.553864    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ce27afe_fbcc_4c82_9986_ab4aa4f575ee.slice/crio-beba93d1438cd64c097132f50e6c5b19a9d4fd9f833c5eb941856c6fb2d545cb WatchSource:0}: Error finding container beba93d1438cd64c097132f50e6c5b19a9d4fd9f833c5eb941856c6fb2d545cb: Status 404 returned error can't find the container with id beba93d1438cd64c097132f50e6c5b19a9d4fd9f833c5eb941856c6fb2d545cb
Oct 30 07:24:59.788832 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:59.787935    6224 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 30 07:24:59.788832 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:59.787999    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v" podUID="ea11f788-17b6-4758-945d-fbb0d273e840" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 30 07:24:59.912078 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:59.912044    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a71a224-wsc72" event={"ID":"4ce27afe-fbcc-4c82-9986-ab4aa4f575ee","Type":"ContainerStarted","Data":"beba93d1438cd64c097132f50e6c5b19a9d4fd9f833c5eb941856c6fb2d545cb"}
Oct 30 07:24:59.959023 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:24:59.958974    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a71a224-wsc72" podStartSLOduration=1.958961006 podStartE2EDuration="1.958961006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:24:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:24:59.957079627 +0000 UTC m=+1658.566486598" watchObservedRunningTime="2026-10-30 07:24:59.958961006 +0000 UTC m=+1658.568367975"
Oct 30 07:25:00.765587 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:00.765513    6224 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 30 07:25:00.765587 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:00.765565    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v" podUID="ea11f788-17b6-4758-945d-fbb0d273e840" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 30 07:25:00.816654 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:00.816624    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7"]
Oct 30 07:25:00.820427 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:00.820406    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7"
Oct 30 07:25:00.847646 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:00.847620    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7"]
Oct 30 07:25:00.900719 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:00.900690    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5611d3ae-2361-411e-903f-ebd0d5b5fba5-workload-certs\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7\" (UID: \"5611d3ae-2361-411e-903f-ebd0d5b5fba5\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7"
Oct 30 07:25:00.900909 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:00.900728    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5611d3ae-2361-411e-903f-ebd0d5b5fba5-credential-socket\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7\" (UID: \"5611d3ae-2361-411e-903f-ebd0d5b5fba5\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7"
Oct 30 07:25:00.900909 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:00.900895    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5611d3ae-2361-411e-903f-ebd0d5b5fba5-istiod-ca-cert\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7\" (UID: \"5611d3ae-2361-411e-903f-ebd0d5b5fba5\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7"
Oct 30 07:25:00.900994 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:00.900967    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5611d3ae-2361-411e-903f-ebd0d5b5fba5-workload-socket\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7\" (UID: \"5611d3ae-2361-411e-903f-ebd0d5b5fba5\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7"
Oct 30 07:25:00.901093 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:00.901075    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h58d\" (UniqueName: \"kubernetes.io/projected/5611d3ae-2361-411e-903f-ebd0d5b5fba5-kube-api-access-2h58d\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7\" (UID: \"5611d3ae-2361-411e-903f-ebd0d5b5fba5\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7"
Oct 30 07:25:00.901128 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:00.901119    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5611d3ae-2361-411e-903f-ebd0d5b5fba5-istio-token\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7\" (UID: \"5611d3ae-2361-411e-903f-ebd0d5b5fba5\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7"
Oct 30 07:25:00.901226 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:00.901210    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5611d3ae-2361-411e-903f-ebd0d5b5fba5-istio-data\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7\" (UID: \"5611d3ae-2361-411e-903f-ebd0d5b5fba5\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7"
Oct 30 07:25:00.901265 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:00.901243    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5611d3ae-2361-411e-903f-ebd0d5b5fba5-istio-envoy\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7\" (UID: \"5611d3ae-2361-411e-903f-ebd0d5b5fba5\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7"
Oct 30 07:25:00.901357 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:00.901338    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5611d3ae-2361-411e-903f-ebd0d5b5fba5-istio-podinfo\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7\" (UID: \"5611d3ae-2361-411e-903f-ebd0d5b5fba5\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7"
Oct 30 07:25:00.924380 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:00.924355    6224 generic.go:356] "Generic (PLEG): container finished" podID="4ce27afe-fbcc-4c82-9986-ab4aa4f575ee" containerID="1dc32a4e513facb27140c6505e4e2bb8b313e6ece9283b7f8902dbb9f77472b5" exitCode=0
Oct 30 07:25:00.924797 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:00.924432    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a71a224-wsc72" event={"ID":"4ce27afe-fbcc-4c82-9986-ab4aa4f575ee","Type":"ContainerDied","Data":"1dc32a4e513facb27140c6505e4e2bb8b313e6ece9283b7f8902dbb9f77472b5"}
Oct 30 07:25:01.003092 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:01.003054    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h58d\" (UniqueName: \"kubernetes.io/projected/5611d3ae-2361-411e-903f-ebd0d5b5fba5-kube-api-access-2h58d\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7\" (UID: \"5611d3ae-2361-411e-903f-ebd0d5b5fba5\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7"
Oct 30 07:25:01.003289 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:01.003113    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5611d3ae-2361-411e-903f-ebd0d5b5fba5-istio-token\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7\" (UID: \"5611d3ae-2361-411e-903f-ebd0d5b5fba5\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7"
Oct 30 07:25:01.003289 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:01.003157    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5611d3ae-2361-411e-903f-ebd0d5b5fba5-istio-data\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7\" (UID: \"5611d3ae-2361-411e-903f-ebd0d5b5fba5\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7"
Oct 30 07:25:01.003289 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:01.003190    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5611d3ae-2361-411e-903f-ebd0d5b5fba5-istio-envoy\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7\" (UID: \"5611d3ae-2361-411e-903f-ebd0d5b5fba5\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7"
Oct 30 07:25:01.003289 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:01.003212    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5611d3ae-2361-411e-903f-ebd0d5b5fba5-istio-podinfo\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7\" (UID: \"5611d3ae-2361-411e-903f-ebd0d5b5fba5\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7"
Oct 30 07:25:01.003289 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:01.003241    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5611d3ae-2361-411e-903f-ebd0d5b5fba5-workload-certs\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7\" (UID: \"5611d3ae-2361-411e-903f-ebd0d5b5fba5\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7"
Oct 30 07:25:01.003561 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:01.003429    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5611d3ae-2361-411e-903f-ebd0d5b5fba5-credential-socket\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7\" (UID: \"5611d3ae-2361-411e-903f-ebd0d5b5fba5\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7"
Oct 30 07:25:01.003616 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:01.003595    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5611d3ae-2361-411e-903f-ebd0d5b5fba5-istio-data\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7\" (UID: \"5611d3ae-2361-411e-903f-ebd0d5b5fba5\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7"
Oct 30 07:25:01.003671 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:01.003614    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5611d3ae-2361-411e-903f-ebd0d5b5fba5-workload-certs\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7\" (UID: \"5611d3ae-2361-411e-903f-ebd0d5b5fba5\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7"
Oct 30 07:25:01.003671 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:01.003598    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5611d3ae-2361-411e-903f-ebd0d5b5fba5-istiod-ca-cert\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7\" (UID: \"5611d3ae-2361-411e-903f-ebd0d5b5fba5\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7"
Oct 30 07:25:01.003776 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:01.003701    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5611d3ae-2361-411e-903f-ebd0d5b5fba5-workload-socket\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7\" (UID: \"5611d3ae-2361-411e-903f-ebd0d5b5fba5\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7"
Oct 30 07:25:01.004155 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:01.004125    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5611d3ae-2361-411e-903f-ebd0d5b5fba5-workload-socket\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7\" (UID: \"5611d3ae-2361-411e-903f-ebd0d5b5fba5\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7"
Oct 30 07:25:01.004271 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:01.004159    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5611d3ae-2361-411e-903f-ebd0d5b5fba5-credential-socket\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7\" (UID: \"5611d3ae-2361-411e-903f-ebd0d5b5fba5\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7"
Oct 30 07:25:01.004414 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:01.004354    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5611d3ae-2361-411e-903f-ebd0d5b5fba5-istiod-ca-cert\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7\" (UID: \"5611d3ae-2361-411e-903f-ebd0d5b5fba5\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7"
Oct 30 07:25:01.006695 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:01.006671    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5611d3ae-2361-411e-903f-ebd0d5b5fba5-istio-envoy\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7\" (UID: \"5611d3ae-2361-411e-903f-ebd0d5b5fba5\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7"
Oct 30 07:25:01.006883 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:01.006866    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5611d3ae-2361-411e-903f-ebd0d5b5fba5-istio-podinfo\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7\" (UID: \"5611d3ae-2361-411e-903f-ebd0d5b5fba5\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7"
Oct 30 07:25:01.028846 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:01.028768    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h58d\" (UniqueName: \"kubernetes.io/projected/5611d3ae-2361-411e-903f-ebd0d5b5fba5-kube-api-access-2h58d\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7\" (UID: \"5611d3ae-2361-411e-903f-ebd0d5b5fba5\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7"
Oct 30 07:25:01.028846 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:01.028791    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5611d3ae-2361-411e-903f-ebd0d5b5fba5-istio-token\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7\" (UID: \"5611d3ae-2361-411e-903f-ebd0d5b5fba5\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7"
Oct 30 07:25:01.136846 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:01.136800    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7"
Oct 30 07:25:01.528844 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:01.528796    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7"]
Oct 30 07:25:01.529122 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:25:01.529101    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5611d3ae_2361_411e_903f_ebd0d5b5fba5.slice/crio-d8c375dd046a792866a2cdc53663e5cee97c7c68dedca4c9d99fc389441020cc WatchSource:0}: Error finding container d8c375dd046a792866a2cdc53663e5cee97c7c68dedca4c9d99fc389441020cc: Status 404 returned error can't find the container with id d8c375dd046a792866a2cdc53663e5cee97c7c68dedca4c9d99fc389441020cc
Oct 30 07:25:01.531427 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:01.531391    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:25:01.531534 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:01.531467    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:25:01.531534 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:01.531512    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:25:01.861017 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:25:01.860909    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:25:01.934630 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:01.934596    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7" event={"ID":"5611d3ae-2361-411e-903f-ebd0d5b5fba5","Type":"ContainerStarted","Data":"c502885b597dc38e28d29895403865104f6c4cb2cd717c8f66e0f73f10a6ba86"}
Oct 30 07:25:01.935146 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:01.934658    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7" event={"ID":"5611d3ae-2361-411e-903f-ebd0d5b5fba5","Type":"ContainerStarted","Data":"d8c375dd046a792866a2cdc53663e5cee97c7c68dedca4c9d99fc389441020cc"}
Oct 30 07:25:01.998258 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:01.997777    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7" podStartSLOduration=1.9977564920000002 podStartE2EDuration="1.997756492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:25:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:25:01.997162696 +0000 UTC m=+1660.606569669" watchObservedRunningTime="2026-10-30 07:25:01.997756492 +0000 UTC m=+1660.607163465"
Oct 30 07:25:02.105904 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:02.105878    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a71a224-wsc72"
Oct 30 07:25:02.137003 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:02.136931    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7"
Oct 30 07:25:02.138625 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:02.138604    6224 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 30 07:25:02.138724 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:02.138650    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7" podUID="5611d3ae-2361-411e-903f-ebd0d5b5fba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 30 07:25:02.227416 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:02.227382    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ce27afe-fbcc-4c82-9986-ab4aa4f575ee-kube-api-access-hn87c\" (UniqueName: \"kubernetes.io/projected/4ce27afe-fbcc-4c82-9986-ab4aa4f575ee-kube-api-access-hn87c\") pod \"4ce27afe-fbcc-4c82-9986-ab4aa4f575ee\" (UID: \"4ce27afe-fbcc-4c82-9986-ab4aa4f575ee\") "
Oct 30 07:25:02.227601 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:02.227475    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ce27afe-fbcc-4c82-9986-ab4aa4f575ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ce27afe-fbcc-4c82-9986-ab4aa4f575ee-maas-api-xks-ca\") pod \"4ce27afe-fbcc-4c82-9986-ab4aa4f575ee\" (UID: \"4ce27afe-fbcc-4c82-9986-ab4aa4f575ee\") "
Oct 30 07:25:02.227601 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:02.227531    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ce27afe-fbcc-4c82-9986-ab4aa4f575ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ce27afe-fbcc-4c82-9986-ab4aa4f575ee-maas-api-openshift-ca\") pod \"4ce27afe-fbcc-4c82-9986-ab4aa4f575ee\" (UID: \"4ce27afe-fbcc-4c82-9986-ab4aa4f575ee\") "
Oct 30 07:25:02.227966 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:02.227936    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ce27afe-fbcc-4c82-9986-ab4aa4f575ee-maas-api-openshift-ca" pod "4ce27afe-fbcc-4c82-9986-ab4aa4f575ee" (UID: "4ce27afe-fbcc-4c82-9986-ab4aa4f575ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:25:02.228161 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:02.228139    6224 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ce27afe-fbcc-4c82-9986-ab4aa4f575ee-maas-api-openshift-ca\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:25:02.229584 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:02.229552    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ce27afe-fbcc-4c82-9986-ab4aa4f575ee-kube-api-access-hn87c" pod "4ce27afe-fbcc-4c82-9986-ab4aa4f575ee" (UID: "4ce27afe-fbcc-4c82-9986-ab4aa4f575ee"). InnerVolumeSpecName "kube-api-access-hn87c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:25:02.230081 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:02.230064    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ce27afe-fbcc-4c82-9986-ab4aa4f575ee-maas-api-xks-ca" pod "4ce27afe-fbcc-4c82-9986-ab4aa4f575ee" (UID: "4ce27afe-fbcc-4c82-9986-ab4aa4f575ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:25:02.329180 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:02.329142    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hn87c\" (UniqueName: \"kubernetes.io/projected/4ce27afe-fbcc-4c82-9986-ab4aa4f575ee-kube-api-access-hn87c\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:25:02.329180 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:02.329183    6224 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ce27afe-fbcc-4c82-9986-ab4aa4f575ee-maas-api-xks-ca\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:25:02.798848 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:02.766370    6224 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 07:25:02.798848 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:02.766435    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v" podUID="ea11f788-17b6-4758-945d-fbb0d273e840" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:25:02.835083 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:02.835053    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v"
Oct 30 07:25:02.835456 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:02.835428    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v"
Oct 30 07:25:02.836163 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:02.836143    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v"
Oct 30 07:25:02.942120 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:02.942073    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a71a224-wsc72"
Oct 30 07:25:02.942120 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:02.942103    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a71a224-wsc72" event={"ID":"4ce27afe-fbcc-4c82-9986-ab4aa4f575ee","Type":"ContainerDied","Data":"beba93d1438cd64c097132f50e6c5b19a9d4fd9f833c5eb941856c6fb2d545cb"}
Oct 30 07:25:02.942629 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:02.942178    6224 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="beba93d1438cd64c097132f50e6c5b19a9d4fd9f833c5eb941856c6fb2d545cb"
Oct 30 07:25:03.138167 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:03.138084    6224 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 30 07:25:03.138167 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:03.138144    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7" podUID="5611d3ae-2361-411e-903f-ebd0d5b5fba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 30 07:25:04.137378 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:04.137339    6224 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 30 07:25:04.137956 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:04.137412    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7" podUID="5611d3ae-2361-411e-903f-ebd0d5b5fba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 30 07:25:05.137368 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:05.137331    6224 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 30 07:25:05.137545 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:05.137389    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7" podUID="5611d3ae-2361-411e-903f-ebd0d5b5fba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 30 07:25:06.137390 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:06.137356    6224 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 30 07:25:06.137575 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:06.137410    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7" podUID="5611d3ae-2361-411e-903f-ebd0d5b5fba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 30 07:25:07.141641 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:07.141602    6224 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 30 07:25:07.142252 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:07.142218    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7" podUID="5611d3ae-2361-411e-903f-ebd0d5b5fba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 30 07:25:08.137822 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:08.137776    6224 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 30 07:25:08.138004 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:08.137871    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7" podUID="5611d3ae-2361-411e-903f-ebd0d5b5fba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 30 07:25:09.271496 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:09.271458    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7"
Oct 30 07:25:09.272038 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:09.271862    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7"
Oct 30 07:25:09.272844 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:09.272800    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7"
Oct 30 07:25:12.046441 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:12.046401    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x"]
Oct 30 07:25:12.047119 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:12.047098    6224 cpu_manager.go:414] "containerMap: removing container" podUID="4ce27afe-fbcc-4c82-9986-ab4aa4f575ee" containerName="revoke-keys"
Oct 30 07:25:12.047211 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:12.047120    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ce27afe-fbcc-4c82-9986-ab4aa4f575ee" containerName="revoke-keys"
Oct 30 07:25:12.047271 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:12.047242    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ce27afe-fbcc-4c82-9986-ab4aa4f575ee" containerName="revoke-keys"
Oct 30 07:25:12.050160 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:12.050139    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x"
Oct 30 07:25:12.077091 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:12.077063    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x"]
Oct 30 07:25:12.185002 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:12.184966    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/883946b4-c9d3-4c98-904f-9076e7597a1f-istio-token\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x\" (UID: \"883946b4-c9d3-4c98-904f-9076e7597a1f\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x"
Oct 30 07:25:12.185002 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:12.185006    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/883946b4-c9d3-4c98-904f-9076e7597a1f-istio-envoy\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x\" (UID: \"883946b4-c9d3-4c98-904f-9076e7597a1f\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x"
Oct 30 07:25:12.185254 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:12.185026    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/883946b4-c9d3-4c98-904f-9076e7597a1f-istiod-ca-cert\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x\" (UID: \"883946b4-c9d3-4c98-904f-9076e7597a1f\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x"
Oct 30 07:25:12.185254 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:12.185235    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/883946b4-c9d3-4c98-904f-9076e7597a1f-workload-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x\" (UID: \"883946b4-c9d3-4c98-904f-9076e7597a1f\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x"
Oct 30 07:25:12.185371 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:12.185314    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4866t\" (UniqueName: \"kubernetes.io/projected/883946b4-c9d3-4c98-904f-9076e7597a1f-kube-api-access-4866t\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x\" (UID: \"883946b4-c9d3-4c98-904f-9076e7597a1f\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x"
Oct 30 07:25:12.185410 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:12.185388    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/883946b4-c9d3-4c98-904f-9076e7597a1f-credential-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x\" (UID: \"883946b4-c9d3-4c98-904f-9076e7597a1f\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x"
Oct 30 07:25:12.185447 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:12.185414    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/883946b4-c9d3-4c98-904f-9076e7597a1f-istio-podinfo\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x\" (UID: \"883946b4-c9d3-4c98-904f-9076e7597a1f\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x"
Oct 30 07:25:12.185539 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:12.185525    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/883946b4-c9d3-4c98-904f-9076e7597a1f-workload-certs\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x\" (UID: \"883946b4-c9d3-4c98-904f-9076e7597a1f\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x"
Oct 30 07:25:12.185574 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:12.185550    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/883946b4-c9d3-4c98-904f-9076e7597a1f-istio-data\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x\" (UID: \"883946b4-c9d3-4c98-904f-9076e7597a1f\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x"
Oct 30 07:25:12.286300 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:12.286263    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/883946b4-c9d3-4c98-904f-9076e7597a1f-istio-token\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x\" (UID: \"883946b4-c9d3-4c98-904f-9076e7597a1f\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x"
Oct 30 07:25:12.286300 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:12.286300    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/883946b4-c9d3-4c98-904f-9076e7597a1f-istio-envoy\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x\" (UID: \"883946b4-c9d3-4c98-904f-9076e7597a1f\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x"
Oct 30 07:25:12.286549 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:12.286329    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/883946b4-c9d3-4c98-904f-9076e7597a1f-istiod-ca-cert\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x\" (UID: \"883946b4-c9d3-4c98-904f-9076e7597a1f\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x"
Oct 30 07:25:12.286549 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:12.286395    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/883946b4-c9d3-4c98-904f-9076e7597a1f-workload-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x\" (UID: \"883946b4-c9d3-4c98-904f-9076e7597a1f\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x"
Oct 30 07:25:12.286549 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:12.286447    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4866t\" (UniqueName: \"kubernetes.io/projected/883946b4-c9d3-4c98-904f-9076e7597a1f-kube-api-access-4866t\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x\" (UID: \"883946b4-c9d3-4c98-904f-9076e7597a1f\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x"
Oct 30 07:25:12.286549 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:12.286487    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/883946b4-c9d3-4c98-904f-9076e7597a1f-credential-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x\" (UID: \"883946b4-c9d3-4c98-904f-9076e7597a1f\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x"
Oct 30 07:25:12.286549 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:12.286521    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/883946b4-c9d3-4c98-904f-9076e7597a1f-istio-podinfo\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x\" (UID: \"883946b4-c9d3-4c98-904f-9076e7597a1f\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x"
Oct 30 07:25:12.286832 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:12.286602    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/883946b4-c9d3-4c98-904f-9076e7597a1f-workload-certs\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x\" (UID: \"883946b4-c9d3-4c98-904f-9076e7597a1f\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x"
Oct 30 07:25:12.286832 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:12.286620    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/883946b4-c9d3-4c98-904f-9076e7597a1f-istio-data\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x\" (UID: \"883946b4-c9d3-4c98-904f-9076e7597a1f\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x"
Oct 30 07:25:12.287044 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:12.287015    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/883946b4-c9d3-4c98-904f-9076e7597a1f-workload-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x\" (UID: \"883946b4-c9d3-4c98-904f-9076e7597a1f\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x"
Oct 30 07:25:12.287044 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:12.287046    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/883946b4-c9d3-4c98-904f-9076e7597a1f-istio-data\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x\" (UID: \"883946b4-c9d3-4c98-904f-9076e7597a1f\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x"
Oct 30 07:25:12.287220 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:12.287200    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/883946b4-c9d3-4c98-904f-9076e7597a1f-istiod-ca-cert\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x\" (UID: \"883946b4-c9d3-4c98-904f-9076e7597a1f\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x"
Oct 30 07:25:12.287293 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:12.287272    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/883946b4-c9d3-4c98-904f-9076e7597a1f-credential-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x\" (UID: \"883946b4-c9d3-4c98-904f-9076e7597a1f\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x"
Oct 30 07:25:12.287550 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:12.287526    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/883946b4-c9d3-4c98-904f-9076e7597a1f-workload-certs\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x\" (UID: \"883946b4-c9d3-4c98-904f-9076e7597a1f\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x"
Oct 30 07:25:12.288762 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:12.288744    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/883946b4-c9d3-4c98-904f-9076e7597a1f-istio-envoy\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x\" (UID: \"883946b4-c9d3-4c98-904f-9076e7597a1f\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x"
Oct 30 07:25:12.289227 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:12.289204    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/883946b4-c9d3-4c98-904f-9076e7597a1f-istio-podinfo\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x\" (UID: \"883946b4-c9d3-4c98-904f-9076e7597a1f\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x"
Oct 30 07:25:12.312831 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:12.312760    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4866t\" (UniqueName: \"kubernetes.io/projected/883946b4-c9d3-4c98-904f-9076e7597a1f-kube-api-access-4866t\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x\" (UID: \"883946b4-c9d3-4c98-904f-9076e7597a1f\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x"
Oct 30 07:25:12.317818 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:12.317783    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/883946b4-c9d3-4c98-904f-9076e7597a1f-istio-token\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x\" (UID: \"883946b4-c9d3-4c98-904f-9076e7597a1f\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x"
Oct 30 07:25:12.365378 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:12.365354    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x"
Oct 30 07:25:12.682820 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:12.682775    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x"]
Oct 30 07:25:12.683141 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:25:12.683121    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod883946b4_c9d3_4c98_904f_9076e7597a1f.slice/crio-61c902f661cf218bb202d8e2e0e105c2eafde7d2365c9d4362fab8d9f3bc966b WatchSource:0}: Error finding container 61c902f661cf218bb202d8e2e0e105c2eafde7d2365c9d4362fab8d9f3bc966b: Status 404 returned error can't find the container with id 61c902f661cf218bb202d8e2e0e105c2eafde7d2365c9d4362fab8d9f3bc966b
Oct 30 07:25:12.685570 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:12.685539    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:25:12.685666 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:12.685606    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:25:12.685666 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:12.685637    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:25:13.055877 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:13.055818    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x" event={"ID":"883946b4-c9d3-4c98-904f-9076e7597a1f","Type":"ContainerStarted","Data":"7bbee109bf27e28dc0874b73c27b9315d236b89c6ea8ccf25c0e0aa5e3f720c4"}
Oct 30 07:25:13.056365 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:13.055892    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x" event={"ID":"883946b4-c9d3-4c98-904f-9076e7597a1f","Type":"ContainerStarted","Data":"61c902f661cf218bb202d8e2e0e105c2eafde7d2365c9d4362fab8d9f3bc966b"}
Oct 30 07:25:13.110521 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:13.110453    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x" podStartSLOduration=1.11043213 podStartE2EDuration="1.11043213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:25:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:25:13.105745361 +0000 UTC m=+1671.715152378" watchObservedRunningTime="2026-10-30 07:25:13.11043213 +0000 UTC m=+1671.719839102"
Oct 30 07:25:13.382957 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:13.378138    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x"
Oct 30 07:25:13.389354 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:13.388779    6224 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 30 07:25:13.389354 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:13.389107    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x" podUID="883946b4-c9d3-4c98-904f-9076e7597a1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 30 07:25:14.378831 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:14.372829    6224 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 30 07:25:14.378831 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:14.372901    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x" podUID="883946b4-c9d3-4c98-904f-9076e7597a1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 30 07:25:15.388421 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:15.388280    6224 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 30 07:25:15.388421 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:15.388345    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x" podUID="883946b4-c9d3-4c98-904f-9076e7597a1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 30 07:25:16.369221 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:16.369183    6224 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 30 07:25:16.369510 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:16.369479    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x" podUID="883946b4-c9d3-4c98-904f-9076e7597a1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 30 07:25:16.442457 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:16.442425    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp"]
Oct 30 07:25:16.447552 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:16.447527    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp"
Oct 30 07:25:16.490898 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:16.490670    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp"]
Oct 30 07:25:16.596727 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:16.596688    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5fe37b6-8794-4108-aee2-5da540d6bc0c-istio-data\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp\" (UID: \"c5fe37b6-8794-4108-aee2-5da540d6bc0c\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp"
Oct 30 07:25:16.597111 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:16.597010    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5fe37b6-8794-4108-aee2-5da540d6bc0c-istiod-ca-cert\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp\" (UID: \"c5fe37b6-8794-4108-aee2-5da540d6bc0c\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp"
Oct 30 07:25:16.597111 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:16.597088    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5fe37b6-8794-4108-aee2-5da540d6bc0c-credential-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp\" (UID: \"c5fe37b6-8794-4108-aee2-5da540d6bc0c\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp"
Oct 30 07:25:16.597286 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:16.597177    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2f44\" (UniqueName: \"kubernetes.io/projected/c5fe37b6-8794-4108-aee2-5da540d6bc0c-kube-api-access-r2f44\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp\" (UID: \"c5fe37b6-8794-4108-aee2-5da540d6bc0c\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp"
Oct 30 07:25:16.597376 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:16.597348    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5fe37b6-8794-4108-aee2-5da540d6bc0c-istio-envoy\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp\" (UID: \"c5fe37b6-8794-4108-aee2-5da540d6bc0c\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp"
Oct 30 07:25:16.597436 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:16.597399    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5fe37b6-8794-4108-aee2-5da540d6bc0c-istio-podinfo\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp\" (UID: \"c5fe37b6-8794-4108-aee2-5da540d6bc0c\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp"
Oct 30 07:25:16.597492 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:16.597425    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5fe37b6-8794-4108-aee2-5da540d6bc0c-istio-token\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp\" (UID: \"c5fe37b6-8794-4108-aee2-5da540d6bc0c\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp"
Oct 30 07:25:16.597573 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:16.597555    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5fe37b6-8794-4108-aee2-5da540d6bc0c-workload-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp\" (UID: \"c5fe37b6-8794-4108-aee2-5da540d6bc0c\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp"
Oct 30 07:25:16.597659 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:16.597644    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5fe37b6-8794-4108-aee2-5da540d6bc0c-workload-certs\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp\" (UID: \"c5fe37b6-8794-4108-aee2-5da540d6bc0c\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp"
Oct 30 07:25:16.698666 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:16.698574    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5fe37b6-8794-4108-aee2-5da540d6bc0c-credential-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp\" (UID: \"c5fe37b6-8794-4108-aee2-5da540d6bc0c\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp"
Oct 30 07:25:16.698666 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:16.698610    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2f44\" (UniqueName: \"kubernetes.io/projected/c5fe37b6-8794-4108-aee2-5da540d6bc0c-kube-api-access-r2f44\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp\" (UID: \"c5fe37b6-8794-4108-aee2-5da540d6bc0c\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp"
Oct 30 07:25:16.698933 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:16.698680    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5fe37b6-8794-4108-aee2-5da540d6bc0c-istio-envoy\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp\" (UID: \"c5fe37b6-8794-4108-aee2-5da540d6bc0c\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp"
Oct 30 07:25:16.698933 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:16.698707    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5fe37b6-8794-4108-aee2-5da540d6bc0c-istio-podinfo\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp\" (UID: \"c5fe37b6-8794-4108-aee2-5da540d6bc0c\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp"
Oct 30 07:25:16.698933 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:16.698728    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5fe37b6-8794-4108-aee2-5da540d6bc0c-istio-token\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp\" (UID: \"c5fe37b6-8794-4108-aee2-5da540d6bc0c\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp"
Oct 30 07:25:16.698933 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:16.698750    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5fe37b6-8794-4108-aee2-5da540d6bc0c-workload-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp\" (UID: \"c5fe37b6-8794-4108-aee2-5da540d6bc0c\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp"
Oct 30 07:25:16.698933 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:16.698788    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5fe37b6-8794-4108-aee2-5da540d6bc0c-workload-certs\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp\" (UID: \"c5fe37b6-8794-4108-aee2-5da540d6bc0c\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp"
Oct 30 07:25:16.698933 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:16.698884    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5fe37b6-8794-4108-aee2-5da540d6bc0c-istio-data\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp\" (UID: \"c5fe37b6-8794-4108-aee2-5da540d6bc0c\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp"
Oct 30 07:25:16.698933 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:16.698908    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5fe37b6-8794-4108-aee2-5da540d6bc0c-istiod-ca-cert\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp\" (UID: \"c5fe37b6-8794-4108-aee2-5da540d6bc0c\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp"
Oct 30 07:25:16.699892 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:16.699523    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5fe37b6-8794-4108-aee2-5da540d6bc0c-credential-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp\" (UID: \"c5fe37b6-8794-4108-aee2-5da540d6bc0c\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp"
Oct 30 07:25:16.699892 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:16.699572    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5fe37b6-8794-4108-aee2-5da540d6bc0c-istiod-ca-cert\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp\" (UID: \"c5fe37b6-8794-4108-aee2-5da540d6bc0c\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp"
Oct 30 07:25:16.699892 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:16.699609    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5fe37b6-8794-4108-aee2-5da540d6bc0c-workload-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp\" (UID: \"c5fe37b6-8794-4108-aee2-5da540d6bc0c\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp"
Oct 30 07:25:16.700572 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:16.700551    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5fe37b6-8794-4108-aee2-5da540d6bc0c-istio-data\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp\" (UID: \"c5fe37b6-8794-4108-aee2-5da540d6bc0c\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp"
Oct 30 07:25:16.700917 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:16.700896    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5fe37b6-8794-4108-aee2-5da540d6bc0c-workload-certs\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp\" (UID: \"c5fe37b6-8794-4108-aee2-5da540d6bc0c\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp"
Oct 30 07:25:16.701774 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:16.701748    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5fe37b6-8794-4108-aee2-5da540d6bc0c-istio-envoy\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp\" (UID: \"c5fe37b6-8794-4108-aee2-5da540d6bc0c\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp"
Oct 30 07:25:16.702141 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:16.702118    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5fe37b6-8794-4108-aee2-5da540d6bc0c-istio-podinfo\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp\" (UID: \"c5fe37b6-8794-4108-aee2-5da540d6bc0c\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp"
Oct 30 07:25:16.725178 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:16.725152    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2f44\" (UniqueName: \"kubernetes.io/projected/c5fe37b6-8794-4108-aee2-5da540d6bc0c-kube-api-access-r2f44\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp\" (UID: \"c5fe37b6-8794-4108-aee2-5da540d6bc0c\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp"
Oct 30 07:25:16.725294 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:16.725186    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5fe37b6-8794-4108-aee2-5da540d6bc0c-istio-token\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp\" (UID: \"c5fe37b6-8794-4108-aee2-5da540d6bc0c\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp"
Oct 30 07:25:16.792069 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:16.792038    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp"
Oct 30 07:25:17.131110 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:25:17.131079    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5fe37b6_8794_4108_aee2_5da540d6bc0c.slice/crio-76dd169ae8fa2a2a4327fb9bfa4c2cae1bf22bb1f24f3d2163a23b4e8d4a2df5 WatchSource:0}: Error finding container 76dd169ae8fa2a2a4327fb9bfa4c2cae1bf22bb1f24f3d2163a23b4e8d4a2df5: Status 404 returned error can't find the container with id 76dd169ae8fa2a2a4327fb9bfa4c2cae1bf22bb1f24f3d2163a23b4e8d4a2df5
Oct 30 07:25:17.132678 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:17.132645    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp"]
Oct 30 07:25:17.133932 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:17.133895    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:25:17.134008 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:17.133977    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:25:17.134049 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:17.134021    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:25:17.370829 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:17.369103    6224 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 30 07:25:17.370829 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:17.369187    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x" podUID="883946b4-c9d3-4c98-904f-9076e7597a1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 30 07:25:18.131831 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:18.131018    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp" event={"ID":"c5fe37b6-8794-4108-aee2-5da540d6bc0c","Type":"ContainerStarted","Data":"5ed20299d171217c79fd9550036ae9d2b0bd263d98e7fc9741170c0b5a3a55ce"}
Oct 30 07:25:18.131831 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:18.131090    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp" event={"ID":"c5fe37b6-8794-4108-aee2-5da540d6bc0c","Type":"ContainerStarted","Data":"76dd169ae8fa2a2a4327fb9bfa4c2cae1bf22bb1f24f3d2163a23b4e8d4a2df5"}
Oct 30 07:25:18.201832 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:18.201728    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp" podStartSLOduration=2.201710003 podStartE2EDuration="2.201710003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:25:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:25:18.201114152 +0000 UTC m=+1676.810521124" watchObservedRunningTime="2026-10-30 07:25:18.201710003 +0000 UTC m=+1676.811116974"
Oct 30 07:25:18.368834 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:18.367142    6224 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 30 07:25:18.368834 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:18.367206    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x" podUID="883946b4-c9d3-4c98-904f-9076e7597a1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 30 07:25:18.798238 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:18.797495    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp"
Oct 30 07:25:18.798238 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:18.797652    6224 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 30 07:25:18.798238 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:18.797710    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp" podUID="c5fe37b6-8794-4108-aee2-5da540d6bc0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 30 07:25:19.370303 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:19.370267    6224 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 30 07:25:19.370935 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:19.370903    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x" podUID="883946b4-c9d3-4c98-904f-9076e7597a1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 30 07:25:19.793286 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:19.793245    6224 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 30 07:25:19.793579 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:19.793544    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp" podUID="c5fe37b6-8794-4108-aee2-5da540d6bc0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 30 07:25:20.367239 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:20.367194    6224 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 30 07:25:20.367534 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:20.367503    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x" podUID="883946b4-c9d3-4c98-904f-9076e7597a1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 30 07:25:20.798395 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:20.798362    6224 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 30 07:25:20.799003 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:20.798956    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp" podUID="c5fe37b6-8794-4108-aee2-5da540d6bc0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 30 07:25:20.859083 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:20.859032    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/simulator-w3-2eda2c-kserve-56776b9849-9bqzm" podUID="482e6b7c-2231-423c-b54f-a54e56f16ab7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:25:20.863206 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:20.862906    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-06b26e/distinct-w0-06b26e-kserve-545cf57898-2fhhd" podUID="71294bfa-ac2c-40b7-b889-0527daa07f63" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:25:21.889832 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:21.876799    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2bab4c/simulator-w4-2bab4c-kserve-6db9444549-7nrk8" podUID="c7b09178-9ac7-492f-890b-5ff3da130981" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Oct 30 07:25:21.889832 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:21.877577    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb6be4/simulator-w2-bb6be4-kserve-8485bbf954-dnbbz" podUID="6370f7e2-d6c8-4fb5-8198-32b38837ca31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:25:22.063934 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:22.063889    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2bab4c/premium-w4-2bab4c-kserve-665887fc68-c6dgf" podUID="f9463476-fcf5-4d66-bfd2-b85ce4ed651f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:25:22.086111 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:22.086056    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-06b26e/distinct-w0-06b26e-kserve-545cf57898-2fhhd" podUID="71294bfa-ac2c-40b7-b889-0527daa07f63" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:25:22.116464 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:22.116417    6224 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-2bab4c/premium-w4-2bab4c-kserve-665887fc68-c6dgf" podUID="f9463476-fcf5-4d66-bfd2-b85ce4ed651f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": context deadline exceeded"
Oct 30 07:25:22.157014 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:22.156923    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2bab4c/distinct-w4-2bab4c-kserve-6868d767c9-j7wlm" podUID="11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Oct 30 07:25:22.157639 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:22.157605    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/simulator-w3-2eda2c-kserve-56776b9849-9bqzm" podUID="482e6b7c-2231-423c-b54f-a54e56f16ab7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded"
Oct 30 07:25:22.158107 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:22.158081    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-06b26e/premium-w0-06b26e-kserve-c9dbd9879-b4jlk" podUID="1230dcae-e1cd-460d-86fc-bd40c8427b5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 30 07:25:22.277006 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:22.192906    6224 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 30 07:25:22.277313 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:22.277279    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp" podUID="c5fe37b6-8794-4108-aee2-5da540d6bc0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 30 07:25:22.753832 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:22.688089    6224 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 07:25:22.821604 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:22.821557    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x" podUID="883946b4-c9d3-4c98-904f-9076e7597a1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:25:22.822309 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:25:22.822279    6224 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.126s"
Oct 30 07:25:22.822668 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:22.698833    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw" podUID="0597696e-26b8-4ef8-8697-64d49b2b9398" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:25:22.827155 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:22.827137    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-06b26e/distinct-w0-06b26e-kserve-545cf57898-2fhhd"
Oct 30 07:25:22.827668 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:22.827652    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2eda2c/simulator-w3-2eda2c-kserve-56776b9849-9bqzm"
Oct 30 07:25:22.836029 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:22.830944    6224 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 30 07:25:22.836029 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:22.830985    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp" podUID="c5fe37b6-8794-4108-aee2-5da540d6bc0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 30 07:25:22.848163 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:22.848138    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2eda2c/simulator-w3-2eda2c-kserve-56776b9849-9bqzm"
Oct 30 07:25:22.900940 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:22.896669    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-06b26e/distinct-w0-06b26e-kserve-545cf57898-2fhhd"
Oct 30 07:25:22.900940 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:22.896848    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb6be4/simulator-w2-bb6be4-kserve-8485bbf954-dnbbz" podUID="6370f7e2-d6c8-4fb5-8198-32b38837ca31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:25:22.900940 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:22.897220    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb6be4/simulator-w2-bb6be4-kserve-8485bbf954-dnbbz"
Oct 30 07:25:22.992297 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:22.992264    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x"
Oct 30 07:25:23.156830 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:23.083873    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x"
Oct 30 07:25:23.156830 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:23.084902    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2bab4c/premium-w4-2bab4c-kserve-665887fc68-c6dgf" podUID="f9463476-fcf5-4d66-bfd2-b85ce4ed651f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:25:23.156830 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:23.085359    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2bab4c/premium-w4-2bab4c-kserve-665887fc68-c6dgf"
Oct 30 07:25:23.156830 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:23.122567    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x"
Oct 30 07:25:23.262683 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:23.262640    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-06b26e/premium-w0-06b26e-kserve-c9dbd9879-b4jlk" podUID="1230dcae-e1cd-460d-86fc-bd40c8427b5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 30 07:25:23.263568 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:23.263528    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-06b26e/premium-w0-06b26e-kserve-c9dbd9879-b4jlk"
Oct 30 07:25:23.434440 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:23.434359    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2bab4c/premium-w4-2bab4c-kserve-665887fc68-c6dgf"
Oct 30 07:25:23.646161 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:23.646121    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb6be4/simulator-w2-bb6be4-kserve-8485bbf954-dnbbz"
Oct 30 07:25:23.646550 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:23.646534    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-06b26e/premium-w0-06b26e-kserve-c9dbd9879-b4jlk"
Oct 30 07:25:23.800890 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:23.798056    6224 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 30 07:25:23.800890 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:23.798103    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp" podUID="c5fe37b6-8794-4108-aee2-5da540d6bc0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 30 07:25:23.819932 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:23.817392    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-932c00-79d49cb7d7-mpkdp"]
Oct 30 07:25:23.839832 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:23.827622    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-932c00-79d49cb7d7-mpkdp"]
Oct 30 07:25:23.839832 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:23.827730    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-932c00-79d49cb7d7-mpkdp"
Oct 30 07:25:23.858798 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:23.858504    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m44j\" (UniqueName: \"kubernetes.io/projected/e2a703c4-5066-4875-be2d-5fc882b38502-kube-api-access-9m44j\") pod \"payload-pre-processing-e2e-shared-a-w2-932c00-79d49cb7d7-mpkdp\" (UID: \"e2a703c4-5066-4875-be2d-5fc882b38502\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-932c00-79d49cb7d7-mpkdp"
Oct 30 07:25:23.858798 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:23.858593    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2a703c4-5066-4875-be2d-5fc882b38502-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-932c00-79d49cb7d7-mpkdp\" (UID: \"e2a703c4-5066-4875-be2d-5fc882b38502\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-932c00-79d49cb7d7-mpkdp"
Oct 30 07:25:23.887508 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:23.887477    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-932c00\""
Oct 30 07:25:23.946431 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:23.946366    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-932c00-dockercfg-7p95s\""
Oct 30 07:25:24.002855 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:23.988798    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m44j\" (UniqueName: \"kubernetes.io/projected/e2a703c4-5066-4875-be2d-5fc882b38502-kube-api-access-9m44j\") pod \"payload-pre-processing-e2e-shared-a-w2-932c00-79d49cb7d7-mpkdp\" (UID: \"e2a703c4-5066-4875-be2d-5fc882b38502\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-932c00-79d49cb7d7-mpkdp"
Oct 30 07:25:24.002855 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:23.988941    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2a703c4-5066-4875-be2d-5fc882b38502-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-932c00-79d49cb7d7-mpkdp\" (UID: \"e2a703c4-5066-4875-be2d-5fc882b38502\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-932c00-79d49cb7d7-mpkdp"
Oct 30 07:25:24.002855 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:23.989956    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2a703c4-5066-4875-be2d-5fc882b38502-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-932c00-79d49cb7d7-mpkdp\" (UID: \"e2a703c4-5066-4875-be2d-5fc882b38502\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-932c00-79d49cb7d7-mpkdp"
Oct 30 07:25:24.106474 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:24.104918    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m44j\" (UniqueName: \"kubernetes.io/projected/e2a703c4-5066-4875-be2d-5fc882b38502-kube-api-access-9m44j\") pod \"payload-pre-processing-e2e-shared-a-w2-932c00-79d49cb7d7-mpkdp\" (UID: \"e2a703c4-5066-4875-be2d-5fc882b38502\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-932c00-79d49cb7d7-mpkdp"
Oct 30 07:25:24.340243 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:24.340199    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-932c00-79d49cb7d7-mpkdp"
Oct 30 07:25:24.801885 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:24.797162    6224 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 30 07:25:24.801885 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:24.797224    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp" podUID="c5fe37b6-8794-4108-aee2-5da540d6bc0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 30 07:25:24.851149 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:24.851117    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-932c00-79d49cb7d7-mpkdp"]
Oct 30 07:25:24.852277 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:25:24.852250    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2a703c4_5066_4875_be2d_5fc882b38502.slice/crio-d153ad1827db43dd101030b2a3b4b3551bd1709b0fc2cec7322eda07a9ff1af5 WatchSource:0}: Error finding container d153ad1827db43dd101030b2a3b4b3551bd1709b0fc2cec7322eda07a9ff1af5: Status 404 returned error can't find the container with id d153ad1827db43dd101030b2a3b4b3551bd1709b0fc2cec7322eda07a9ff1af5
Oct 30 07:25:25.010706 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:25.010664    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-932c00-79d49cb7d7-mpkdp" event={"ID":"e2a703c4-5066-4875-be2d-5fc882b38502","Type":"ContainerStarted","Data":"d61135b21220c000ef954bbee94eeda14dd755a6da49dc0d668914a7524a638a"}
Oct 30 07:25:25.011186 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:25.010736    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-932c00-79d49cb7d7-mpkdp" event={"ID":"e2a703c4-5066-4875-be2d-5fc882b38502","Type":"ContainerStarted","Data":"d153ad1827db43dd101030b2a3b4b3551bd1709b0fc2cec7322eda07a9ff1af5"}
Oct 30 07:25:25.011186 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:25.010988    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-932c00-79d49cb7d7-mpkdp"
Oct 30 07:25:25.054971 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:25.054876    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-932c00-79d49cb7d7-mpkdp" podStartSLOduration=2.054859827 podStartE2EDuration="2.054859827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:25:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:25:25.054445894 +0000 UTC m=+1683.663852864" watchObservedRunningTime="2026-10-30 07:25:25.054859827 +0000 UTC m=+1683.664266798"
Oct 30 07:25:25.793364 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:25.793331    6224 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 30 07:25:25.793556 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:25.793386    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp" podUID="c5fe37b6-8794-4108-aee2-5da540d6bc0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 30 07:25:27.833964 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:27.833927    6224 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 07:25:27.834366 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:27.833984    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp" podUID="c5fe37b6-8794-4108-aee2-5da540d6bc0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:25:27.994153 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:27.994121    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp"
Oct 30 07:25:27.994492 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:27.994467    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp"
Oct 30 07:25:28.000188 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:28.000167    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp"
Oct 30 07:25:28.807667 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:28.807610    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-04aede-776cc889f-47jhw"]
Oct 30 07:25:28.812060 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:28.812035    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-04aede-776cc889f-47jhw"
Oct 30 07:25:28.836008 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:28.835983    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-04aede-dockercfg-btdnj\""
Oct 30 07:25:28.836606 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:28.836584    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-04aede\""
Oct 30 07:25:28.841539 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:28.840981    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-04aede-776cc889f-47jhw"]
Oct 30 07:25:28.872575 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:28.872528    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdfxw\" (UniqueName: \"kubernetes.io/projected/c23e777f-2b9b-45fb-8110-62912dcbbaa6-kube-api-access-qdfxw\") pod \"payload-pre-processing-e2e-autores-04aede-776cc889f-47jhw\" (UID: \"c23e777f-2b9b-45fb-8110-62912dcbbaa6\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-04aede-776cc889f-47jhw"
Oct 30 07:25:28.872733 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:28.872660    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c23e777f-2b9b-45fb-8110-62912dcbbaa6-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-04aede-776cc889f-47jhw\" (UID: \"c23e777f-2b9b-45fb-8110-62912dcbbaa6\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-04aede-776cc889f-47jhw"
Oct 30 07:25:28.928322 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:28.928293    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-04aede-5cd4c88478-kbvnt"]
Oct 30 07:25:28.933101 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:28.933076    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-04aede-5cd4c88478-kbvnt"
Oct 30 07:25:28.998647 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:28.998603    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf4f18e1-ad1c-4315-9c07-d756d6d70476-plugins-config-volume\") pod \"payload-processing-e2e-autores-04aede-5cd4c88478-kbvnt\" (UID: \"bf4f18e1-ad1c-4315-9c07-d756d6d70476\") " pod="openshift-ingress/payload-processing-e2e-autores-04aede-5cd4c88478-kbvnt"
Oct 30 07:25:29.017060 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:29.017017    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdfxw\" (UniqueName: \"kubernetes.io/projected/c23e777f-2b9b-45fb-8110-62912dcbbaa6-kube-api-access-qdfxw\") pod \"payload-pre-processing-e2e-autores-04aede-776cc889f-47jhw\" (UID: \"c23e777f-2b9b-45fb-8110-62912dcbbaa6\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-04aede-776cc889f-47jhw"
Oct 30 07:25:29.017302 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:29.017280    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2grh\" (UniqueName: \"kubernetes.io/projected/bf4f18e1-ad1c-4315-9c07-d756d6d70476-kube-api-access-d2grh\") pod \"payload-processing-e2e-autores-04aede-5cd4c88478-kbvnt\" (UID: \"bf4f18e1-ad1c-4315-9c07-d756d6d70476\") " pod="openshift-ingress/payload-processing-e2e-autores-04aede-5cd4c88478-kbvnt"
Oct 30 07:25:29.017575 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:29.017559    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c23e777f-2b9b-45fb-8110-62912dcbbaa6-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-04aede-776cc889f-47jhw\" (UID: \"c23e777f-2b9b-45fb-8110-62912dcbbaa6\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-04aede-776cc889f-47jhw"
Oct 30 07:25:29.018645 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:29.018618    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c23e777f-2b9b-45fb-8110-62912dcbbaa6-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-04aede-776cc889f-47jhw\" (UID: \"c23e777f-2b9b-45fb-8110-62912dcbbaa6\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-04aede-776cc889f-47jhw"
Oct 30 07:25:29.042827 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:29.031002    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-04aede-5cd4c88478-kbvnt"]
Oct 30 07:25:29.079362 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:29.079284    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdfxw\" (UniqueName: \"kubernetes.io/projected/c23e777f-2b9b-45fb-8110-62912dcbbaa6-kube-api-access-qdfxw\") pod \"payload-pre-processing-e2e-autores-04aede-776cc889f-47jhw\" (UID: \"c23e777f-2b9b-45fb-8110-62912dcbbaa6\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-04aede-776cc889f-47jhw"
Oct 30 07:25:29.118479 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:29.118438    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf4f18e1-ad1c-4315-9c07-d756d6d70476-plugins-config-volume\") pod \"payload-processing-e2e-autores-04aede-5cd4c88478-kbvnt\" (UID: \"bf4f18e1-ad1c-4315-9c07-d756d6d70476\") " pod="openshift-ingress/payload-processing-e2e-autores-04aede-5cd4c88478-kbvnt"
Oct 30 07:25:29.118674 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:29.118524    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2grh\" (UniqueName: \"kubernetes.io/projected/bf4f18e1-ad1c-4315-9c07-d756d6d70476-kube-api-access-d2grh\") pod \"payload-processing-e2e-autores-04aede-5cd4c88478-kbvnt\" (UID: \"bf4f18e1-ad1c-4315-9c07-d756d6d70476\") " pod="openshift-ingress/payload-processing-e2e-autores-04aede-5cd4c88478-kbvnt"
Oct 30 07:25:29.119779 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:29.119745    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf4f18e1-ad1c-4315-9c07-d756d6d70476-plugins-config-volume\") pod \"payload-processing-e2e-autores-04aede-5cd4c88478-kbvnt\" (UID: \"bf4f18e1-ad1c-4315-9c07-d756d6d70476\") " pod="openshift-ingress/payload-processing-e2e-autores-04aede-5cd4c88478-kbvnt"
Oct 30 07:25:29.127896 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:29.127864    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-04aede-776cc889f-47jhw"
Oct 30 07:25:29.147820 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:29.147787    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2grh\" (UniqueName: \"kubernetes.io/projected/bf4f18e1-ad1c-4315-9c07-d756d6d70476-kube-api-access-d2grh\") pod \"payload-processing-e2e-autores-04aede-5cd4c88478-kbvnt\" (UID: \"bf4f18e1-ad1c-4315-9c07-d756d6d70476\") " pod="openshift-ingress/payload-processing-e2e-autores-04aede-5cd4c88478-kbvnt"
Oct 30 07:25:29.256561 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:29.256511    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-04aede-5cd4c88478-kbvnt"
Oct 30 07:25:29.445956 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:25:29.445921    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc23e777f_2b9b_45fb_8110_62912dcbbaa6.slice/crio-e6d7bfaebf65798ce017d133c83d3eae7fb6c0178a19056d15dd8759abb3a24d WatchSource:0}: Error finding container e6d7bfaebf65798ce017d133c83d3eae7fb6c0178a19056d15dd8759abb3a24d: Status 404 returned error can't find the container with id e6d7bfaebf65798ce017d133c83d3eae7fb6c0178a19056d15dd8759abb3a24d
Oct 30 07:25:29.446117 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:29.446056    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-04aede-776cc889f-47jhw"]
Oct 30 07:25:29.573599 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:25:29.573557    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf4f18e1_ad1c_4315_9c07_d756d6d70476.slice/crio-a7741dead3d13aaafc67c7aa1f760bfb678e708e550f806b8e13500e5d7150eb WatchSource:0}: Error finding container a7741dead3d13aaafc67c7aa1f760bfb678e708e550f806b8e13500e5d7150eb: Status 404 returned error can't find the container with id a7741dead3d13aaafc67c7aa1f760bfb678e708e550f806b8e13500e5d7150eb
Oct 30 07:25:29.574862 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:29.574802    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-04aede-5cd4c88478-kbvnt"]
Oct 30 07:25:30.065947 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:30.065902    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-04aede-776cc889f-47jhw" event={"ID":"c23e777f-2b9b-45fb-8110-62912dcbbaa6","Type":"ContainerStarted","Data":"8d20287cb61e3868c7478b45b7d527b6067091566f9a07d7d2725b8fbc99c50e"}
Oct 30 07:25:30.066412 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:30.065972    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-04aede-776cc889f-47jhw"
Oct 30 07:25:30.066412 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:30.066000    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-04aede-776cc889f-47jhw" event={"ID":"c23e777f-2b9b-45fb-8110-62912dcbbaa6","Type":"ContainerStarted","Data":"e6d7bfaebf65798ce017d133c83d3eae7fb6c0178a19056d15dd8759abb3a24d"}
Oct 30 07:25:30.067732 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:30.067696    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-04aede-5cd4c88478-kbvnt" event={"ID":"bf4f18e1-ad1c-4315-9c07-d756d6d70476","Type":"ContainerStarted","Data":"390329941ea55222838eb235936ba87674826be3cf0a03aa5e901ad9b6034968"}
Oct 30 07:25:30.067871 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:30.067749    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-04aede-5cd4c88478-kbvnt" event={"ID":"bf4f18e1-ad1c-4315-9c07-d756d6d70476","Type":"ContainerStarted","Data":"a7741dead3d13aaafc67c7aa1f760bfb678e708e550f806b8e13500e5d7150eb"}
Oct 30 07:25:30.067871 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:30.067799    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-04aede-5cd4c88478-kbvnt"
Oct 30 07:25:30.110639 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:30.110579    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-04aede-776cc889f-47jhw" podStartSLOduration=2.110562586 podStartE2EDuration="2.110562586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:25:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:25:30.109112746 +0000 UTC m=+1688.718519717" watchObservedRunningTime="2026-10-30 07:25:30.110562586 +0000 UTC m=+1688.719969557"
Oct 30 07:25:30.155724 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:30.155671    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-04aede-5cd4c88478-kbvnt" podStartSLOduration=2.155658161 podStartE2EDuration="2.155658161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:25:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:25:30.152893105 +0000 UTC m=+1688.762300077" watchObservedRunningTime="2026-10-30 07:25:30.155658161 +0000 UTC m=+1688.765065131"
Oct 30 07:25:31.486791 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:31.486754    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj"]
Oct 30 07:25:31.492605 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:31.492573    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj"
Oct 30 07:25:31.508975 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:31.508943    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj"]
Oct 30 07:25:31.660719 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:31.660687    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1096c1ae-7e73-4107-ba58-bd4ea5096de2-istiod-ca-cert\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj\" (UID: \"1096c1ae-7e73-4107-ba58-bd4ea5096de2\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj"
Oct 30 07:25:31.660915 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:31.660733    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1096c1ae-7e73-4107-ba58-bd4ea5096de2-workload-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj\" (UID: \"1096c1ae-7e73-4107-ba58-bd4ea5096de2\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj"
Oct 30 07:25:31.660915 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:31.660768    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1096c1ae-7e73-4107-ba58-bd4ea5096de2-istio-token\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj\" (UID: \"1096c1ae-7e73-4107-ba58-bd4ea5096de2\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj"
Oct 30 07:25:31.660915 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:31.660885    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1096c1ae-7e73-4107-ba58-bd4ea5096de2-credential-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj\" (UID: \"1096c1ae-7e73-4107-ba58-bd4ea5096de2\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj"
Oct 30 07:25:31.661042 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:31.660920    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1096c1ae-7e73-4107-ba58-bd4ea5096de2-istio-envoy\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj\" (UID: \"1096c1ae-7e73-4107-ba58-bd4ea5096de2\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj"
Oct 30 07:25:31.661082 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:31.661058    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql2fq\" (UniqueName: \"kubernetes.io/projected/1096c1ae-7e73-4107-ba58-bd4ea5096de2-kube-api-access-ql2fq\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj\" (UID: \"1096c1ae-7e73-4107-ba58-bd4ea5096de2\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj"
Oct 30 07:25:31.661147 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:31.661122    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1096c1ae-7e73-4107-ba58-bd4ea5096de2-istio-data\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj\" (UID: \"1096c1ae-7e73-4107-ba58-bd4ea5096de2\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj"
Oct 30 07:25:31.661218 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:31.661189    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1096c1ae-7e73-4107-ba58-bd4ea5096de2-istio-podinfo\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj\" (UID: \"1096c1ae-7e73-4107-ba58-bd4ea5096de2\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj"
Oct 30 07:25:31.661280 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:31.661232    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1096c1ae-7e73-4107-ba58-bd4ea5096de2-workload-certs\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj\" (UID: \"1096c1ae-7e73-4107-ba58-bd4ea5096de2\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj"
Oct 30 07:25:31.762666 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:31.762595    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1096c1ae-7e73-4107-ba58-bd4ea5096de2-istiod-ca-cert\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj\" (UID: \"1096c1ae-7e73-4107-ba58-bd4ea5096de2\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj"
Oct 30 07:25:31.762666 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:31.762639    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1096c1ae-7e73-4107-ba58-bd4ea5096de2-workload-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj\" (UID: \"1096c1ae-7e73-4107-ba58-bd4ea5096de2\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj"
Oct 30 07:25:31.762893 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:31.762672    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1096c1ae-7e73-4107-ba58-bd4ea5096de2-istio-token\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj\" (UID: \"1096c1ae-7e73-4107-ba58-bd4ea5096de2\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj"
Oct 30 07:25:31.762893 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:31.762731    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1096c1ae-7e73-4107-ba58-bd4ea5096de2-credential-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj\" (UID: \"1096c1ae-7e73-4107-ba58-bd4ea5096de2\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj"
Oct 30 07:25:31.762893 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:31.762782    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1096c1ae-7e73-4107-ba58-bd4ea5096de2-istio-envoy\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj\" (UID: \"1096c1ae-7e73-4107-ba58-bd4ea5096de2\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj"
Oct 30 07:25:31.763035 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:31.762895    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql2fq\" (UniqueName: \"kubernetes.io/projected/1096c1ae-7e73-4107-ba58-bd4ea5096de2-kube-api-access-ql2fq\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj\" (UID: \"1096c1ae-7e73-4107-ba58-bd4ea5096de2\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj"
Oct 30 07:25:31.763035 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:31.762949    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1096c1ae-7e73-4107-ba58-bd4ea5096de2-istio-data\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj\" (UID: \"1096c1ae-7e73-4107-ba58-bd4ea5096de2\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj"
Oct 30 07:25:31.763035 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:31.762976    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1096c1ae-7e73-4107-ba58-bd4ea5096de2-istio-podinfo\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj\" (UID: \"1096c1ae-7e73-4107-ba58-bd4ea5096de2\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj"
Oct 30 07:25:31.763035 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:31.763011    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1096c1ae-7e73-4107-ba58-bd4ea5096de2-workload-certs\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj\" (UID: \"1096c1ae-7e73-4107-ba58-bd4ea5096de2\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj"
Oct 30 07:25:31.763207 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:31.763091    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1096c1ae-7e73-4107-ba58-bd4ea5096de2-workload-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj\" (UID: \"1096c1ae-7e73-4107-ba58-bd4ea5096de2\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj"
Oct 30 07:25:31.763207 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:31.763182    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1096c1ae-7e73-4107-ba58-bd4ea5096de2-credential-socket\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj\" (UID: \"1096c1ae-7e73-4107-ba58-bd4ea5096de2\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj"
Oct 30 07:25:31.763331 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:31.763307    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1096c1ae-7e73-4107-ba58-bd4ea5096de2-istiod-ca-cert\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj\" (UID: \"1096c1ae-7e73-4107-ba58-bd4ea5096de2\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj"
Oct 30 07:25:31.763442 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:31.763424    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1096c1ae-7e73-4107-ba58-bd4ea5096de2-workload-certs\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj\" (UID: \"1096c1ae-7e73-4107-ba58-bd4ea5096de2\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj"
Oct 30 07:25:31.763442 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:31.763434    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1096c1ae-7e73-4107-ba58-bd4ea5096de2-istio-data\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj\" (UID: \"1096c1ae-7e73-4107-ba58-bd4ea5096de2\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj"
Oct 30 07:25:31.765244 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:31.765224    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1096c1ae-7e73-4107-ba58-bd4ea5096de2-istio-envoy\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj\" (UID: \"1096c1ae-7e73-4107-ba58-bd4ea5096de2\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj"
Oct 30 07:25:31.765358 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:31.765295    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1096c1ae-7e73-4107-ba58-bd4ea5096de2-istio-podinfo\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj\" (UID: \"1096c1ae-7e73-4107-ba58-bd4ea5096de2\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj"
Oct 30 07:25:31.790373 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:31.790333    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1096c1ae-7e73-4107-ba58-bd4ea5096de2-istio-token\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj\" (UID: \"1096c1ae-7e73-4107-ba58-bd4ea5096de2\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj"
Oct 30 07:25:31.790932 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:31.790906    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql2fq\" (UniqueName: \"kubernetes.io/projected/1096c1ae-7e73-4107-ba58-bd4ea5096de2-kube-api-access-ql2fq\") pod \"e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj\" (UID: \"1096c1ae-7e73-4107-ba58-bd4ea5096de2\") " pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj"
Oct 30 07:25:31.813569 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:31.813534    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj"
Oct 30 07:25:31.866366 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:25:31.866318    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:25:32.142745 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:25:32.142711    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1096c1ae_7e73_4107_ba58_bd4ea5096de2.slice/crio-2a5d536c33ae644a1445ebe4137807f162df4ede7995f15021cd59c8e028e8a6 WatchSource:0}: Error finding container 2a5d536c33ae644a1445ebe4137807f162df4ede7995f15021cd59c8e028e8a6: Status 404 returned error can't find the container with id 2a5d536c33ae644a1445ebe4137807f162df4ede7995f15021cd59c8e028e8a6
Oct 30 07:25:32.143599 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:32.143569    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj"]
Oct 30 07:25:32.145358 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:32.145327    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:25:32.145452 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:32.145392    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:25:32.145452 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:32.145419    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:25:32.691139 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:32.691091    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-265384-7f894cf8b5-wccfq"]
Oct 30 07:25:32.695993 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:32.695955    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-265384-7f894cf8b5-wccfq"
Oct 30 07:25:32.714799 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:32.714588    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-265384-7f894cf8b5-wccfq"]
Oct 30 07:25:32.730841 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:32.723871    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-265384-serving-cert\""
Oct 30 07:25:32.895013 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:32.894973    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dda66230-cf6c-4f92-a50a-2edf07a1b0b8-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-265384-7f894cf8b5-wccfq\" (UID: \"dda66230-cf6c-4f92-a50a-2edf07a1b0b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-265384-7f894cf8b5-wccfq"
Oct 30 07:25:32.895228 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:32.895102    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb47v\" (UniqueName: \"kubernetes.io/projected/dda66230-cf6c-4f92-a50a-2edf07a1b0b8-kube-api-access-tb47v\") pod \"maas-api-e2e-shared-a-w1-265384-7f894cf8b5-wccfq\" (UID: \"dda66230-cf6c-4f92-a50a-2edf07a1b0b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-265384-7f894cf8b5-wccfq"
Oct 30 07:25:32.895305 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:32.895226    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dda66230-cf6c-4f92-a50a-2edf07a1b0b8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-265384-7f894cf8b5-wccfq\" (UID: \"dda66230-cf6c-4f92-a50a-2edf07a1b0b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-265384-7f894cf8b5-wccfq"
Oct 30 07:25:32.895305 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:32.895266    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dda66230-cf6c-4f92-a50a-2edf07a1b0b8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-265384-7f894cf8b5-wccfq\" (UID: \"dda66230-cf6c-4f92-a50a-2edf07a1b0b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-265384-7f894cf8b5-wccfq"
Oct 30 07:25:32.997372 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:32.997286    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dda66230-cf6c-4f92-a50a-2edf07a1b0b8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-265384-7f894cf8b5-wccfq\" (UID: \"dda66230-cf6c-4f92-a50a-2edf07a1b0b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-265384-7f894cf8b5-wccfq"
Oct 30 07:25:32.997372 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:32.997341    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dda66230-cf6c-4f92-a50a-2edf07a1b0b8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-265384-7f894cf8b5-wccfq\" (UID: \"dda66230-cf6c-4f92-a50a-2edf07a1b0b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-265384-7f894cf8b5-wccfq"
Oct 30 07:25:32.997568 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:32.997393    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dda66230-cf6c-4f92-a50a-2edf07a1b0b8-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-265384-7f894cf8b5-wccfq\" (UID: \"dda66230-cf6c-4f92-a50a-2edf07a1b0b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-265384-7f894cf8b5-wccfq"
Oct 30 07:25:32.997568 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:32.997471    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb47v\" (UniqueName: \"kubernetes.io/projected/dda66230-cf6c-4f92-a50a-2edf07a1b0b8-kube-api-access-tb47v\") pod \"maas-api-e2e-shared-a-w1-265384-7f894cf8b5-wccfq\" (UID: \"dda66230-cf6c-4f92-a50a-2edf07a1b0b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-265384-7f894cf8b5-wccfq"
Oct 30 07:25:32.998701 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:32.998668    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dda66230-cf6c-4f92-a50a-2edf07a1b0b8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-265384-7f894cf8b5-wccfq\" (UID: \"dda66230-cf6c-4f92-a50a-2edf07a1b0b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-265384-7f894cf8b5-wccfq"
Oct 30 07:25:33.000519 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:33.000489    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dda66230-cf6c-4f92-a50a-2edf07a1b0b8-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-265384-7f894cf8b5-wccfq\" (UID: \"dda66230-cf6c-4f92-a50a-2edf07a1b0b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-265384-7f894cf8b5-wccfq"
Oct 30 07:25:33.001041 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:33.001019    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dda66230-cf6c-4f92-a50a-2edf07a1b0b8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-265384-7f894cf8b5-wccfq\" (UID: \"dda66230-cf6c-4f92-a50a-2edf07a1b0b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-265384-7f894cf8b5-wccfq"
Oct 30 07:25:33.024575 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:33.024542    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb47v\" (UniqueName: \"kubernetes.io/projected/dda66230-cf6c-4f92-a50a-2edf07a1b0b8-kube-api-access-tb47v\") pod \"maas-api-e2e-shared-a-w1-265384-7f894cf8b5-wccfq\" (UID: \"dda66230-cf6c-4f92-a50a-2edf07a1b0b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-265384-7f894cf8b5-wccfq"
Oct 30 07:25:33.096936 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:33.096899    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj" event={"ID":"1096c1ae-7e73-4107-ba58-bd4ea5096de2","Type":"ContainerStarted","Data":"b6496a9ec85face9542aee7bce22549c3304ddfb5d188a98dce113bd8ac4826b"}
Oct 30 07:25:33.097095 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:33.096986    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj" event={"ID":"1096c1ae-7e73-4107-ba58-bd4ea5096de2","Type":"ContainerStarted","Data":"2a5d536c33ae644a1445ebe4137807f162df4ede7995f15021cd59c8e028e8a6"}
Oct 30 07:25:33.146744 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:33.146691    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj" podStartSLOduration=2.146677648 podStartE2EDuration="2.146677648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:25:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:25:33.144953584 +0000 UTC m=+1691.754360559" watchObservedRunningTime="2026-10-30 07:25:33.146677648 +0000 UTC m=+1691.756084617"
Oct 30 07:25:33.309046 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:33.309009    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-265384-7f894cf8b5-wccfq"
Oct 30 07:25:33.649097 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:25:33.649075    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddda66230_cf6c_4f92_a50a_2edf07a1b0b8.slice/crio-f139f30555d1ffeb17b7e3410b582089a33a71b102c8914b371c9953740aad49 WatchSource:0}: Error finding container f139f30555d1ffeb17b7e3410b582089a33a71b102c8914b371c9953740aad49: Status 404 returned error can't find the container with id f139f30555d1ffeb17b7e3410b582089a33a71b102c8914b371c9953740aad49
Oct 30 07:25:33.649373 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:33.649352    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-265384-7f894cf8b5-wccfq"]
Oct 30 07:25:33.650566 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:33.650550    6224 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 07:25:33.814400 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:33.814370    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj"
Oct 30 07:25:33.816387 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:33.816359    6224 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 30 07:25:33.816510 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:33.816417    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj" podUID="1096c1ae-7e73-4107-ba58-bd4ea5096de2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 30 07:25:34.104200 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:34.104165    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-265384-7f894cf8b5-wccfq" event={"ID":"dda66230-cf6c-4f92-a50a-2edf07a1b0b8","Type":"ContainerStarted","Data":"f139f30555d1ffeb17b7e3410b582089a33a71b102c8914b371c9953740aad49"}
Oct 30 07:25:34.814173 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:34.814138    6224 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 30 07:25:34.814335 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:34.814188    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj" podUID="1096c1ae-7e73-4107-ba58-bd4ea5096de2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 30 07:25:35.111905 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:35.111788    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-265384-7f894cf8b5-wccfq" event={"ID":"dda66230-cf6c-4f92-a50a-2edf07a1b0b8","Type":"ContainerStarted","Data":"b0f51c3703bb788b066ccf04d98c0c6bb7c65aac6432d2165cf0f3ac6c6cafa4"}
Oct 30 07:25:35.111905 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:35.111885    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-265384-7f894cf8b5-wccfq"
Oct 30 07:25:35.160773 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:35.160722    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-265384-7f894cf8b5-wccfq" podStartSLOduration=2.687520271 podStartE2EDuration="3.160708947s" totalImagesPullingTime="473.188676ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:25:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:25:35.158537728 +0000 UTC m=+1693.767944697" watchObservedRunningTime="2026-10-30 07:25:35.160708947 +0000 UTC m=+1693.770115916"
Oct 30 07:25:35.402881 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:35.399038    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-bd8589-7c4b96684-wqlnp"]
Oct 30 07:25:35.420183 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:35.420151    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-bd8589-7c4b96684-wqlnp"
Oct 30 07:25:35.438367 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:35.428616    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2l7w\" (UniqueName: \"kubernetes.io/projected/e0988047-1b97-49f4-96c4-a5da4a25d114-kube-api-access-f2l7w\") pod \"payload-processing-e2e-shared-a-w0-bd8589-7c4b96684-wqlnp\" (UID: \"e0988047-1b97-49f4-96c4-a5da4a25d114\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-bd8589-7c4b96684-wqlnp"
Oct 30 07:25:35.438367 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:35.428681    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0988047-1b97-49f4-96c4-a5da4a25d114-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-bd8589-7c4b96684-wqlnp\" (UID: \"e0988047-1b97-49f4-96c4-a5da4a25d114\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-bd8589-7c4b96684-wqlnp"
Oct 30 07:25:35.452826 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:35.449322    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-bd8589\""
Oct 30 07:25:35.457355 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:35.457328    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-bd8589-dockercfg-xkwsn\""
Oct 30 07:25:35.459777 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:35.459742    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-bd8589-7c4b96684-wqlnp"]
Oct 30 07:25:35.529716 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:35.529670    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2l7w\" (UniqueName: \"kubernetes.io/projected/e0988047-1b97-49f4-96c4-a5da4a25d114-kube-api-access-f2l7w\") pod \"payload-processing-e2e-shared-a-w0-bd8589-7c4b96684-wqlnp\" (UID: \"e0988047-1b97-49f4-96c4-a5da4a25d114\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-bd8589-7c4b96684-wqlnp"
Oct 30 07:25:35.529930 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:35.529758    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0988047-1b97-49f4-96c4-a5da4a25d114-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-bd8589-7c4b96684-wqlnp\" (UID: \"e0988047-1b97-49f4-96c4-a5da4a25d114\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-bd8589-7c4b96684-wqlnp"
Oct 30 07:25:35.531054 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:35.530874    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0988047-1b97-49f4-96c4-a5da4a25d114-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-bd8589-7c4b96684-wqlnp\" (UID: \"e0988047-1b97-49f4-96c4-a5da4a25d114\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-bd8589-7c4b96684-wqlnp"
Oct 30 07:25:35.560935 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:35.560908    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2l7w\" (UniqueName: \"kubernetes.io/projected/e0988047-1b97-49f4-96c4-a5da4a25d114-kube-api-access-f2l7w\") pod \"payload-processing-e2e-shared-a-w0-bd8589-7c4b96684-wqlnp\" (UID: \"e0988047-1b97-49f4-96c4-a5da4a25d114\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-bd8589-7c4b96684-wqlnp"
Oct 30 07:25:35.814455 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:35.814425    6224 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 30 07:25:35.814647 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:35.814482    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj" podUID="1096c1ae-7e73-4107-ba58-bd4ea5096de2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 30 07:25:35.831958 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:35.831933    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-bd8589-7c4b96684-wqlnp"
Oct 30 07:25:36.019613 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:36.019573    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-932c00-79d49cb7d7-mpkdp"
Oct 30 07:25:36.159170 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:25:36.159126    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0988047_1b97_49f4_96c4_a5da4a25d114.slice/crio-5bd5341ef76174103b4d969c167700fc03ec9c8aa823299e7ea119144515b9ca WatchSource:0}: Error finding container 5bd5341ef76174103b4d969c167700fc03ec9c8aa823299e7ea119144515b9ca: Status 404 returned error can't find the container with id 5bd5341ef76174103b4d969c167700fc03ec9c8aa823299e7ea119144515b9ca
Oct 30 07:25:36.159515 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:36.159465    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-bd8589-7c4b96684-wqlnp"]
Oct 30 07:25:36.814049 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:36.814007    6224 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 30 07:25:36.814218 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:36.814065    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj" podUID="1096c1ae-7e73-4107-ba58-bd4ea5096de2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 30 07:25:37.127477 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:37.127380    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-bd8589-7c4b96684-wqlnp" event={"ID":"e0988047-1b97-49f4-96c4-a5da4a25d114","Type":"ContainerStarted","Data":"48436f50ba4dc1d50634248e25191e7dbcc333a65c182d498eded4710caf6bba"}
Oct 30 07:25:37.127477 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:37.127445    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-bd8589-7c4b96684-wqlnp" event={"ID":"e0988047-1b97-49f4-96c4-a5da4a25d114","Type":"ContainerStarted","Data":"5bd5341ef76174103b4d969c167700fc03ec9c8aa823299e7ea119144515b9ca"}
Oct 30 07:25:37.127689 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:37.127606    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-bd8589-7c4b96684-wqlnp"
Oct 30 07:25:37.814140 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:37.814107    6224 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 30 07:25:37.814548 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:37.814160    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj" podUID="1096c1ae-7e73-4107-ba58-bd4ea5096de2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 30 07:25:38.814054 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:38.814021    6224 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 30 07:25:38.814238 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:38.814071    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj" podUID="1096c1ae-7e73-4107-ba58-bd4ea5096de2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 30 07:25:39.818608 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:39.818578    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj"
Oct 30 07:25:39.819068 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:39.818912    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj"
Oct 30 07:25:39.819604 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:39.819584    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj"
Oct 30 07:25:39.882264 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:39.882222    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-bd8589-7c4b96684-wqlnp" podStartSLOduration=4.882209289 podStartE2EDuration="4.882209289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:25:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:25:37.194098162 +0000 UTC m=+1695.803505144" watchObservedRunningTime="2026-10-30 07:25:39.882209289 +0000 UTC m=+1698.491616282"
Oct 30 07:25:41.124274 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:41.124227    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-04aede-5cd4c88478-kbvnt"
Oct 30 07:25:41.126098 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:41.126075    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-04aede-776cc889f-47jhw"
Oct 30 07:25:41.130993 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:41.130969    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:25:41.133756 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:41.133709    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-265384-7f894cf8b5-wccfq"
Oct 30 07:25:45.945950 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:45.945914    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl"]
Oct 30 07:25:45.949874 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:45.949855    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl"
Oct 30 07:25:45.975957 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:45.975930    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl"]
Oct 30 07:25:46.083453 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:46.082124    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a470b54-ad63-40f7-b306-f8ee1dc8cdcd-workload-certs\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl\" (UID: \"1a470b54-ad63-40f7-b306-f8ee1dc8cdcd\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl"
Oct 30 07:25:46.083453 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:46.082174    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a470b54-ad63-40f7-b306-f8ee1dc8cdcd-credential-socket\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl\" (UID: \"1a470b54-ad63-40f7-b306-f8ee1dc8cdcd\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl"
Oct 30 07:25:46.083453 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:46.082264    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a470b54-ad63-40f7-b306-f8ee1dc8cdcd-istio-token\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl\" (UID: \"1a470b54-ad63-40f7-b306-f8ee1dc8cdcd\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl"
Oct 30 07:25:46.083453 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:46.082365    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a470b54-ad63-40f7-b306-f8ee1dc8cdcd-istio-envoy\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl\" (UID: \"1a470b54-ad63-40f7-b306-f8ee1dc8cdcd\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl"
Oct 30 07:25:46.083453 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:46.082419    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a470b54-ad63-40f7-b306-f8ee1dc8cdcd-istio-data\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl\" (UID: \"1a470b54-ad63-40f7-b306-f8ee1dc8cdcd\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl"
Oct 30 07:25:46.083453 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:46.082463    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2t4h\" (UniqueName: \"kubernetes.io/projected/1a470b54-ad63-40f7-b306-f8ee1dc8cdcd-kube-api-access-j2t4h\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl\" (UID: \"1a470b54-ad63-40f7-b306-f8ee1dc8cdcd\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl"
Oct 30 07:25:46.083453 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:46.082516    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a470b54-ad63-40f7-b306-f8ee1dc8cdcd-workload-socket\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl\" (UID: \"1a470b54-ad63-40f7-b306-f8ee1dc8cdcd\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl"
Oct 30 07:25:46.083453 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:46.082551    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a470b54-ad63-40f7-b306-f8ee1dc8cdcd-istio-podinfo\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl\" (UID: \"1a470b54-ad63-40f7-b306-f8ee1dc8cdcd\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl"
Oct 30 07:25:46.083453 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:46.082575    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a470b54-ad63-40f7-b306-f8ee1dc8cdcd-istiod-ca-cert\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl\" (UID: \"1a470b54-ad63-40f7-b306-f8ee1dc8cdcd\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl"
Oct 30 07:25:46.184226 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:46.184193    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a470b54-ad63-40f7-b306-f8ee1dc8cdcd-istio-podinfo\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl\" (UID: \"1a470b54-ad63-40f7-b306-f8ee1dc8cdcd\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl"
Oct 30 07:25:46.184431 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:46.184234    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a470b54-ad63-40f7-b306-f8ee1dc8cdcd-istiod-ca-cert\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl\" (UID: \"1a470b54-ad63-40f7-b306-f8ee1dc8cdcd\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl"
Oct 30 07:25:46.184431 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:46.184296    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a470b54-ad63-40f7-b306-f8ee1dc8cdcd-workload-certs\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl\" (UID: \"1a470b54-ad63-40f7-b306-f8ee1dc8cdcd\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl"
Oct 30 07:25:46.184431 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:46.184320    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a470b54-ad63-40f7-b306-f8ee1dc8cdcd-credential-socket\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl\" (UID: \"1a470b54-ad63-40f7-b306-f8ee1dc8cdcd\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl"
Oct 30 07:25:46.184431 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:46.184374    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a470b54-ad63-40f7-b306-f8ee1dc8cdcd-istio-token\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl\" (UID: \"1a470b54-ad63-40f7-b306-f8ee1dc8cdcd\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl"
Oct 30 07:25:46.184639 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:46.184454    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a470b54-ad63-40f7-b306-f8ee1dc8cdcd-istio-envoy\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl\" (UID: \"1a470b54-ad63-40f7-b306-f8ee1dc8cdcd\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl"
Oct 30 07:25:46.184639 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:46.184512    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a470b54-ad63-40f7-b306-f8ee1dc8cdcd-istio-data\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl\" (UID: \"1a470b54-ad63-40f7-b306-f8ee1dc8cdcd\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl"
Oct 30 07:25:46.184639 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:46.184569    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2t4h\" (UniqueName: \"kubernetes.io/projected/1a470b54-ad63-40f7-b306-f8ee1dc8cdcd-kube-api-access-j2t4h\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl\" (UID: \"1a470b54-ad63-40f7-b306-f8ee1dc8cdcd\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl"
Oct 30 07:25:46.184819 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:46.184637    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a470b54-ad63-40f7-b306-f8ee1dc8cdcd-workload-socket\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl\" (UID: \"1a470b54-ad63-40f7-b306-f8ee1dc8cdcd\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl"
Oct 30 07:25:46.184819 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:46.184753    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a470b54-ad63-40f7-b306-f8ee1dc8cdcd-workload-certs\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl\" (UID: \"1a470b54-ad63-40f7-b306-f8ee1dc8cdcd\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl"
Oct 30 07:25:46.184935 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:46.184801    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a470b54-ad63-40f7-b306-f8ee1dc8cdcd-credential-socket\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl\" (UID: \"1a470b54-ad63-40f7-b306-f8ee1dc8cdcd\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl"
Oct 30 07:25:46.185029 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:46.185005    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a470b54-ad63-40f7-b306-f8ee1dc8cdcd-workload-socket\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl\" (UID: \"1a470b54-ad63-40f7-b306-f8ee1dc8cdcd\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl"
Oct 30 07:25:46.185147 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:46.185111    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a470b54-ad63-40f7-b306-f8ee1dc8cdcd-istio-data\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl\" (UID: \"1a470b54-ad63-40f7-b306-f8ee1dc8cdcd\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl"
Oct 30 07:25:46.185276 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:46.185205    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a470b54-ad63-40f7-b306-f8ee1dc8cdcd-istiod-ca-cert\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl\" (UID: \"1a470b54-ad63-40f7-b306-f8ee1dc8cdcd\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl"
Oct 30 07:25:46.186854 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:46.186827    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a470b54-ad63-40f7-b306-f8ee1dc8cdcd-istio-podinfo\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl\" (UID: \"1a470b54-ad63-40f7-b306-f8ee1dc8cdcd\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl"
Oct 30 07:25:46.186854 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:46.186834    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a470b54-ad63-40f7-b306-f8ee1dc8cdcd-istio-envoy\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl\" (UID: \"1a470b54-ad63-40f7-b306-f8ee1dc8cdcd\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl"
Oct 30 07:25:46.209646 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:46.209591    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a470b54-ad63-40f7-b306-f8ee1dc8cdcd-istio-token\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl\" (UID: \"1a470b54-ad63-40f7-b306-f8ee1dc8cdcd\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl"
Oct 30 07:25:46.209743 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:46.209668    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2t4h\" (UniqueName: \"kubernetes.io/projected/1a470b54-ad63-40f7-b306-f8ee1dc8cdcd-kube-api-access-j2t4h\") pod \"e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl\" (UID: \"1a470b54-ad63-40f7-b306-f8ee1dc8cdcd\") " pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl"
Oct 30 07:25:46.263062 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:46.263034    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl"
Oct 30 07:25:46.583128 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:25:46.583091    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a470b54_ad63_40f7_b306_f8ee1dc8cdcd.slice/crio-a68dfdb117487926c4abc34778881f396884fdaff79ea43e5484490d4840a897 WatchSource:0}: Error finding container a68dfdb117487926c4abc34778881f396884fdaff79ea43e5484490d4840a897: Status 404 returned error can't find the container with id a68dfdb117487926c4abc34778881f396884fdaff79ea43e5484490d4840a897
Oct 30 07:25:46.583857 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:46.583827    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl"]
Oct 30 07:25:46.585643 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:46.585604    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:25:46.585741 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:46.585688    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:25:46.585741 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:46.585731    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:25:46.804293 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:46.804255    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g"]
Oct 30 07:25:46.809113 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:46.809089    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g"
Oct 30 07:25:46.862992 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:46.852253    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g"]
Oct 30 07:25:46.904859 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:46.895619    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6169ad23-c3a7-40de-bfd2-8f232a428238-credential-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g\" (UID: \"6169ad23-c3a7-40de-bfd2-8f232a428238\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g"
Oct 30 07:25:46.904859 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:46.895687    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6169ad23-c3a7-40de-bfd2-8f232a428238-istio-token\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g\" (UID: \"6169ad23-c3a7-40de-bfd2-8f232a428238\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g"
Oct 30 07:25:46.904859 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:46.895919    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6169ad23-c3a7-40de-bfd2-8f232a428238-istio-data\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g\" (UID: \"6169ad23-c3a7-40de-bfd2-8f232a428238\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g"
Oct 30 07:25:46.904859 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:46.895980    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6169ad23-c3a7-40de-bfd2-8f232a428238-istio-podinfo\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g\" (UID: \"6169ad23-c3a7-40de-bfd2-8f232a428238\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g"
Oct 30 07:25:46.904859 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:46.896014    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6169ad23-c3a7-40de-bfd2-8f232a428238-workload-certs\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g\" (UID: \"6169ad23-c3a7-40de-bfd2-8f232a428238\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g"
Oct 30 07:25:46.904859 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:46.896128    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6169ad23-c3a7-40de-bfd2-8f232a428238-workload-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g\" (UID: \"6169ad23-c3a7-40de-bfd2-8f232a428238\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g"
Oct 30 07:25:46.904859 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:46.896158    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6169ad23-c3a7-40de-bfd2-8f232a428238-istio-envoy\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g\" (UID: \"6169ad23-c3a7-40de-bfd2-8f232a428238\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g"
Oct 30 07:25:46.904859 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:46.896235    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwpk5\" (UniqueName: \"kubernetes.io/projected/6169ad23-c3a7-40de-bfd2-8f232a428238-kube-api-access-pwpk5\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g\" (UID: \"6169ad23-c3a7-40de-bfd2-8f232a428238\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g"
Oct 30 07:25:46.904859 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:46.896276    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6169ad23-c3a7-40de-bfd2-8f232a428238-istiod-ca-cert\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g\" (UID: \"6169ad23-c3a7-40de-bfd2-8f232a428238\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g"
Oct 30 07:25:46.997377 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:46.997334    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6169ad23-c3a7-40de-bfd2-8f232a428238-credential-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g\" (UID: \"6169ad23-c3a7-40de-bfd2-8f232a428238\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g"
Oct 30 07:25:46.997861 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:46.997412    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6169ad23-c3a7-40de-bfd2-8f232a428238-istio-token\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g\" (UID: \"6169ad23-c3a7-40de-bfd2-8f232a428238\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g"
Oct 30 07:25:46.997861 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:46.997464    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6169ad23-c3a7-40de-bfd2-8f232a428238-istio-data\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g\" (UID: \"6169ad23-c3a7-40de-bfd2-8f232a428238\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g"
Oct 30 07:25:46.997861 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:46.997508    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6169ad23-c3a7-40de-bfd2-8f232a428238-istio-podinfo\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g\" (UID: \"6169ad23-c3a7-40de-bfd2-8f232a428238\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g"
Oct 30 07:25:46.997861 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:46.997553    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6169ad23-c3a7-40de-bfd2-8f232a428238-workload-certs\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g\" (UID: \"6169ad23-c3a7-40de-bfd2-8f232a428238\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g"
Oct 30 07:25:46.997861 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:46.997652    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6169ad23-c3a7-40de-bfd2-8f232a428238-workload-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g\" (UID: \"6169ad23-c3a7-40de-bfd2-8f232a428238\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g"
Oct 30 07:25:46.997861 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:46.997697    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6169ad23-c3a7-40de-bfd2-8f232a428238-istio-envoy\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g\" (UID: \"6169ad23-c3a7-40de-bfd2-8f232a428238\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g"
Oct 30 07:25:46.997861 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:46.997772    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwpk5\" (UniqueName: \"kubernetes.io/projected/6169ad23-c3a7-40de-bfd2-8f232a428238-kube-api-access-pwpk5\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g\" (UID: \"6169ad23-c3a7-40de-bfd2-8f232a428238\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g"
Oct 30 07:25:46.997861 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:46.997850    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6169ad23-c3a7-40de-bfd2-8f232a428238-credential-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g\" (UID: \"6169ad23-c3a7-40de-bfd2-8f232a428238\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g"
Oct 30 07:25:46.998335 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:46.997856    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6169ad23-c3a7-40de-bfd2-8f232a428238-istiod-ca-cert\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g\" (UID: \"6169ad23-c3a7-40de-bfd2-8f232a428238\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g"
Oct 30 07:25:46.998335 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:46.998321    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6169ad23-c3a7-40de-bfd2-8f232a428238-workload-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g\" (UID: \"6169ad23-c3a7-40de-bfd2-8f232a428238\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g"
Oct 30 07:25:46.998433 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:46.998416    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6169ad23-c3a7-40de-bfd2-8f232a428238-istio-data\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g\" (UID: \"6169ad23-c3a7-40de-bfd2-8f232a428238\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g"
Oct 30 07:25:46.998507 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:46.998482    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6169ad23-c3a7-40de-bfd2-8f232a428238-workload-certs\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g\" (UID: \"6169ad23-c3a7-40de-bfd2-8f232a428238\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g"
Oct 30 07:25:46.999361 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:46.999333    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6169ad23-c3a7-40de-bfd2-8f232a428238-istiod-ca-cert\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g\" (UID: \"6169ad23-c3a7-40de-bfd2-8f232a428238\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g"
Oct 30 07:25:47.000523 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:47.000499    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6169ad23-c3a7-40de-bfd2-8f232a428238-istio-envoy\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g\" (UID: \"6169ad23-c3a7-40de-bfd2-8f232a428238\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g"
Oct 30 07:25:47.000731 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:47.000710    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6169ad23-c3a7-40de-bfd2-8f232a428238-istio-podinfo\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g\" (UID: \"6169ad23-c3a7-40de-bfd2-8f232a428238\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g"
Oct 30 07:25:47.022637 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:47.022608    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6169ad23-c3a7-40de-bfd2-8f232a428238-istio-token\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g\" (UID: \"6169ad23-c3a7-40de-bfd2-8f232a428238\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g"
Oct 30 07:25:47.022909 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:47.022885    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwpk5\" (UniqueName: \"kubernetes.io/projected/6169ad23-c3a7-40de-bfd2-8f232a428238-kube-api-access-pwpk5\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g\" (UID: \"6169ad23-c3a7-40de-bfd2-8f232a428238\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g"
Oct 30 07:25:47.125855 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:47.125752    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g"
Oct 30 07:25:47.226648 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:47.225899    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl" event={"ID":"1a470b54-ad63-40f7-b306-f8ee1dc8cdcd","Type":"ContainerStarted","Data":"6d13c4ff0518e5f5b38459897146a543bb2115422d7bfb74102c81906fbbc42f"}
Oct 30 07:25:47.226648 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:47.225965    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl" event={"ID":"1a470b54-ad63-40f7-b306-f8ee1dc8cdcd","Type":"ContainerStarted","Data":"a68dfdb117487926c4abc34778881f396884fdaff79ea43e5484490d4840a897"}
Oct 30 07:25:47.265933 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:47.264696    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl"
Oct 30 07:25:47.266115 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:47.265964    6224 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 30 07:25:47.266115 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:47.266016    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl" podUID="1a470b54-ad63-40f7-b306-f8ee1dc8cdcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 30 07:25:47.285742 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:47.285679    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl" podStartSLOduration=2.285659419 podStartE2EDuration="2.285659419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:25:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:25:47.282319051 +0000 UTC m=+1705.891726022" watchObservedRunningTime="2026-10-30 07:25:47.285659419 +0000 UTC m=+1705.895066412"
Oct 30 07:25:47.451624 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:47.451595    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g"]
Oct 30 07:25:47.453608 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:47.453568    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:25:47.453683 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:47.453642    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:25:47.453726 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:47.453679    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:25:48.136978 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:48.136950    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-bd8589-7c4b96684-wqlnp"
Oct 30 07:25:48.236279 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:48.236241    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g" event={"ID":"6169ad23-c3a7-40de-bfd2-8f232a428238","Type":"ContainerStarted","Data":"95a0c499d0623ec7ee4f606403bcb87f45ed644f99923fe6e7cd98e0cd6934fd"}
Oct 30 07:25:48.236573 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:48.236303    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g" event={"ID":"6169ad23-c3a7-40de-bfd2-8f232a428238","Type":"ContainerStarted","Data":"36f2843cf3cfd54545508a06328a139c17fe6ab3e7323fcfbba36e1152ffa1ce"}
Oct 30 07:25:48.286830 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:48.283380    6224 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 30 07:25:48.286830 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:48.283458    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl" podUID="1a470b54-ad63-40f7-b306-f8ee1dc8cdcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 30 07:25:48.303844 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:48.303014    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g" podStartSLOduration=2.30299302 podStartE2EDuration="2.30299302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:25:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:25:48.302139693 +0000 UTC m=+1706.911546676" watchObservedRunningTime="2026-10-30 07:25:48.30299302 +0000 UTC m=+1706.912399990"
Oct 30 07:25:49.126864 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:49.126828    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g"
Oct 30 07:25:49.128246 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:49.128223    6224 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 30 07:25:49.128323 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:49.128273    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g" podUID="6169ad23-c3a7-40de-bfd2-8f232a428238" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 30 07:25:49.284831 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:49.267231    6224 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 30 07:25:49.284831 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:49.267291    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl" podUID="1a470b54-ad63-40f7-b306-f8ee1dc8cdcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 30 07:25:50.126461 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:50.126425    6224 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 30 07:25:50.126647 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:50.126479    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g" podUID="6169ad23-c3a7-40de-bfd2-8f232a428238" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 30 07:25:50.287624 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:50.287192    6224 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 30 07:25:50.287624 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:50.287256    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl" podUID="1a470b54-ad63-40f7-b306-f8ee1dc8cdcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 30 07:25:51.126760 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:51.126724    6224 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 30 07:25:51.126953 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:51.126772    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g" podUID="6169ad23-c3a7-40de-bfd2-8f232a428238" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 30 07:25:51.283830 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:51.275304    6224 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 30 07:25:51.283830 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:51.275379    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl" podUID="1a470b54-ad63-40f7-b306-f8ee1dc8cdcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 30 07:25:52.126359 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:52.126134    6224 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 30 07:25:52.126359 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:52.126182    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g" podUID="6169ad23-c3a7-40de-bfd2-8f232a428238" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 30 07:25:52.289852 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:52.284191    6224 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 30 07:25:52.289852 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:52.284252    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl" podUID="1a470b54-ad63-40f7-b306-f8ee1dc8cdcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 30 07:25:53.126477 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:53.126440    6224 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 30 07:25:53.126890 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:53.126498    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g" podUID="6169ad23-c3a7-40de-bfd2-8f232a428238" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 30 07:25:53.267832 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:53.267166    6224 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 30 07:25:53.267832 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:53.267230    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl" podUID="1a470b54-ad63-40f7-b306-f8ee1dc8cdcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 30 07:25:54.126249 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:54.126209    6224 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 30 07:25:54.126572 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:54.126535    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g" podUID="6169ad23-c3a7-40de-bfd2-8f232a428238" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 30 07:25:54.573170 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:54.573121    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl"
Oct 30 07:25:54.574247 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:54.574070    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl"
Oct 30 07:25:54.575172 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:54.575136    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl"
Oct 30 07:25:55.205529 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:55.205488    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g"
Oct 30 07:25:55.206156 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:55.206096    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g"
Oct 30 07:25:55.207160 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:55.207138    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g"
Oct 30 07:25:57.165424 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:57.165392    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45"]
Oct 30 07:25:57.171246 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:57.171223    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45"
Oct 30 07:25:57.196134 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:57.196083    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45"]
Oct 30 07:25:57.214507 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:57.201289    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04947652-67c7-45de-a2dc-5c2856bfdad8-istiod-ca-cert\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45\" (UID: \"04947652-67c7-45de-a2dc-5c2856bfdad8\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45"
Oct 30 07:25:57.214847 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:57.214817    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04947652-67c7-45de-a2dc-5c2856bfdad8-istio-podinfo\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45\" (UID: \"04947652-67c7-45de-a2dc-5c2856bfdad8\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45"
Oct 30 07:25:57.214997 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:57.214901    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04947652-67c7-45de-a2dc-5c2856bfdad8-istio-envoy\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45\" (UID: \"04947652-67c7-45de-a2dc-5c2856bfdad8\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45"
Oct 30 07:25:57.214997 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:57.214926    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04947652-67c7-45de-a2dc-5c2856bfdad8-istio-data\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45\" (UID: \"04947652-67c7-45de-a2dc-5c2856bfdad8\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45"
Oct 30 07:25:57.215129 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:57.215098    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04947652-67c7-45de-a2dc-5c2856bfdad8-workload-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45\" (UID: \"04947652-67c7-45de-a2dc-5c2856bfdad8\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45"
Oct 30 07:25:57.215181 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:57.215153    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44c6m\" (UniqueName: \"kubernetes.io/projected/04947652-67c7-45de-a2dc-5c2856bfdad8-kube-api-access-44c6m\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45\" (UID: \"04947652-67c7-45de-a2dc-5c2856bfdad8\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45"
Oct 30 07:25:57.215302 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:57.215282    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04947652-67c7-45de-a2dc-5c2856bfdad8-workload-certs\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45\" (UID: \"04947652-67c7-45de-a2dc-5c2856bfdad8\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45"
Oct 30 07:25:57.215478 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:57.215457    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04947652-67c7-45de-a2dc-5c2856bfdad8-istio-token\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45\" (UID: \"04947652-67c7-45de-a2dc-5c2856bfdad8\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45"
Oct 30 07:25:57.215563 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:57.215498    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04947652-67c7-45de-a2dc-5c2856bfdad8-credential-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45\" (UID: \"04947652-67c7-45de-a2dc-5c2856bfdad8\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45"
Oct 30 07:25:57.317161 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:57.317119    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04947652-67c7-45de-a2dc-5c2856bfdad8-istio-token\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45\" (UID: \"04947652-67c7-45de-a2dc-5c2856bfdad8\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45"
Oct 30 07:25:57.317341 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:57.317171    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04947652-67c7-45de-a2dc-5c2856bfdad8-credential-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45\" (UID: \"04947652-67c7-45de-a2dc-5c2856bfdad8\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45"
Oct 30 07:25:57.317341 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:57.317213    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04947652-67c7-45de-a2dc-5c2856bfdad8-istiod-ca-cert\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45\" (UID: \"04947652-67c7-45de-a2dc-5c2856bfdad8\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45"
Oct 30 07:25:57.317341 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:57.317237    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04947652-67c7-45de-a2dc-5c2856bfdad8-istio-podinfo\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45\" (UID: \"04947652-67c7-45de-a2dc-5c2856bfdad8\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45"
Oct 30 07:25:57.317341 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:57.317273    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04947652-67c7-45de-a2dc-5c2856bfdad8-istio-envoy\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45\" (UID: \"04947652-67c7-45de-a2dc-5c2856bfdad8\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45"
Oct 30 07:25:57.317341 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:57.317297    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04947652-67c7-45de-a2dc-5c2856bfdad8-istio-data\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45\" (UID: \"04947652-67c7-45de-a2dc-5c2856bfdad8\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45"
Oct 30 07:25:57.317607 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:57.317365    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04947652-67c7-45de-a2dc-5c2856bfdad8-workload-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45\" (UID: \"04947652-67c7-45de-a2dc-5c2856bfdad8\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45"
Oct 30 07:25:57.317607 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:57.317402    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44c6m\" (UniqueName: \"kubernetes.io/projected/04947652-67c7-45de-a2dc-5c2856bfdad8-kube-api-access-44c6m\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45\" (UID: \"04947652-67c7-45de-a2dc-5c2856bfdad8\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45"
Oct 30 07:25:57.317607 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:57.317447    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04947652-67c7-45de-a2dc-5c2856bfdad8-workload-certs\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45\" (UID: \"04947652-67c7-45de-a2dc-5c2856bfdad8\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45"
Oct 30 07:25:57.317777 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:57.317622    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04947652-67c7-45de-a2dc-5c2856bfdad8-credential-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45\" (UID: \"04947652-67c7-45de-a2dc-5c2856bfdad8\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45"
Oct 30 07:25:57.317777 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:57.317765    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04947652-67c7-45de-a2dc-5c2856bfdad8-workload-certs\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45\" (UID: \"04947652-67c7-45de-a2dc-5c2856bfdad8\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45"
Oct 30 07:25:57.317908 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:57.317799    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04947652-67c7-45de-a2dc-5c2856bfdad8-workload-socket\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45\" (UID: \"04947652-67c7-45de-a2dc-5c2856bfdad8\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45"
Oct 30 07:25:57.317972 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:57.317960    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04947652-67c7-45de-a2dc-5c2856bfdad8-istio-data\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45\" (UID: \"04947652-67c7-45de-a2dc-5c2856bfdad8\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45"
Oct 30 07:25:57.318114 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:57.318090    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04947652-67c7-45de-a2dc-5c2856bfdad8-istiod-ca-cert\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45\" (UID: \"04947652-67c7-45de-a2dc-5c2856bfdad8\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45"
Oct 30 07:25:57.319610 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:57.319593    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04947652-67c7-45de-a2dc-5c2856bfdad8-istio-envoy\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45\" (UID: \"04947652-67c7-45de-a2dc-5c2856bfdad8\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45"
Oct 30 07:25:57.319854 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:57.319835    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04947652-67c7-45de-a2dc-5c2856bfdad8-istio-podinfo\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45\" (UID: \"04947652-67c7-45de-a2dc-5c2856bfdad8\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45"
Oct 30 07:25:57.341996 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:57.341971    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04947652-67c7-45de-a2dc-5c2856bfdad8-istio-token\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45\" (UID: \"04947652-67c7-45de-a2dc-5c2856bfdad8\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45"
Oct 30 07:25:57.342221 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:57.342200    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44c6m\" (UniqueName: \"kubernetes.io/projected/04947652-67c7-45de-a2dc-5c2856bfdad8-kube-api-access-44c6m\") pod \"e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45\" (UID: \"04947652-67c7-45de-a2dc-5c2856bfdad8\") " pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45"
Oct 30 07:25:57.486093 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:57.486018    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45"
Oct 30 07:25:57.834337 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:25:57.834296    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04947652_67c7_45de_a2dc_5c2856bfdad8.slice/crio-82ba5abeec7ebcc9d1df96f20909ea3a73edb40e85ecd63cc4857b6a4601453a WatchSource:0}: Error finding container 82ba5abeec7ebcc9d1df96f20909ea3a73edb40e85ecd63cc4857b6a4601453a: Status 404 returned error can't find the container with id 82ba5abeec7ebcc9d1df96f20909ea3a73edb40e85ecd63cc4857b6a4601453a
Oct 30 07:25:57.834547 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:57.834525    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45"]
Oct 30 07:25:57.836708 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:57.836678    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:25:57.836777 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:57.836738    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:25:57.836777 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:57.836767    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:25:58.043279 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:58.043247    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc"]
Oct 30 07:25:58.061181 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:58.061150    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc"
Oct 30 07:25:58.072870 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:58.072841    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc"]
Oct 30 07:25:58.088126 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:58.088065    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-f5ceca-openshift-default-dockercfg-4ch7q\""
Oct 30 07:25:58.127483 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:58.127449    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50bc25e6-c53f-4d16-b1fd-738cdd8c6a10-workload-socket\") pod \"e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc\" (UID: \"50bc25e6-c53f-4d16-b1fd-738cdd8c6a10\") " pod="openshift-ingress/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc"
Oct 30 07:25:58.127650 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:58.127496    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50bc25e6-c53f-4d16-b1fd-738cdd8c6a10-credential-socket\") pod \"e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc\" (UID: \"50bc25e6-c53f-4d16-b1fd-738cdd8c6a10\") " pod="openshift-ingress/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc"
Oct 30 07:25:58.127650 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:58.127610    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50bc25e6-c53f-4d16-b1fd-738cdd8c6a10-istio-data\") pod \"e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc\" (UID: \"50bc25e6-c53f-4d16-b1fd-738cdd8c6a10\") " pod="openshift-ingress/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc"
Oct 30 07:25:58.127825 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:58.127711    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50bc25e6-c53f-4d16-b1fd-738cdd8c6a10-istio-token\") pod \"e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc\" (UID: \"50bc25e6-c53f-4d16-b1fd-738cdd8c6a10\") " pod="openshift-ingress/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc"
Oct 30 07:25:58.127825 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:58.127779    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50bc25e6-c53f-4d16-b1fd-738cdd8c6a10-workload-certs\") pod \"e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc\" (UID: \"50bc25e6-c53f-4d16-b1fd-738cdd8c6a10\") " pod="openshift-ingress/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc"
Oct 30 07:25:58.127929 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:58.127900    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spknf\" (UniqueName: \"kubernetes.io/projected/50bc25e6-c53f-4d16-b1fd-738cdd8c6a10-kube-api-access-spknf\") pod \"e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc\" (UID: \"50bc25e6-c53f-4d16-b1fd-738cdd8c6a10\") " pod="openshift-ingress/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc"
Oct 30 07:25:58.128029 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:58.128009    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50bc25e6-c53f-4d16-b1fd-738cdd8c6a10-istiod-ca-cert\") pod \"e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc\" (UID: \"50bc25e6-c53f-4d16-b1fd-738cdd8c6a10\") " pod="openshift-ingress/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc"
Oct 30 07:25:58.128111 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:58.128096    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50bc25e6-c53f-4d16-b1fd-738cdd8c6a10-istio-envoy\") pod \"e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc\" (UID: \"50bc25e6-c53f-4d16-b1fd-738cdd8c6a10\") " pod="openshift-ingress/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc"
Oct 30 07:25:58.128218 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:58.128203    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50bc25e6-c53f-4d16-b1fd-738cdd8c6a10-istio-podinfo\") pod \"e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc\" (UID: \"50bc25e6-c53f-4d16-b1fd-738cdd8c6a10\") " pod="openshift-ingress/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc"
Oct 30 07:25:58.230200 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:58.230163    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50bc25e6-c53f-4d16-b1fd-738cdd8c6a10-istio-data\") pod \"e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc\" (UID: \"50bc25e6-c53f-4d16-b1fd-738cdd8c6a10\") " pod="openshift-ingress/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc"
Oct 30 07:25:58.231231 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:58.231210    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50bc25e6-c53f-4d16-b1fd-738cdd8c6a10-istio-token\") pod \"e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc\" (UID: \"50bc25e6-c53f-4d16-b1fd-738cdd8c6a10\") " pod="openshift-ingress/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc"
Oct 30 07:25:58.231570 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:58.231548    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50bc25e6-c53f-4d16-b1fd-738cdd8c6a10-workload-certs\") pod \"e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc\" (UID: \"50bc25e6-c53f-4d16-b1fd-738cdd8c6a10\") " pod="openshift-ingress/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc"
Oct 30 07:25:58.231756 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:58.231738    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spknf\" (UniqueName: \"kubernetes.io/projected/50bc25e6-c53f-4d16-b1fd-738cdd8c6a10-kube-api-access-spknf\") pod \"e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc\" (UID: \"50bc25e6-c53f-4d16-b1fd-738cdd8c6a10\") " pod="openshift-ingress/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc"
Oct 30 07:25:58.232012 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:58.231986    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50bc25e6-c53f-4d16-b1fd-738cdd8c6a10-istiod-ca-cert\") pod \"e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc\" (UID: \"50bc25e6-c53f-4d16-b1fd-738cdd8c6a10\") " pod="openshift-ingress/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc"
Oct 30 07:25:58.232201 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:58.232183    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50bc25e6-c53f-4d16-b1fd-738cdd8c6a10-istio-envoy\") pod \"e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc\" (UID: \"50bc25e6-c53f-4d16-b1fd-738cdd8c6a10\") " pod="openshift-ingress/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc"
Oct 30 07:25:58.233016 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:58.232996    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50bc25e6-c53f-4d16-b1fd-738cdd8c6a10-istio-podinfo\") pod \"e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc\" (UID: \"50bc25e6-c53f-4d16-b1fd-738cdd8c6a10\") " pod="openshift-ingress/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc"
Oct 30 07:25:58.233233 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:58.233215    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50bc25e6-c53f-4d16-b1fd-738cdd8c6a10-workload-socket\") pod \"e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc\" (UID: \"50bc25e6-c53f-4d16-b1fd-738cdd8c6a10\") " pod="openshift-ingress/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc"
Oct 30 07:25:58.233356 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:58.233329    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50bc25e6-c53f-4d16-b1fd-738cdd8c6a10-istiod-ca-cert\") pod \"e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc\" (UID: \"50bc25e6-c53f-4d16-b1fd-738cdd8c6a10\") " pod="openshift-ingress/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc"
Oct 30 07:25:58.233420 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:58.233343    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50bc25e6-c53f-4d16-b1fd-738cdd8c6a10-credential-socket\") pod \"e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc\" (UID: \"50bc25e6-c53f-4d16-b1fd-738cdd8c6a10\") " pod="openshift-ingress/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc"
Oct 30 07:25:58.233492 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:58.231118    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50bc25e6-c53f-4d16-b1fd-738cdd8c6a10-istio-data\") pod \"e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc\" (UID: \"50bc25e6-c53f-4d16-b1fd-738cdd8c6a10\") " pod="openshift-ingress/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc"
Oct 30 07:25:58.233550 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:58.232569    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50bc25e6-c53f-4d16-b1fd-738cdd8c6a10-workload-certs\") pod \"e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc\" (UID: \"50bc25e6-c53f-4d16-b1fd-738cdd8c6a10\") " pod="openshift-ingress/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc"
Oct 30 07:25:58.233912 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:58.233890    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50bc25e6-c53f-4d16-b1fd-738cdd8c6a10-credential-socket\") pod \"e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc\" (UID: \"50bc25e6-c53f-4d16-b1fd-738cdd8c6a10\") " pod="openshift-ingress/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc"
Oct 30 07:25:58.234221 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:58.234197    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50bc25e6-c53f-4d16-b1fd-738cdd8c6a10-workload-socket\") pod \"e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc\" (UID: \"50bc25e6-c53f-4d16-b1fd-738cdd8c6a10\") " pod="openshift-ingress/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc"
Oct 30 07:25:58.249855 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:58.247402    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50bc25e6-c53f-4d16-b1fd-738cdd8c6a10-istio-podinfo\") pod \"e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc\" (UID: \"50bc25e6-c53f-4d16-b1fd-738cdd8c6a10\") " pod="openshift-ingress/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc"
Oct 30 07:25:58.249855 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:58.247606    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50bc25e6-c53f-4d16-b1fd-738cdd8c6a10-istio-envoy\") pod \"e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc\" (UID: \"50bc25e6-c53f-4d16-b1fd-738cdd8c6a10\") " pod="openshift-ingress/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc"
Oct 30 07:25:58.260641 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:58.260606    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spknf\" (UniqueName: \"kubernetes.io/projected/50bc25e6-c53f-4d16-b1fd-738cdd8c6a10-kube-api-access-spknf\") pod \"e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc\" (UID: \"50bc25e6-c53f-4d16-b1fd-738cdd8c6a10\") " pod="openshift-ingress/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc"
Oct 30 07:25:58.262487 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:58.262459    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50bc25e6-c53f-4d16-b1fd-738cdd8c6a10-istio-token\") pod \"e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc\" (UID: \"50bc25e6-c53f-4d16-b1fd-738cdd8c6a10\") " pod="openshift-ingress/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc"
Oct 30 07:25:58.341280 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:58.340112    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45" event={"ID":"04947652-67c7-45de-a2dc-5c2856bfdad8","Type":"ContainerStarted","Data":"343928d02c00e86c5d9e05408d492bca5b3d5a6897e4ce066a6f675d3c14e862"}
Oct 30 07:25:58.341280 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:58.340182    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45" event={"ID":"04947652-67c7-45de-a2dc-5c2856bfdad8","Type":"ContainerStarted","Data":"82ba5abeec7ebcc9d1df96f20909ea3a73edb40e85ecd63cc4857b6a4601453a"}
Oct 30 07:25:58.388562 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:58.388521    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc"
Oct 30 07:25:58.391973 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:58.391918    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45" podStartSLOduration=1.391902035 podStartE2EDuration="1.391902035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:25:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:25:58.3894319 +0000 UTC m=+1716.998838869" watchObservedRunningTime="2026-10-30 07:25:58.391902035 +0000 UTC m=+1717.001309006"
Oct 30 07:25:58.495919 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:58.495596    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45"
Oct 30 07:25:58.495919 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:58.495733    6224 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 30 07:25:58.495919 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:58.495777    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45" podUID="04947652-67c7-45de-a2dc-5c2856bfdad8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 30 07:25:58.790468 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:58.788743    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc"]
Oct 30 07:25:58.796347 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:25:58.796305    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50bc25e6_c53f_4d16_b1fd_738cdd8c6a10.slice/crio-e7414fe7f209e309195d94257281958b9dda7d1d861f1788b7eddb0870148c7c WatchSource:0}: Error finding container e7414fe7f209e309195d94257281958b9dda7d1d861f1788b7eddb0870148c7c: Status 404 returned error can't find the container with id e7414fe7f209e309195d94257281958b9dda7d1d861f1788b7eddb0870148c7c
Oct 30 07:25:58.799531 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:58.799494    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:25:58.799640 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:58.799566    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:25:58.799640 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:58.799605    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:25:59.348723 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:59.348673    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc" event={"ID":"50bc25e6-c53f-4d16-b1fd-738cdd8c6a10","Type":"ContainerStarted","Data":"30703ecef0c1c81dea8fc3b3d293f83ed29e7952b657931054ebd57c0632aeb3"}
Oct 30 07:25:59.349177 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:59.348749    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc" event={"ID":"50bc25e6-c53f-4d16-b1fd-738cdd8c6a10","Type":"ContainerStarted","Data":"e7414fe7f209e309195d94257281958b9dda7d1d861f1788b7eddb0870148c7c"}
Oct 30 07:25:59.388918 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:59.388881    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc"
Oct 30 07:25:59.413798 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:59.413733    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc" podStartSLOduration=1.413713016 podStartE2EDuration="1.413713016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:25:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:25:59.412165048 +0000 UTC m=+1718.021572079" watchObservedRunningTime="2026-10-30 07:25:59.413713016 +0000 UTC m=+1718.023119986"
Oct 30 07:25:59.504833 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:59.500933    6224 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 30 07:25:59.504833 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:25:59.501866    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45" podUID="04947652-67c7-45de-a2dc-5c2856bfdad8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 30 07:26:00.395230 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:00.395201    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc"
Oct 30 07:26:00.488325 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:00.488289    6224 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 30 07:26:00.488502 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:00.488365    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45" podUID="04947652-67c7-45de-a2dc-5c2856bfdad8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 30 07:26:01.366832 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:01.363802    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc"
Oct 30 07:26:01.388047 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:01.388017    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc"
Oct 30 07:26:01.494831 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:01.487168    6224 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 30 07:26:01.494831 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:01.487229    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45" podUID="04947652-67c7-45de-a2dc-5c2856bfdad8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 30 07:26:01.862751 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:26:01.862701    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:26:01.916906 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:01.916872    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5"]
Oct 30 07:26:01.921577 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:01.921545    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5"]
Oct 30 07:26:01.921748 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:01.921626    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5"
Oct 30 07:26:02.021146 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:02.021108    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3290bdb2-873c-4502-bb84-e8205fabbedf-istio-data\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5\" (UID: \"3290bdb2-873c-4502-bb84-e8205fabbedf\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5"
Oct 30 07:26:02.021146 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:02.021152    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3290bdb2-873c-4502-bb84-e8205fabbedf-istio-token\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5\" (UID: \"3290bdb2-873c-4502-bb84-e8205fabbedf\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5"
Oct 30 07:26:02.021357 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:02.021271    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3290bdb2-873c-4502-bb84-e8205fabbedf-istiod-ca-cert\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5\" (UID: \"3290bdb2-873c-4502-bb84-e8205fabbedf\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5"
Oct 30 07:26:02.021357 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:02.021315    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3290bdb2-873c-4502-bb84-e8205fabbedf-workload-certs\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5\" (UID: \"3290bdb2-873c-4502-bb84-e8205fabbedf\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5"
Oct 30 07:26:02.021471 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:02.021449    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3290bdb2-873c-4502-bb84-e8205fabbedf-istio-envoy\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5\" (UID: \"3290bdb2-873c-4502-bb84-e8205fabbedf\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5"
Oct 30 07:26:02.021507 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:02.021480    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkcd4\" (UniqueName: \"kubernetes.io/projected/3290bdb2-873c-4502-bb84-e8205fabbedf-kube-api-access-hkcd4\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5\" (UID: \"3290bdb2-873c-4502-bb84-e8205fabbedf\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5"
Oct 30 07:26:02.021553 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:02.021509    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3290bdb2-873c-4502-bb84-e8205fabbedf-workload-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5\" (UID: \"3290bdb2-873c-4502-bb84-e8205fabbedf\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5"
Oct 30 07:26:02.021639 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:02.021613    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3290bdb2-873c-4502-bb84-e8205fabbedf-istio-podinfo\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5\" (UID: \"3290bdb2-873c-4502-bb84-e8205fabbedf\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5"
Oct 30 07:26:02.021727 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:02.021712    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3290bdb2-873c-4502-bb84-e8205fabbedf-credential-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5\" (UID: \"3290bdb2-873c-4502-bb84-e8205fabbedf\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5"
Oct 30 07:26:02.128949 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:02.128684    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3290bdb2-873c-4502-bb84-e8205fabbedf-workload-certs\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5\" (UID: \"3290bdb2-873c-4502-bb84-e8205fabbedf\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5"
Oct 30 07:26:02.128949 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:02.128886    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3290bdb2-873c-4502-bb84-e8205fabbedf-istio-envoy\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5\" (UID: \"3290bdb2-873c-4502-bb84-e8205fabbedf\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5"
Oct 30 07:26:02.128949 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:02.128922    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkcd4\" (UniqueName: \"kubernetes.io/projected/3290bdb2-873c-4502-bb84-e8205fabbedf-kube-api-access-hkcd4\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5\" (UID: \"3290bdb2-873c-4502-bb84-e8205fabbedf\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5"
Oct 30 07:26:02.129273 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:02.128967    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3290bdb2-873c-4502-bb84-e8205fabbedf-workload-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5\" (UID: \"3290bdb2-873c-4502-bb84-e8205fabbedf\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5"
Oct 30 07:26:02.129273 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:02.128999    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3290bdb2-873c-4502-bb84-e8205fabbedf-istio-podinfo\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5\" (UID: \"3290bdb2-873c-4502-bb84-e8205fabbedf\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5"
Oct 30 07:26:02.129273 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:02.129058    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3290bdb2-873c-4502-bb84-e8205fabbedf-credential-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5\" (UID: \"3290bdb2-873c-4502-bb84-e8205fabbedf\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5"
Oct 30 07:26:02.129273 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:02.129177    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3290bdb2-873c-4502-bb84-e8205fabbedf-istio-data\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5\" (UID: \"3290bdb2-873c-4502-bb84-e8205fabbedf\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5"
Oct 30 07:26:02.129273 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:02.129205    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3290bdb2-873c-4502-bb84-e8205fabbedf-istio-token\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5\" (UID: \"3290bdb2-873c-4502-bb84-e8205fabbedf\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5"
Oct 30 07:26:02.129549 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:02.129293    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3290bdb2-873c-4502-bb84-e8205fabbedf-istiod-ca-cert\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5\" (UID: \"3290bdb2-873c-4502-bb84-e8205fabbedf\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5"
Oct 30 07:26:02.132828 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:02.130186    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3290bdb2-873c-4502-bb84-e8205fabbedf-istiod-ca-cert\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5\" (UID: \"3290bdb2-873c-4502-bb84-e8205fabbedf\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5"
Oct 30 07:26:02.132828 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:02.130489    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3290bdb2-873c-4502-bb84-e8205fabbedf-workload-certs\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5\" (UID: \"3290bdb2-873c-4502-bb84-e8205fabbedf\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5"
Oct 30 07:26:02.132828 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:02.132186    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3290bdb2-873c-4502-bb84-e8205fabbedf-workload-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5\" (UID: \"3290bdb2-873c-4502-bb84-e8205fabbedf\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5"
Oct 30 07:26:02.132828 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:02.132480    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3290bdb2-873c-4502-bb84-e8205fabbedf-istio-data\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5\" (UID: \"3290bdb2-873c-4502-bb84-e8205fabbedf\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5"
Oct 30 07:26:02.132828 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:02.132623    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3290bdb2-873c-4502-bb84-e8205fabbedf-credential-socket\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5\" (UID: \"3290bdb2-873c-4502-bb84-e8205fabbedf\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5"
Oct 30 07:26:02.134896 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:02.134867    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3290bdb2-873c-4502-bb84-e8205fabbedf-istio-envoy\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5\" (UID: \"3290bdb2-873c-4502-bb84-e8205fabbedf\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5"
Oct 30 07:26:02.135051 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:02.135027    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3290bdb2-873c-4502-bb84-e8205fabbedf-istio-podinfo\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5\" (UID: \"3290bdb2-873c-4502-bb84-e8205fabbedf\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5"
Oct 30 07:26:02.160643 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:02.160607    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkcd4\" (UniqueName: \"kubernetes.io/projected/3290bdb2-873c-4502-bb84-e8205fabbedf-kube-api-access-hkcd4\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5\" (UID: \"3290bdb2-873c-4502-bb84-e8205fabbedf\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5"
Oct 30 07:26:02.183844 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:02.175312    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3290bdb2-873c-4502-bb84-e8205fabbedf-istio-token\") pod \"e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5\" (UID: \"3290bdb2-873c-4502-bb84-e8205fabbedf\") " pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5"
Oct 30 07:26:02.240933 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:02.240906    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5"
Oct 30 07:26:02.491924 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:02.487215    6224 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 30 07:26:02.491924 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:02.487280    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45" podUID="04947652-67c7-45de-a2dc-5c2856bfdad8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 30 07:26:02.601260 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:02.601161    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5"]
Oct 30 07:26:02.605399 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:02.605333    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:26:02.605510 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:02.605428    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:26:02.605510 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:02.605472    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:26:03.416780 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:03.416742    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5" event={"ID":"3290bdb2-873c-4502-bb84-e8205fabbedf","Type":"ContainerStarted","Data":"adb93b29a9d3570a89b11a5c032c2808a0a011118ac49905dc715d1d7d2c11e1"}
Oct 30 07:26:03.417014 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:03.416864    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5" event={"ID":"3290bdb2-873c-4502-bb84-e8205fabbedf","Type":"ContainerStarted","Data":"f048f97f854335f9aa1be896d5a41e05432486a34061af69b98e77b95190533b"}
Oct 30 07:26:03.482832 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:03.482536    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5" podStartSLOduration=2.482517316 podStartE2EDuration="2.482517316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:26:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:26:03.46704326 +0000 UTC m=+1722.076450232" watchObservedRunningTime="2026-10-30 07:26:03.482517316 +0000 UTC m=+1722.091924288"
Oct 30 07:26:03.490857 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:03.489231    6224 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 30 07:26:03.490857 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:03.489296    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45" podUID="04947652-67c7-45de-a2dc-5c2856bfdad8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 30 07:26:04.062331 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:04.062216    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-f5ceca-5f57ff8b46-28rhc"]
Oct 30 07:26:04.130907 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:04.130876    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-f5ceca-5f57ff8b46-28rhc"]
Oct 30 07:26:04.131204 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:04.131189    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f5ceca-5f57ff8b46-28rhc"
Oct 30 07:26:04.161035 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:04.161005    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-f5ceca\""
Oct 30 07:26:04.237168 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:04.237134    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-f5ceca-dockercfg-mtscz\""
Oct 30 07:26:04.241930 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:04.241856    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5"
Oct 30 07:26:04.242982 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:04.242959    6224 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Oct 30 07:26:04.243145 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:04.243125    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5" podUID="3290bdb2-873c-4502-bb84-e8205fabbedf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Oct 30 07:26:04.293761 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:04.293719    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2hwj\" (UniqueName: \"kubernetes.io/projected/671ab50b-8475-463c-8815-767e495feaa9-kube-api-access-x2hwj\") pod \"payload-processing-e2e-shared-b-w3-f5ceca-5f57ff8b46-28rhc\" (UID: \"671ab50b-8475-463c-8815-767e495feaa9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f5ceca-5f57ff8b46-28rhc"
Oct 30 07:26:04.294027 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:04.293994    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/671ab50b-8475-463c-8815-767e495feaa9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-f5ceca-5f57ff8b46-28rhc\" (UID: \"671ab50b-8475-463c-8815-767e495feaa9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f5ceca-5f57ff8b46-28rhc"
Oct 30 07:26:04.399132 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:04.398702    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2hwj\" (UniqueName: \"kubernetes.io/projected/671ab50b-8475-463c-8815-767e495feaa9-kube-api-access-x2hwj\") pod \"payload-processing-e2e-shared-b-w3-f5ceca-5f57ff8b46-28rhc\" (UID: \"671ab50b-8475-463c-8815-767e495feaa9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f5ceca-5f57ff8b46-28rhc"
Oct 30 07:26:04.399464 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:04.399436    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/671ab50b-8475-463c-8815-767e495feaa9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-f5ceca-5f57ff8b46-28rhc\" (UID: \"671ab50b-8475-463c-8815-767e495feaa9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f5ceca-5f57ff8b46-28rhc"
Oct 30 07:26:04.400291 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:04.400181    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/671ab50b-8475-463c-8815-767e495feaa9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-f5ceca-5f57ff8b46-28rhc\" (UID: \"671ab50b-8475-463c-8815-767e495feaa9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f5ceca-5f57ff8b46-28rhc"
Oct 30 07:26:04.428301 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:04.428266    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2hwj\" (UniqueName: \"kubernetes.io/projected/671ab50b-8475-463c-8815-767e495feaa9-kube-api-access-x2hwj\") pod \"payload-processing-e2e-shared-b-w3-f5ceca-5f57ff8b46-28rhc\" (UID: \"671ab50b-8475-463c-8815-767e495feaa9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f5ceca-5f57ff8b46-28rhc"
Oct 30 07:26:04.485831 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:04.484038    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f5ceca-5f57ff8b46-28rhc"
Oct 30 07:26:04.494355 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:04.490792    6224 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 30 07:26:04.494355 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:04.490872    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45" podUID="04947652-67c7-45de-a2dc-5c2856bfdad8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 30 07:26:04.817189 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:26:04.817157    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod671ab50b_8475_463c_8815_767e495feaa9.slice/crio-a282392afa187a06e3c8a27b05fa693a90b45b8a891c486587a8d73440f14086 WatchSource:0}: Error finding container a282392afa187a06e3c8a27b05fa693a90b45b8a891c486587a8d73440f14086: Status 404 returned error can't find the container with id a282392afa187a06e3c8a27b05fa693a90b45b8a891c486587a8d73440f14086
Oct 30 07:26:04.817329 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:04.817251    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-f5ceca-5f57ff8b46-28rhc"]
Oct 30 07:26:05.243901 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:05.242208    6224 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Oct 30 07:26:05.243901 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:05.242268    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5" podUID="3290bdb2-873c-4502-bb84-e8205fabbedf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Oct 30 07:26:05.517046 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:05.516821    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f5ceca-5f57ff8b46-28rhc" event={"ID":"671ab50b-8475-463c-8815-767e495feaa9","Type":"ContainerStarted","Data":"f7fa1c951f077cf238ea5cadcae2815dddf74de59b376504e87d545da522246e"}
Oct 30 07:26:05.517046 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:05.516881    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f5ceca-5f57ff8b46-28rhc" event={"ID":"671ab50b-8475-463c-8815-767e495feaa9","Type":"ContainerStarted","Data":"a282392afa187a06e3c8a27b05fa693a90b45b8a891c486587a8d73440f14086"}
Oct 30 07:26:05.517046 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:05.516950    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f5ceca-5f57ff8b46-28rhc"
Oct 30 07:26:05.570198 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:05.570138    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f5ceca-5f57ff8b46-28rhc" podStartSLOduration=2.570119148 podStartE2EDuration="2.570119148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:26:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:26:05.56796699 +0000 UTC m=+1724.177373961" watchObservedRunningTime="2026-10-30 07:26:05.570119148 +0000 UTC m=+1724.179526119"
Oct 30 07:26:06.076736 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:06.074187    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45"
Oct 30 07:26:06.076736 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:06.075933    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45"
Oct 30 07:26:06.081829 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:06.080062    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45"
Oct 30 07:26:06.243797 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:06.243760    6224 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Oct 30 07:26:06.244002 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:06.243887    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5" podUID="3290bdb2-873c-4502-bb84-e8205fabbedf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Oct 30 07:26:07.241624 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:07.241596    6224 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Oct 30 07:26:07.241797 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:07.241661    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5" podUID="3290bdb2-873c-4502-bb84-e8205fabbedf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Oct 30 07:26:07.715101 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:07.715062    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:26:08.241830 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:08.241777    6224 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Oct 30 07:26:08.241996 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:08.241857    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5" podUID="3290bdb2-873c-4502-bb84-e8205fabbedf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Oct 30 07:26:09.241703 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:09.241667    6224 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Oct 30 07:26:09.241703 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:09.241725    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5" podUID="3290bdb2-873c-4502-bb84-e8205fabbedf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Oct 30 07:26:10.456551 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:10.456365    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5"
Oct 30 07:26:10.457082 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:10.456757    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5"
Oct 30 07:26:10.457663 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:10.457637    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5"
Oct 30 07:26:13.455100 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:13.455066    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-06e19a-579597b4c-wlbjb"]
Oct 30 07:26:13.460128 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:13.460106    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-06e19a-579597b4c-wlbjb"
Oct 30 07:26:13.476299 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:13.476275    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-06e19a-579597b4c-wlbjb"]
Oct 30 07:26:13.486051 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:13.486017    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-06e19a\""
Oct 30 07:26:13.501551 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:13.501529    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-06e19a-dockercfg-5kdnz\""
Oct 30 07:26:13.529867 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:13.529837    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d82945b-c43f-413a-a2a4-941c9be966c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-06e19a-579597b4c-wlbjb\" (UID: \"7d82945b-c43f-413a-a2a4-941c9be966c4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-06e19a-579597b4c-wlbjb"
Oct 30 07:26:13.530008 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:13.529878    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt76t\" (UniqueName: \"kubernetes.io/projected/7d82945b-c43f-413a-a2a4-941c9be966c4-kube-api-access-lt76t\") pod \"payload-pre-processing-e2e-shared-b-w1-06e19a-579597b4c-wlbjb\" (UID: \"7d82945b-c43f-413a-a2a4-941c9be966c4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-06e19a-579597b4c-wlbjb"
Oct 30 07:26:13.645161 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:13.631672    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d82945b-c43f-413a-a2a4-941c9be966c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-06e19a-579597b4c-wlbjb\" (UID: \"7d82945b-c43f-413a-a2a4-941c9be966c4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-06e19a-579597b4c-wlbjb"
Oct 30 07:26:13.645161 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:13.631732    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt76t\" (UniqueName: \"kubernetes.io/projected/7d82945b-c43f-413a-a2a4-941c9be966c4-kube-api-access-lt76t\") pod \"payload-pre-processing-e2e-shared-b-w1-06e19a-579597b4c-wlbjb\" (UID: \"7d82945b-c43f-413a-a2a4-941c9be966c4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-06e19a-579597b4c-wlbjb"
Oct 30 07:26:13.645161 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:13.633473    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d82945b-c43f-413a-a2a4-941c9be966c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-06e19a-579597b4c-wlbjb\" (UID: \"7d82945b-c43f-413a-a2a4-941c9be966c4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-06e19a-579597b4c-wlbjb"
Oct 30 07:26:13.661420 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:13.661385    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt76t\" (UniqueName: \"kubernetes.io/projected/7d82945b-c43f-413a-a2a4-941c9be966c4-kube-api-access-lt76t\") pod \"payload-pre-processing-e2e-shared-b-w1-06e19a-579597b4c-wlbjb\" (UID: \"7d82945b-c43f-413a-a2a4-941c9be966c4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-06e19a-579597b4c-wlbjb"
Oct 30 07:26:13.775623 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:13.775541    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-06e19a-579597b4c-wlbjb"
Oct 30 07:26:14.100483 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:14.100455    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-06e19a-579597b4c-wlbjb"]
Oct 30 07:26:14.101138 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:26:14.101100    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d82945b_c43f_413a_a2a4_941c9be966c4.slice/crio-f9fdfa3d32bf77d53e7edc13bca1d909fa8ba505eca069e51efb0122e8107160 WatchSource:0}: Error finding container f9fdfa3d32bf77d53e7edc13bca1d909fa8ba505eca069e51efb0122e8107160: Status 404 returned error can't find the container with id f9fdfa3d32bf77d53e7edc13bca1d909fa8ba505eca069e51efb0122e8107160
Oct 30 07:26:14.593075 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:14.593040    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-06e19a-579597b4c-wlbjb" event={"ID":"7d82945b-c43f-413a-a2a4-941c9be966c4","Type":"ContainerStarted","Data":"f589568d6251293ab9f4a737e50212d59e2801afb5f1d338f830bbf49c7981eb"}
Oct 30 07:26:14.593508 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:14.593098    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-06e19a-579597b4c-wlbjb" event={"ID":"7d82945b-c43f-413a-a2a4-941c9be966c4","Type":"ContainerStarted","Data":"f9fdfa3d32bf77d53e7edc13bca1d909fa8ba505eca069e51efb0122e8107160"}
Oct 30 07:26:14.593508 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:14.593263    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-06e19a-579597b4c-wlbjb"
Oct 30 07:26:14.639482 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:14.639433    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-06e19a-579597b4c-wlbjb" podStartSLOduration=1.639419913 podStartE2EDuration="1.639419913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:26:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:26:14.636677052 +0000 UTC m=+1733.246084021" watchObservedRunningTime="2026-10-30 07:26:14.639419913 +0000 UTC m=+1733.248826884"
Oct 30 07:26:17.616059 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:17.616026    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f5ceca-5f57ff8b46-28rhc"
Oct 30 07:26:22.717379 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:22.717345    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-137122-7f8b9567cf-jmkdv"]
Oct 30 07:26:22.722856 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:22.722834    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-137122-7f8b9567cf-jmkdv"
Oct 30 07:26:22.742746 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:22.742720    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-137122-7f8b9567cf-jmkdv"]
Oct 30 07:26:22.751721 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:22.751696    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-137122\""
Oct 30 07:26:22.751877 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:22.751760    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-137122-dockercfg-gqcxw\""
Oct 30 07:26:22.913880 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:22.913849    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-137122-646879997d-8z24f"]
Oct 30 07:26:22.924301 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:22.924263    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b8c015e-9234-4ecb-a908-9ce7eb600db5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-137122-7f8b9567cf-jmkdv\" (UID: \"7b8c015e-9234-4ecb-a908-9ce7eb600db5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-137122-7f8b9567cf-jmkdv"
Oct 30 07:26:22.924695 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:22.924672    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7sqd\" (UniqueName: \"kubernetes.io/projected/7b8c015e-9234-4ecb-a908-9ce7eb600db5-kube-api-access-c7sqd\") pod \"payload-pre-processing-e2e-shared-b-w2-137122-7f8b9567cf-jmkdv\" (UID: \"7b8c015e-9234-4ecb-a908-9ce7eb600db5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-137122-7f8b9567cf-jmkdv"
Oct 30 07:26:22.944159 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:22.944121    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-137122-646879997d-8z24f"]
Oct 30 07:26:22.950926 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:22.944253    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-137122-646879997d-8z24f"
Oct 30 07:26:23.025846 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:23.025740    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b8c015e-9234-4ecb-a908-9ce7eb600db5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-137122-7f8b9567cf-jmkdv\" (UID: \"7b8c015e-9234-4ecb-a908-9ce7eb600db5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-137122-7f8b9567cf-jmkdv"
Oct 30 07:26:23.025846 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:23.025797    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9s5g\" (UniqueName: \"kubernetes.io/projected/30fa2cf6-c67b-4be5-ab71-df50207183c5-kube-api-access-p9s5g\") pod \"payload-processing-e2e-shared-b-w2-137122-646879997d-8z24f\" (UID: \"30fa2cf6-c67b-4be5-ab71-df50207183c5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-137122-646879997d-8z24f"
Oct 30 07:26:23.026062 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:23.025906    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7sqd\" (UniqueName: \"kubernetes.io/projected/7b8c015e-9234-4ecb-a908-9ce7eb600db5-kube-api-access-c7sqd\") pod \"payload-pre-processing-e2e-shared-b-w2-137122-7f8b9567cf-jmkdv\" (UID: \"7b8c015e-9234-4ecb-a908-9ce7eb600db5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-137122-7f8b9567cf-jmkdv"
Oct 30 07:26:23.026129 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:23.026075    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30fa2cf6-c67b-4be5-ab71-df50207183c5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-137122-646879997d-8z24f\" (UID: \"30fa2cf6-c67b-4be5-ab71-df50207183c5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-137122-646879997d-8z24f"
Oct 30 07:26:23.026419 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:23.026397    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b8c015e-9234-4ecb-a908-9ce7eb600db5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-137122-7f8b9567cf-jmkdv\" (UID: \"7b8c015e-9234-4ecb-a908-9ce7eb600db5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-137122-7f8b9567cf-jmkdv"
Oct 30 07:26:23.054539 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:23.054505    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7sqd\" (UniqueName: \"kubernetes.io/projected/7b8c015e-9234-4ecb-a908-9ce7eb600db5-kube-api-access-c7sqd\") pod \"payload-pre-processing-e2e-shared-b-w2-137122-7f8b9567cf-jmkdv\" (UID: \"7b8c015e-9234-4ecb-a908-9ce7eb600db5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-137122-7f8b9567cf-jmkdv"
Oct 30 07:26:23.127695 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:23.127655    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30fa2cf6-c67b-4be5-ab71-df50207183c5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-137122-646879997d-8z24f\" (UID: \"30fa2cf6-c67b-4be5-ab71-df50207183c5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-137122-646879997d-8z24f"
Oct 30 07:26:23.128190 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:23.128150    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9s5g\" (UniqueName: \"kubernetes.io/projected/30fa2cf6-c67b-4be5-ab71-df50207183c5-kube-api-access-p9s5g\") pod \"payload-processing-e2e-shared-b-w2-137122-646879997d-8z24f\" (UID: \"30fa2cf6-c67b-4be5-ab71-df50207183c5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-137122-646879997d-8z24f"
Oct 30 07:26:23.130084 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:23.130058    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30fa2cf6-c67b-4be5-ab71-df50207183c5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-137122-646879997d-8z24f\" (UID: \"30fa2cf6-c67b-4be5-ab71-df50207183c5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-137122-646879997d-8z24f"
Oct 30 07:26:23.159402 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:23.159367    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9s5g\" (UniqueName: \"kubernetes.io/projected/30fa2cf6-c67b-4be5-ab71-df50207183c5-kube-api-access-p9s5g\") pod \"payload-processing-e2e-shared-b-w2-137122-646879997d-8z24f\" (UID: \"30fa2cf6-c67b-4be5-ab71-df50207183c5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-137122-646879997d-8z24f"
Oct 30 07:26:23.278244 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:23.278169    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-137122-646879997d-8z24f"
Oct 30 07:26:23.343066 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:23.343032    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-137122-7f8b9567cf-jmkdv"
Oct 30 07:26:23.581825 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:26:23.581781    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30fa2cf6_c67b_4be5_ab71_df50207183c5.slice/crio-7476c4bc695c8d5839c17e53470d909778247993aa94f1177cf343127f648548 WatchSource:0}: Error finding container 7476c4bc695c8d5839c17e53470d909778247993aa94f1177cf343127f648548: Status 404 returned error can't find the container with id 7476c4bc695c8d5839c17e53470d909778247993aa94f1177cf343127f648548
Oct 30 07:26:23.582599 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:23.582578    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-137122-646879997d-8z24f"]
Oct 30 07:26:23.669581 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:23.669553    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-137122-646879997d-8z24f" event={"ID":"30fa2cf6-c67b-4be5-ab71-df50207183c5","Type":"ContainerStarted","Data":"97a9e8d097ab24b618858cb8dcf8492d636dcbba860450c0a260708a7596cfe4"}
Oct 30 07:26:23.669702 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:23.669606    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-137122-646879997d-8z24f" event={"ID":"30fa2cf6-c67b-4be5-ab71-df50207183c5","Type":"ContainerStarted","Data":"7476c4bc695c8d5839c17e53470d909778247993aa94f1177cf343127f648548"}
Oct 30 07:26:23.669702 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:23.669636    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-137122-646879997d-8z24f"
Oct 30 07:26:23.694574 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:26:23.694550    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b8c015e_9234_4ecb_a908_9ce7eb600db5.slice/crio-c44ef6663fafb44f9260275a1d3ad4636ff32e78eb481c5633c4248d48ac1972 WatchSource:0}: Error finding container c44ef6663fafb44f9260275a1d3ad4636ff32e78eb481c5633c4248d48ac1972: Status 404 returned error can't find the container with id c44ef6663fafb44f9260275a1d3ad4636ff32e78eb481c5633c4248d48ac1972
Oct 30 07:26:23.695332 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:23.695305    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-137122-7f8b9567cf-jmkdv"]
Oct 30 07:26:23.720611 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:23.720568    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-137122-646879997d-8z24f" podStartSLOduration=1.720555303 podStartE2EDuration="1.720555303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:26:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:26:23.717903297 +0000 UTC m=+1742.327310270" watchObservedRunningTime="2026-10-30 07:26:23.720555303 +0000 UTC m=+1742.329962284"
Oct 30 07:26:24.339655 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:24.339623    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs"]
Oct 30 07:26:24.344380 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:24.344355    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs"
Oct 30 07:26:24.362869 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:24.362842    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/454321e8-5bec-497e-9789-7ed6604ed72c-credential-socket\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs\" (UID: \"454321e8-5bec-497e-9789-7ed6604ed72c\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs"
Oct 30 07:26:24.363044 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:24.362882    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/454321e8-5bec-497e-9789-7ed6604ed72c-istio-data\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs\" (UID: \"454321e8-5bec-497e-9789-7ed6604ed72c\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs"
Oct 30 07:26:24.363044 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:24.362963    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/454321e8-5bec-497e-9789-7ed6604ed72c-istio-podinfo\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs\" (UID: \"454321e8-5bec-497e-9789-7ed6604ed72c\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs"
Oct 30 07:26:24.363044 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:24.363010    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/454321e8-5bec-497e-9789-7ed6604ed72c-workload-certs\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs\" (UID: \"454321e8-5bec-497e-9789-7ed6604ed72c\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs"
Oct 30 07:26:24.363207 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:24.363072    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/454321e8-5bec-497e-9789-7ed6604ed72c-workload-socket\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs\" (UID: \"454321e8-5bec-497e-9789-7ed6604ed72c\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs"
Oct 30 07:26:24.363207 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:24.363107    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/454321e8-5bec-497e-9789-7ed6604ed72c-istio-token\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs\" (UID: \"454321e8-5bec-497e-9789-7ed6604ed72c\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs"
Oct 30 07:26:24.363538 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:24.363258    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/454321e8-5bec-497e-9789-7ed6604ed72c-istiod-ca-cert\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs\" (UID: \"454321e8-5bec-497e-9789-7ed6604ed72c\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs"
Oct 30 07:26:24.363538 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:24.363322    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/454321e8-5bec-497e-9789-7ed6604ed72c-istio-envoy\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs\" (UID: \"454321e8-5bec-497e-9789-7ed6604ed72c\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs"
Oct 30 07:26:24.363538 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:24.363391    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwjg6\" (UniqueName: \"kubernetes.io/projected/454321e8-5bec-497e-9789-7ed6604ed72c-kube-api-access-rwjg6\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs\" (UID: \"454321e8-5bec-497e-9789-7ed6604ed72c\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs"
Oct 30 07:26:24.382866 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:24.380410    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs"]
Oct 30 07:26:24.440379 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:24.440348    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-265384-openshift-default-dockercfg-tcbq7\""
Oct 30 07:26:24.464439 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:24.464400    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/454321e8-5bec-497e-9789-7ed6604ed72c-istiod-ca-cert\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs\" (UID: \"454321e8-5bec-497e-9789-7ed6604ed72c\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs"
Oct 30 07:26:24.464661 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:24.464480    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/454321e8-5bec-497e-9789-7ed6604ed72c-istio-envoy\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs\" (UID: \"454321e8-5bec-497e-9789-7ed6604ed72c\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs"
Oct 30 07:26:24.464661 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:24.464556    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwjg6\" (UniqueName: \"kubernetes.io/projected/454321e8-5bec-497e-9789-7ed6604ed72c-kube-api-access-rwjg6\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs\" (UID: \"454321e8-5bec-497e-9789-7ed6604ed72c\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs"
Oct 30 07:26:24.464661 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:24.464612    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/454321e8-5bec-497e-9789-7ed6604ed72c-credential-socket\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs\" (UID: \"454321e8-5bec-497e-9789-7ed6604ed72c\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs"
Oct 30 07:26:24.464661 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:24.464651    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/454321e8-5bec-497e-9789-7ed6604ed72c-istio-data\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs\" (UID: \"454321e8-5bec-497e-9789-7ed6604ed72c\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs"
Oct 30 07:26:24.464941 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:24.464726    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/454321e8-5bec-497e-9789-7ed6604ed72c-istio-podinfo\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs\" (UID: \"454321e8-5bec-497e-9789-7ed6604ed72c\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs"
Oct 30 07:26:24.464941 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:24.464776    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/454321e8-5bec-497e-9789-7ed6604ed72c-workload-certs\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs\" (UID: \"454321e8-5bec-497e-9789-7ed6604ed72c\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs"
Oct 30 07:26:24.464941 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:24.464857    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/454321e8-5bec-497e-9789-7ed6604ed72c-workload-socket\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs\" (UID: \"454321e8-5bec-497e-9789-7ed6604ed72c\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs"
Oct 30 07:26:24.464941 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:24.464890    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/454321e8-5bec-497e-9789-7ed6604ed72c-istio-token\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs\" (UID: \"454321e8-5bec-497e-9789-7ed6604ed72c\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs"
Oct 30 07:26:24.465189 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:24.465168    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/454321e8-5bec-497e-9789-7ed6604ed72c-istiod-ca-cert\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs\" (UID: \"454321e8-5bec-497e-9789-7ed6604ed72c\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs"
Oct 30 07:26:24.465417 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:24.465398    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/454321e8-5bec-497e-9789-7ed6604ed72c-istio-data\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs\" (UID: \"454321e8-5bec-497e-9789-7ed6604ed72c\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs"
Oct 30 07:26:24.465979 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:24.465948    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/454321e8-5bec-497e-9789-7ed6604ed72c-workload-certs\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs\" (UID: \"454321e8-5bec-497e-9789-7ed6604ed72c\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs"
Oct 30 07:26:24.466137 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:24.466120    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/454321e8-5bec-497e-9789-7ed6604ed72c-credential-socket\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs\" (UID: \"454321e8-5bec-497e-9789-7ed6604ed72c\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs"
Oct 30 07:26:24.466340 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:24.466324    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/454321e8-5bec-497e-9789-7ed6604ed72c-workload-socket\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs\" (UID: \"454321e8-5bec-497e-9789-7ed6604ed72c\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs"
Oct 30 07:26:24.468016 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:24.467983    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/454321e8-5bec-497e-9789-7ed6604ed72c-istio-envoy\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs\" (UID: \"454321e8-5bec-497e-9789-7ed6604ed72c\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs"
Oct 30 07:26:24.468641 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:24.468614    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/454321e8-5bec-497e-9789-7ed6604ed72c-istio-podinfo\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs\" (UID: \"454321e8-5bec-497e-9789-7ed6604ed72c\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs"
Oct 30 07:26:24.490411 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:24.490387    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/454321e8-5bec-497e-9789-7ed6604ed72c-istio-token\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs\" (UID: \"454321e8-5bec-497e-9789-7ed6604ed72c\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs"
Oct 30 07:26:24.490746 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:24.490724    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwjg6\" (UniqueName: \"kubernetes.io/projected/454321e8-5bec-497e-9789-7ed6604ed72c-kube-api-access-rwjg6\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs\" (UID: \"454321e8-5bec-497e-9789-7ed6604ed72c\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs"
Oct 30 07:26:24.671556 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:24.671480    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs"
Oct 30 07:26:24.677446 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:24.677416    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-137122-7f8b9567cf-jmkdv" event={"ID":"7b8c015e-9234-4ecb-a908-9ce7eb600db5","Type":"ContainerStarted","Data":"1c6d9ff5eb3c83e6257554d95af42851649952b47d29f1b18af3286bcedef025"}
Oct 30 07:26:24.677580 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:24.677474    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-137122-7f8b9567cf-jmkdv" event={"ID":"7b8c015e-9234-4ecb-a908-9ce7eb600db5","Type":"ContainerStarted","Data":"c44ef6663fafb44f9260275a1d3ad4636ff32e78eb481c5633c4248d48ac1972"}
Oct 30 07:26:24.677580 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:24.677504    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-137122-7f8b9567cf-jmkdv"
Oct 30 07:26:24.729180 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:24.729114    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-137122-7f8b9567cf-jmkdv" podStartSLOduration=2.72909811 podStartE2EDuration="2.72909811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:26:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:26:24.723904359 +0000 UTC m=+1743.333311345" watchObservedRunningTime="2026-10-30 07:26:24.72909811 +0000 UTC m=+1743.338505080"
Oct 30 07:26:24.992908 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:26:24.992871    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod454321e8_5bec_497e_9789_7ed6604ed72c.slice/crio-af37ff448b9c70fae180276f7d982c22420e433b1da9f1bc0695f3ac5f342d69 WatchSource:0}: Error finding container af37ff448b9c70fae180276f7d982c22420e433b1da9f1bc0695f3ac5f342d69: Status 404 returned error can't find the container with id af37ff448b9c70fae180276f7d982c22420e433b1da9f1bc0695f3ac5f342d69
Oct 30 07:26:24.994304 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:24.994281    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs"]
Oct 30 07:26:24.995629 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:24.995598    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:26:24.995708 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:24.995658    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:26:24.995708 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:24.995683    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:26:25.602778 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:25.602749    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-06e19a-579597b4c-wlbjb"
Oct 30 07:26:25.689389 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:25.689218    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs" event={"ID":"454321e8-5bec-497e-9789-7ed6604ed72c","Type":"ContainerStarted","Data":"b9556ab6882d2a50c45cc2cb15b9772a45b75b6ab3042e6e5d1029600136791d"}
Oct 30 07:26:25.689389 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:25.689325    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs" event={"ID":"454321e8-5bec-497e-9789-7ed6604ed72c","Type":"ContainerStarted","Data":"af37ff448b9c70fae180276f7d982c22420e433b1da9f1bc0695f3ac5f342d69"}
Oct 30 07:26:25.746465 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:25.746390    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs" podStartSLOduration=1.7463665590000002 podStartE2EDuration="1.746366559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:26:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:26:25.742869768 +0000 UTC m=+1744.352276754" watchObservedRunningTime="2026-10-30 07:26:25.746366559 +0000 UTC m=+1744.355773531"
Oct 30 07:26:26.684511 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:26.684475    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs"
Oct 30 07:26:26.689007 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:26.687908    6224 patch_prober.go:29] interesting pod/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Oct 30 07:26:26.689007 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:26.687969    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs" podUID="454321e8-5bec-497e-9789-7ed6604ed72c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Oct 30 07:26:26.695410 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:26.695386    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-04aede/autores-model-04aede-kserve-5547c4789c-d5v66"]
Oct 30 07:26:26.706396 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:26.706376    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-04aede/autores-model-04aede-kserve-5547c4789c-d5v66"
Oct 30 07:26:26.722755 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:26.722727    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-04aede/autores-model-04aede-kserve-5547c4789c-d5v66"]
Oct 30 07:26:26.728934 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:26.728900    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-04aede\"/\"autores-model-04aede-kserve-self-signed-certs\""
Oct 30 07:26:26.729051 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:26.728916    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-04aede\"/\"openshift-service-ca.crt\""
Oct 30 07:26:26.739127 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:26.739106    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-04aede\"/\"default-dockercfg-2g4r5\""
Oct 30 07:26:26.739233 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:26.739161    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-04aede\"/\"kube-root-ca.crt\""
Oct 30 07:26:26.812393 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:26.812354    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe-dshm\") pod \"autores-model-04aede-kserve-5547c4789c-d5v66\" (UID: \"7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe\") " pod="ai-tenant-e2e-autores-04aede/autores-model-04aede-kserve-5547c4789c-d5v66"
Oct 30 07:26:26.812800 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:26.812473    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe-model-cache\") pod \"autores-model-04aede-kserve-5547c4789c-d5v66\" (UID: \"7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe\") " pod="ai-tenant-e2e-autores-04aede/autores-model-04aede-kserve-5547c4789c-d5v66"
Oct 30 07:26:26.812800 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:26.812521    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggh4c\" (UniqueName: \"kubernetes.io/projected/7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe-kube-api-access-ggh4c\") pod \"autores-model-04aede-kserve-5547c4789c-d5v66\" (UID: \"7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe\") " pod="ai-tenant-e2e-autores-04aede/autores-model-04aede-kserve-5547c4789c-d5v66"
Oct 30 07:26:26.812800 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:26.812708    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe-tls-certs\") pod \"autores-model-04aede-kserve-5547c4789c-d5v66\" (UID: \"7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe\") " pod="ai-tenant-e2e-autores-04aede/autores-model-04aede-kserve-5547c4789c-d5v66"
Oct 30 07:26:26.813077 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:26.812853    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe-home\") pod \"autores-model-04aede-kserve-5547c4789c-d5v66\" (UID: \"7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe\") " pod="ai-tenant-e2e-autores-04aede/autores-model-04aede-kserve-5547c4789c-d5v66"
Oct 30 07:26:26.813077 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:26.812929    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe-tmp-dir\") pod \"autores-model-04aede-kserve-5547c4789c-d5v66\" (UID: \"7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe\") " pod="ai-tenant-e2e-autores-04aede/autores-model-04aede-kserve-5547c4789c-d5v66"
Oct 30 07:26:26.914968 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:26.914933    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe-tls-certs\") pod \"autores-model-04aede-kserve-5547c4789c-d5v66\" (UID: \"7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe\") " pod="ai-tenant-e2e-autores-04aede/autores-model-04aede-kserve-5547c4789c-d5v66"
Oct 30 07:26:26.915173 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:26.915013    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe-home\") pod \"autores-model-04aede-kserve-5547c4789c-d5v66\" (UID: \"7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe\") " pod="ai-tenant-e2e-autores-04aede/autores-model-04aede-kserve-5547c4789c-d5v66"
Oct 30 07:26:26.915173 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:26.915072    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe-tmp-dir\") pod \"autores-model-04aede-kserve-5547c4789c-d5v66\" (UID: \"7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe\") " pod="ai-tenant-e2e-autores-04aede/autores-model-04aede-kserve-5547c4789c-d5v66"
Oct 30 07:26:26.915297 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:26.915206    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe-dshm\") pod \"autores-model-04aede-kserve-5547c4789c-d5v66\" (UID: \"7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe\") " pod="ai-tenant-e2e-autores-04aede/autores-model-04aede-kserve-5547c4789c-d5v66"
Oct 30 07:26:26.915297 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:26.915262    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe-model-cache\") pod \"autores-model-04aede-kserve-5547c4789c-d5v66\" (UID: \"7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe\") " pod="ai-tenant-e2e-autores-04aede/autores-model-04aede-kserve-5547c4789c-d5v66"
Oct 30 07:26:26.915396 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:26.915307    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggh4c\" (UniqueName: \"kubernetes.io/projected/7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe-kube-api-access-ggh4c\") pod \"autores-model-04aede-kserve-5547c4789c-d5v66\" (UID: \"7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe\") " pod="ai-tenant-e2e-autores-04aede/autores-model-04aede-kserve-5547c4789c-d5v66"
Oct 30 07:26:26.915660 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:26.915639    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe-tmp-dir\") pod \"autores-model-04aede-kserve-5547c4789c-d5v66\" (UID: \"7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe\") " pod="ai-tenant-e2e-autores-04aede/autores-model-04aede-kserve-5547c4789c-d5v66"
Oct 30 07:26:26.915765 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:26.915700    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe-model-cache\") pod \"autores-model-04aede-kserve-5547c4789c-d5v66\" (UID: \"7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe\") " pod="ai-tenant-e2e-autores-04aede/autores-model-04aede-kserve-5547c4789c-d5v66"
Oct 30 07:26:26.915850 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:26.915782    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe-home\") pod \"autores-model-04aede-kserve-5547c4789c-d5v66\" (UID: \"7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe\") " pod="ai-tenant-e2e-autores-04aede/autores-model-04aede-kserve-5547c4789c-d5v66"
Oct 30 07:26:26.917868 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:26.917837    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe-dshm\") pod \"autores-model-04aede-kserve-5547c4789c-d5v66\" (UID: \"7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe\") " pod="ai-tenant-e2e-autores-04aede/autores-model-04aede-kserve-5547c4789c-d5v66"
Oct 30 07:26:26.918414 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:26.918392    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe-tls-certs\") pod \"autores-model-04aede-kserve-5547c4789c-d5v66\" (UID: \"7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe\") " pod="ai-tenant-e2e-autores-04aede/autores-model-04aede-kserve-5547c4789c-d5v66"
Oct 30 07:26:26.940355 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:26.940294    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggh4c\" (UniqueName: \"kubernetes.io/projected/7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe-kube-api-access-ggh4c\") pod \"autores-model-04aede-kserve-5547c4789c-d5v66\" (UID: \"7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe\") " pod="ai-tenant-e2e-autores-04aede/autores-model-04aede-kserve-5547c4789c-d5v66"
Oct 30 07:26:27.170831 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:27.166273    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-04aede/autores-model-04aede-kserve-5547c4789c-d5v66"
Oct 30 07:26:27.593893 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:27.593858    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-04aede/autores-model-04aede-kserve-5547c4789c-d5v66"]
Oct 30 07:26:27.597915 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:26:27.597882    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c7cda87_d4ae_45d4_b56b_5c71e8c32dbe.slice/crio-12e84b7608d8b4cedc8073fc8935f6cb2e3f845d66c2c1b9794cfe6491a081a4 WatchSource:0}: Error finding container 12e84b7608d8b4cedc8073fc8935f6cb2e3f845d66c2c1b9794cfe6491a081a4: Status 404 returned error can't find the container with id 12e84b7608d8b4cedc8073fc8935f6cb2e3f845d66c2c1b9794cfe6491a081a4
Oct 30 07:26:27.688265 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:27.688236    6224 patch_prober.go:29] interesting pod/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Oct 30 07:26:27.688410 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:27.688290    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs" podUID="454321e8-5bec-497e-9789-7ed6604ed72c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Oct 30 07:26:27.713015 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:27.712976    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-04aede/autores-model-04aede-kserve-5547c4789c-d5v66" event={"ID":"7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe","Type":"ContainerStarted","Data":"c7cfd2c1cb5c8083267c6e9c7793bb353a6551411f090d3427cad8f27dc7fe6c"}
Oct 30 07:26:27.713225 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:27.713025    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-04aede/autores-model-04aede-kserve-5547c4789c-d5v66" event={"ID":"7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe","Type":"ContainerStarted","Data":"12e84b7608d8b4cedc8073fc8935f6cb2e3f845d66c2c1b9794cfe6491a081a4"}
Oct 30 07:26:27.768838 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:27.764026    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-autores-04aede/autores-model-04aede-kserve-5547c4789c-d5v66" podStartSLOduration=1.764003846 podStartE2EDuration="1.764003846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:26:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:26:27.762313319 +0000 UTC m=+1746.371720290" watchObservedRunningTime="2026-10-30 07:26:27.764003846 +0000 UTC m=+1746.373410818"
Oct 30 07:26:28.169938 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:28.169896    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-04aede/autores-model-04aede-kserve-5547c4789c-d5v66"
Oct 30 07:26:28.673548 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:28.673136    6224 patch_prober.go:29] interesting pod/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Oct 30 07:26:28.673548 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:28.673196    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs" podUID="454321e8-5bec-497e-9789-7ed6604ed72c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Oct 30 07:26:29.685067 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:29.685026    6224 patch_prober.go:29] interesting pod/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Oct 30 07:26:29.686108 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:29.685661    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs" podUID="454321e8-5bec-497e-9789-7ed6604ed72c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Oct 30 07:26:30.676831 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:30.672179    6224 patch_prober.go:29] interesting pod/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Oct 30 07:26:30.676831 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:30.672256    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs" podUID="454321e8-5bec-497e-9789-7ed6604ed72c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Oct 30 07:26:31.677287 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:31.677255    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs"
Oct 30 07:26:31.677742 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:31.677723    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs"
Oct 30 07:26:31.678587 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:31.678565    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs"
Oct 30 07:26:31.867239 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:26:31.867174    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:26:34.680742 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:34.680707    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-137122-646879997d-8z24f"
Oct 30 07:26:35.693470 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:35.693444    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-137122-7f8b9567cf-jmkdv"
Oct 30 07:26:37.167471 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:37.167439    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-04aede/autores-model-04aede-kserve-5547c4789c-d5v66"
Oct 30 07:26:37.180168 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:37.180145    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-04aede/autores-model-04aede-kserve-5547c4789c-d5v66"
Oct 30 07:26:37.804719 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:37.804688    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-04aede/autores-model-04aede-kserve-5547c4789c-d5v66"
Oct 30 07:26:37.939565 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:37.939536    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-b5998f-7bff7846d-254sj"]
Oct 30 07:26:37.945192 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:37.945167    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-b5998f-7bff7846d-254sj"
Oct 30 07:26:37.962840 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:37.962793    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-b5998f-7bff7846d-254sj"]
Oct 30 07:26:37.964248 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:37.964228    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-b5998f\""
Oct 30 07:26:37.965036 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:37.965018    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-b5998f-dockercfg-bt4n5\""
Oct 30 07:26:38.041867 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:38.028762    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ccf1d22-7e51-4883-86b0-3309d4ffc899-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-b5998f-7bff7846d-254sj\" (UID: \"1ccf1d22-7e51-4883-86b0-3309d4ffc899\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-b5998f-7bff7846d-254sj"
Oct 30 07:26:38.042363 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:38.042331    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5khl8\" (UniqueName: \"kubernetes.io/projected/1ccf1d22-7e51-4883-86b0-3309d4ffc899-kube-api-access-5khl8\") pod \"payload-processing-e2e-shared-b-w0-b5998f-7bff7846d-254sj\" (UID: \"1ccf1d22-7e51-4883-86b0-3309d4ffc899\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-b5998f-7bff7846d-254sj"
Oct 30 07:26:38.145006 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:38.144914    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ccf1d22-7e51-4883-86b0-3309d4ffc899-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-b5998f-7bff7846d-254sj\" (UID: \"1ccf1d22-7e51-4883-86b0-3309d4ffc899\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-b5998f-7bff7846d-254sj"
Oct 30 07:26:38.145170 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:38.145098    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5khl8\" (UniqueName: \"kubernetes.io/projected/1ccf1d22-7e51-4883-86b0-3309d4ffc899-kube-api-access-5khl8\") pod \"payload-processing-e2e-shared-b-w0-b5998f-7bff7846d-254sj\" (UID: \"1ccf1d22-7e51-4883-86b0-3309d4ffc899\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-b5998f-7bff7846d-254sj"
Oct 30 07:26:38.146002 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:38.145974    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ccf1d22-7e51-4883-86b0-3309d4ffc899-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-b5998f-7bff7846d-254sj\" (UID: \"1ccf1d22-7e51-4883-86b0-3309d4ffc899\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-b5998f-7bff7846d-254sj"
Oct 30 07:26:38.170535 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:38.170506    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5khl8\" (UniqueName: \"kubernetes.io/projected/1ccf1d22-7e51-4883-86b0-3309d4ffc899-kube-api-access-5khl8\") pod \"payload-processing-e2e-shared-b-w0-b5998f-7bff7846d-254sj\" (UID: \"1ccf1d22-7e51-4883-86b0-3309d4ffc899\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-b5998f-7bff7846d-254sj"
Oct 30 07:26:38.265038 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:38.264965    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-b5998f-7bff7846d-254sj"
Oct 30 07:26:38.579565 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:26:38.579525    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ccf1d22_7e51_4883_86b0_3309d4ffc899.slice/crio-5af74ec41d8fbad291d4d1f99c99f0d9b62770060afeacd108ba5d3c13015fb7 WatchSource:0}: Error finding container 5af74ec41d8fbad291d4d1f99c99f0d9b62770060afeacd108ba5d3c13015fb7: Status 404 returned error can't find the container with id 5af74ec41d8fbad291d4d1f99c99f0d9b62770060afeacd108ba5d3c13015fb7
Oct 30 07:26:38.581444 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:38.581417    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-b5998f-7bff7846d-254sj"]
Oct 30 07:26:38.802771 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:38.802732    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-b5998f-7bff7846d-254sj" event={"ID":"1ccf1d22-7e51-4883-86b0-3309d4ffc899","Type":"ContainerStarted","Data":"dcfacb6809f42e60f210e518aa6646b95cc467799fec9e0fa5145d4523d52981"}
Oct 30 07:26:38.802994 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:38.802794    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-b5998f-7bff7846d-254sj" event={"ID":"1ccf1d22-7e51-4883-86b0-3309d4ffc899","Type":"ContainerStarted","Data":"5af74ec41d8fbad291d4d1f99c99f0d9b62770060afeacd108ba5d3c13015fb7"}
Oct 30 07:26:38.802994 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:38.802976    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-b5998f-7bff7846d-254sj"
Oct 30 07:26:38.889904 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:38.889817    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-b5998f-7bff7846d-254sj" podStartSLOduration=1.889789358 podStartE2EDuration="1.889789358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:26:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:26:38.885461325 +0000 UTC m=+1757.494868295" watchObservedRunningTime="2026-10-30 07:26:38.889789358 +0000 UTC m=+1757.499196330"
Oct 30 07:26:49.098101 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:49.098053    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:26:49.813677 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:49.813648    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-b5998f-7bff7846d-254sj"
Oct 30 07:26:51.342829 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:51.341023    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-04aede/autores-model-04aede-kserve-5547c4789c-d5v66"]
Oct 30 07:26:54.405488 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:54.405455    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg"]
Oct 30 07:26:54.410893 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:54.410871    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg"
Oct 30 07:26:54.436487 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:54.436463    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg"]
Oct 30 07:26:54.517452 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:54.517411    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c257ea9-cd39-419d-8960-43069d3eaecc-istiod-ca-cert\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg\" (UID: \"3c257ea9-cd39-419d-8960-43069d3eaecc\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg"
Oct 30 07:26:54.517643 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:54.517483    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c257ea9-cd39-419d-8960-43069d3eaecc-credential-socket\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg\" (UID: \"3c257ea9-cd39-419d-8960-43069d3eaecc\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg"
Oct 30 07:26:54.517643 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:54.517554    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c257ea9-cd39-419d-8960-43069d3eaecc-istio-envoy\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg\" (UID: \"3c257ea9-cd39-419d-8960-43069d3eaecc\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg"
Oct 30 07:26:54.517643 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:54.517591    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c257ea9-cd39-419d-8960-43069d3eaecc-istio-podinfo\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg\" (UID: \"3c257ea9-cd39-419d-8960-43069d3eaecc\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg"
Oct 30 07:26:54.517643 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:54.517626    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c257ea9-cd39-419d-8960-43069d3eaecc-workload-socket\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg\" (UID: \"3c257ea9-cd39-419d-8960-43069d3eaecc\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg"
Oct 30 07:26:54.517902 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:54.517775    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c257ea9-cd39-419d-8960-43069d3eaecc-istio-token\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg\" (UID: \"3c257ea9-cd39-419d-8960-43069d3eaecc\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg"
Oct 30 07:26:54.517960 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:54.517897    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c257ea9-cd39-419d-8960-43069d3eaecc-workload-certs\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg\" (UID: \"3c257ea9-cd39-419d-8960-43069d3eaecc\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg"
Oct 30 07:26:54.518021 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:54.517979    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsjlk\" (UniqueName: \"kubernetes.io/projected/3c257ea9-cd39-419d-8960-43069d3eaecc-kube-api-access-dsjlk\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg\" (UID: \"3c257ea9-cd39-419d-8960-43069d3eaecc\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg"
Oct 30 07:26:54.518127 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:54.518108    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c257ea9-cd39-419d-8960-43069d3eaecc-istio-data\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg\" (UID: \"3c257ea9-cd39-419d-8960-43069d3eaecc\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg"
Oct 30 07:26:54.619334 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:54.619297    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c257ea9-cd39-419d-8960-43069d3eaecc-workload-certs\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg\" (UID: \"3c257ea9-cd39-419d-8960-43069d3eaecc\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg"
Oct 30 07:26:54.619517 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:54.619381    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsjlk\" (UniqueName: \"kubernetes.io/projected/3c257ea9-cd39-419d-8960-43069d3eaecc-kube-api-access-dsjlk\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg\" (UID: \"3c257ea9-cd39-419d-8960-43069d3eaecc\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg"
Oct 30 07:26:54.619579 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:54.619536    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c257ea9-cd39-419d-8960-43069d3eaecc-istio-data\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg\" (UID: \"3c257ea9-cd39-419d-8960-43069d3eaecc\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg"
Oct 30 07:26:54.619642 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:54.619609    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c257ea9-cd39-419d-8960-43069d3eaecc-istiod-ca-cert\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg\" (UID: \"3c257ea9-cd39-419d-8960-43069d3eaecc\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg"
Oct 30 07:26:54.619693 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:54.619659    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c257ea9-cd39-419d-8960-43069d3eaecc-credential-socket\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg\" (UID: \"3c257ea9-cd39-419d-8960-43069d3eaecc\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg"
Oct 30 07:26:54.619752 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:54.619724    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c257ea9-cd39-419d-8960-43069d3eaecc-istio-envoy\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg\" (UID: \"3c257ea9-cd39-419d-8960-43069d3eaecc\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg"
Oct 30 07:26:54.619752 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:54.619735    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c257ea9-cd39-419d-8960-43069d3eaecc-workload-certs\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg\" (UID: \"3c257ea9-cd39-419d-8960-43069d3eaecc\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg"
Oct 30 07:26:54.619876 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:54.619777    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c257ea9-cd39-419d-8960-43069d3eaecc-istio-podinfo\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg\" (UID: \"3c257ea9-cd39-419d-8960-43069d3eaecc\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg"
Oct 30 07:26:54.619876 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:54.619832    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c257ea9-cd39-419d-8960-43069d3eaecc-workload-socket\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg\" (UID: \"3c257ea9-cd39-419d-8960-43069d3eaecc\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg"
Oct 30 07:26:54.619979 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:54.619895    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c257ea9-cd39-419d-8960-43069d3eaecc-istio-token\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg\" (UID: \"3c257ea9-cd39-419d-8960-43069d3eaecc\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg"
Oct 30 07:26:54.620056 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:54.620019    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c257ea9-cd39-419d-8960-43069d3eaecc-istio-data\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg\" (UID: \"3c257ea9-cd39-419d-8960-43069d3eaecc\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg"
Oct 30 07:26:54.620259 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:54.620230    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c257ea9-cd39-419d-8960-43069d3eaecc-workload-socket\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg\" (UID: \"3c257ea9-cd39-419d-8960-43069d3eaecc\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg"
Oct 30 07:26:54.620324 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:54.620286    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c257ea9-cd39-419d-8960-43069d3eaecc-credential-socket\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg\" (UID: \"3c257ea9-cd39-419d-8960-43069d3eaecc\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg"
Oct 30 07:26:54.620395 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:54.620379    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c257ea9-cd39-419d-8960-43069d3eaecc-istiod-ca-cert\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg\" (UID: \"3c257ea9-cd39-419d-8960-43069d3eaecc\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg"
Oct 30 07:26:54.622665 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:54.622629    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c257ea9-cd39-419d-8960-43069d3eaecc-istio-envoy\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg\" (UID: \"3c257ea9-cd39-419d-8960-43069d3eaecc\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg"
Oct 30 07:26:54.622850 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:54.622786    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c257ea9-cd39-419d-8960-43069d3eaecc-istio-podinfo\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg\" (UID: \"3c257ea9-cd39-419d-8960-43069d3eaecc\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg"
Oct 30 07:26:54.645296 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:54.645270    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c257ea9-cd39-419d-8960-43069d3eaecc-istio-token\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg\" (UID: \"3c257ea9-cd39-419d-8960-43069d3eaecc\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg"
Oct 30 07:26:54.645418 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:54.645381    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsjlk\" (UniqueName: \"kubernetes.io/projected/3c257ea9-cd39-419d-8960-43069d3eaecc-kube-api-access-dsjlk\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg\" (UID: \"3c257ea9-cd39-419d-8960-43069d3eaecc\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg"
Oct 30 07:26:54.726985 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:54.726909    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg"
Oct 30 07:26:55.092428 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:26:55.092395    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c257ea9_cd39_419d_8960_43069d3eaecc.slice/crio-af44eae60be019f8527384d1932d28752af473ca7492799b61acd8d763b1e968 WatchSource:0}: Error finding container af44eae60be019f8527384d1932d28752af473ca7492799b61acd8d763b1e968: Status 404 returned error can't find the container with id af44eae60be019f8527384d1932d28752af473ca7492799b61acd8d763b1e968
Oct 30 07:26:55.093545 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:55.093524    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg"]
Oct 30 07:26:55.095340 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:55.095307    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:26:55.095427 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:55.095372    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:26:55.095427 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:55.095403    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:26:55.936797 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:55.936759    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg" event={"ID":"3c257ea9-cd39-419d-8960-43069d3eaecc","Type":"ContainerStarted","Data":"c1d89ca8a7acf24ba597ebbfe9247c1d752ac7a493d0a9339d94b2f630bd49ce"}
Oct 30 07:26:55.937223 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:55.936851    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg" event={"ID":"3c257ea9-cd39-419d-8960-43069d3eaecc","Type":"ContainerStarted","Data":"af44eae60be019f8527384d1932d28752af473ca7492799b61acd8d763b1e968"}
Oct 30 07:26:56.021677 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:56.021624    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg" podStartSLOduration=2.021608614 podStartE2EDuration="2.021608614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:26:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:26:56.006929508 +0000 UTC m=+1774.616336479" watchObservedRunningTime="2026-10-30 07:26:56.021608614 +0000 UTC m=+1774.631015583"
Oct 30 07:26:56.727509 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:56.727464    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg"
Oct 30 07:26:56.728921 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:56.728894    6224 patch_prober.go:29] interesting pod/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Oct 30 07:26:56.729069 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:56.728937    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg" podUID="3c257ea9-cd39-419d-8960-43069d3eaecc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Oct 30 07:26:57.728351 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:57.728285    6224 patch_prober.go:29] interesting pod/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Oct 30 07:26:57.728935 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:57.728383    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg" podUID="3c257ea9-cd39-419d-8960-43069d3eaecc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Oct 30 07:26:58.727797 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:58.727758    6224 patch_prober.go:29] interesting pod/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Oct 30 07:26:58.728026 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:58.727829    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg" podUID="3c257ea9-cd39-419d-8960-43069d3eaecc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Oct 30 07:26:59.727556 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:59.727513    6224 patch_prober.go:29] interesting pod/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Oct 30 07:26:59.728083 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:26:59.727570    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg" podUID="3c257ea9-cd39-419d-8960-43069d3eaecc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Oct 30 07:27:00.727611 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:00.727570    6224 patch_prober.go:29] interesting pod/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Oct 30 07:27:00.728040 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:00.727640    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg" podUID="3c257ea9-cd39-419d-8960-43069d3eaecc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Oct 30 07:27:01.365085 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:01.365052    6224 patch_prober.go:29] interesting pod/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Oct 30 07:27:01.365314 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:01.365116    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc" podUID="50bc25e6-c53f-4d16-b1fd-738cdd8c6a10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Oct 30 07:27:01.731850 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:01.731754    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg"
Oct 30 07:27:01.732312 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:01.732199    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg"
Oct 30 07:27:01.733062 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:01.733036    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg"
Oct 30 07:27:01.885559 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:27:01.885519    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:27:02.009117 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:02.009041    6224 generic.go:356] "Generic (PLEG): container finished" podID="50bc25e6-c53f-4d16-b1fd-738cdd8c6a10" containerID="30703ecef0c1c81dea8fc3b3d293f83ed29e7952b657931054ebd57c0632aeb3" exitCode=0
Oct 30 07:27:02.009274 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:02.009114    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc" event={"ID":"50bc25e6-c53f-4d16-b1fd-738cdd8c6a10","Type":"ContainerDied","Data":"30703ecef0c1c81dea8fc3b3d293f83ed29e7952b657931054ebd57c0632aeb3"}
Oct 30 07:27:02.009707 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:02.009692    6224 scope.go:122] "RemoveContainer" containerID="30703ecef0c1c81dea8fc3b3d293f83ed29e7952b657931054ebd57c0632aeb3"
Oct 30 07:27:02.011206 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:02.011182    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:27:02.011279 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:02.011256    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:27:02.011316 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:02.011289    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:27:02.291908 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:02.291875    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-04aede-5cd4c88478-kbvnt"]
Oct 30 07:27:02.358105 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:02.358068    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-04aede-776cc889f-47jhw"]
Oct 30 07:27:03.063829 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:03.051268    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc" event={"ID":"50bc25e6-c53f-4d16-b1fd-738cdd8c6a10","Type":"ContainerStarted","Data":"f8a5146a122aefa7f66c946baa4d92150b599ed06fd877587ef627bb17b887cc"}
Oct 30 07:27:03.389888 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:03.389762    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc"
Oct 30 07:27:03.390076 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:03.389930    6224 patch_prober.go:29] interesting pod/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Oct 30 07:27:03.390076 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:03.389983    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc" podUID="50bc25e6-c53f-4d16-b1fd-738cdd8c6a10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Oct 30 07:27:04.216953 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:04.216910    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b5998f-57685c68cb-xts98"]
Oct 30 07:27:04.221409 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:04.221385    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b5998f-57685c68cb-xts98"
Oct 30 07:27:04.242132 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:04.242108    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b5998f-57685c68cb-xts98"]
Oct 30 07:27:04.252122 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:04.252092    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-b5998f-serving-cert\""
Oct 30 07:27:04.296334 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:04.296298    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqfft\" (UniqueName: \"kubernetes.io/projected/afe4bac2-032c-41ce-90a8-34d3619eebb1-kube-api-access-xqfft\") pod \"maas-api-e2e-shared-b-w0-b5998f-57685c68cb-xts98\" (UID: \"afe4bac2-032c-41ce-90a8-34d3619eebb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b5998f-57685c68cb-xts98"
Oct 30 07:27:04.296594 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:04.296573    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afe4bac2-032c-41ce-90a8-34d3619eebb1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-b5998f-57685c68cb-xts98\" (UID: \"afe4bac2-032c-41ce-90a8-34d3619eebb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b5998f-57685c68cb-xts98"
Oct 30 07:27:04.296776 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:04.296758    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afe4bac2-032c-41ce-90a8-34d3619eebb1-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-b5998f-57685c68cb-xts98\" (UID: \"afe4bac2-032c-41ce-90a8-34d3619eebb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b5998f-57685c68cb-xts98"
Oct 30 07:27:04.296994 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:04.296966    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afe4bac2-032c-41ce-90a8-34d3619eebb1-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-b5998f-57685c68cb-xts98\" (UID: \"afe4bac2-032c-41ce-90a8-34d3619eebb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b5998f-57685c68cb-xts98"
Oct 30 07:27:04.389372 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:04.389233    6224 patch_prober.go:29] interesting pod/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Oct 30 07:27:04.389372 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:04.389297    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc" podUID="50bc25e6-c53f-4d16-b1fd-738cdd8c6a10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Oct 30 07:27:04.398683 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:04.398655    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afe4bac2-032c-41ce-90a8-34d3619eebb1-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-b5998f-57685c68cb-xts98\" (UID: \"afe4bac2-032c-41ce-90a8-34d3619eebb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b5998f-57685c68cb-xts98"
Oct 30 07:27:04.398890 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:04.398872    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqfft\" (UniqueName: \"kubernetes.io/projected/afe4bac2-032c-41ce-90a8-34d3619eebb1-kube-api-access-xqfft\") pod \"maas-api-e2e-shared-b-w0-b5998f-57685c68cb-xts98\" (UID: \"afe4bac2-032c-41ce-90a8-34d3619eebb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b5998f-57685c68cb-xts98"
Oct 30 07:27:04.398966 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:04.398916    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afe4bac2-032c-41ce-90a8-34d3619eebb1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-b5998f-57685c68cb-xts98\" (UID: \"afe4bac2-032c-41ce-90a8-34d3619eebb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b5998f-57685c68cb-xts98"
Oct 30 07:27:04.398966 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:27:04.398920    6224 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b5998f-serving-cert: secret "maas-api-e2e-shared-b-w0-b5998f-serving-cert" not found
Oct 30 07:27:04.398966 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:04.398942    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afe4bac2-032c-41ce-90a8-34d3619eebb1-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-b5998f-57685c68cb-xts98\" (UID: \"afe4bac2-032c-41ce-90a8-34d3619eebb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b5998f-57685c68cb-xts98"
Oct 30 07:27:04.399135 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:27:04.398979    6224 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afe4bac2-032c-41ce-90a8-34d3619eebb1-maas-api-tls podName:afe4bac2-032c-41ce-90a8-34d3619eebb1 nodeName:}" failed. No retries permitted until 2026-10-30 07:27:04.898960029 +0000 UTC m=+1783.508366994 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/afe4bac2-032c-41ce-90a8-34d3619eebb1-maas-api-tls") pod "maas-api-e2e-shared-b-w0-b5998f-57685c68cb-xts98" (UID: "afe4bac2-032c-41ce-90a8-34d3619eebb1") : secret "maas-api-e2e-shared-b-w0-b5998f-serving-cert" not found
Oct 30 07:27:04.399576 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:04.399556    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afe4bac2-032c-41ce-90a8-34d3619eebb1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-b5998f-57685c68cb-xts98\" (UID: \"afe4bac2-032c-41ce-90a8-34d3619eebb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b5998f-57685c68cb-xts98"
Oct 30 07:27:04.401294 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:04.401272    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afe4bac2-032c-41ce-90a8-34d3619eebb1-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-b5998f-57685c68cb-xts98\" (UID: \"afe4bac2-032c-41ce-90a8-34d3619eebb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b5998f-57685c68cb-xts98"
Oct 30 07:27:04.426082 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:04.426040    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqfft\" (UniqueName: \"kubernetes.io/projected/afe4bac2-032c-41ce-90a8-34d3619eebb1-kube-api-access-xqfft\") pod \"maas-api-e2e-shared-b-w0-b5998f-57685c68cb-xts98\" (UID: \"afe4bac2-032c-41ce-90a8-34d3619eebb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b5998f-57685c68cb-xts98"
Oct 30 07:27:04.907702 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:04.907667    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afe4bac2-032c-41ce-90a8-34d3619eebb1-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-b5998f-57685c68cb-xts98\" (UID: \"afe4bac2-032c-41ce-90a8-34d3619eebb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b5998f-57685c68cb-xts98"
Oct 30 07:27:04.910582 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:04.910554    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afe4bac2-032c-41ce-90a8-34d3619eebb1-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-b5998f-57685c68cb-xts98\" (UID: \"afe4bac2-032c-41ce-90a8-34d3619eebb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b5998f-57685c68cb-xts98"
Oct 30 07:27:05.135526 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:05.135498    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b5998f-57685c68cb-xts98"
Oct 30 07:27:05.449829 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:05.449158    6224 patch_prober.go:29] interesting pod/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Oct 30 07:27:05.449829 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:05.449218    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc" podUID="50bc25e6-c53f-4d16-b1fd-738cdd8c6a10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Oct 30 07:27:05.581832 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:05.565593    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b5998f-57685c68cb-xts98"]
Oct 30 07:27:05.621067 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:27:05.621031    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafe4bac2_032c_41ce_90a8_34d3619eebb1.slice/crio-4c09f7e3483e8c6599238713a635d5ace0188bb48b142793c8bb137eb7039907 WatchSource:0}: Error finding container 4c09f7e3483e8c6599238713a635d5ace0188bb48b142793c8bb137eb7039907: Status 404 returned error can't find the container with id 4c09f7e3483e8c6599238713a635d5ace0188bb48b142793c8bb137eb7039907
Oct 30 07:27:06.097248 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:06.097219    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b5998f-57685c68cb-xts98" event={"ID":"afe4bac2-032c-41ce-90a8-34d3619eebb1","Type":"ContainerStarted","Data":"4c09f7e3483e8c6599238713a635d5ace0188bb48b142793c8bb137eb7039907"}
Oct 30 07:27:06.367996 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:06.367951    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-autores-04aede/autores-model-04aede-kserve-5547c4789c-d5v66" podUID="7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe" containerName="main" containerID="cri-o://c7cfd2c1cb5c8083267c6e9c7793bb353a6551411f090d3427cad8f27dc7fe6c" gracePeriod=45
Oct 30 07:27:06.397827 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:06.393434    6224 patch_prober.go:29] interesting pod/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Oct 30 07:27:06.397827 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:06.393506    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc" podUID="50bc25e6-c53f-4d16-b1fd-738cdd8c6a10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Oct 30 07:27:06.793792 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:06.793759    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-04aede/autores-model-04aede-kserve-5547c4789c-d5v66"
Oct 30 07:27:06.993917 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:06.971005    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe-dshm\") pod \"7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe\" (UID: \"7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe\") "
Oct 30 07:27:06.993917 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:06.971086    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe-kube-api-access-ggh4c\" (UniqueName: \"kubernetes.io/projected/7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe-kube-api-access-ggh4c\") pod \"7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe\" (UID: \"7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe\") "
Oct 30 07:27:06.993917 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:06.971284    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe-tls-certs\" (UniqueName: \"kubernetes.io/secret/7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe-tls-certs\") pod \"7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe\" (UID: \"7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe\") "
Oct 30 07:27:06.993917 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:06.971323    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe-model-cache\") pod \"7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe\" (UID: \"7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe\") "
Oct 30 07:27:06.993917 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:06.971355    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe-home\" (UniqueName: \"kubernetes.io/empty-dir/7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe-home\") pod \"7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe\" (UID: \"7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe\") "
Oct 30 07:27:06.993917 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:06.971510    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe-tmp-dir\") pod \"7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe\" (UID: \"7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe\") "
Oct 30 07:27:06.993917 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:06.990303    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe-tmp-dir" pod "7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe" (UID: "7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:27:06.993917 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:06.990555    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe-model-cache" pod "7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe" (UID: "7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:27:06.993917 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:06.990732    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe-home" pod "7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe" (UID: "7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:27:06.993917 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:06.993453    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe-dshm" pod "7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe" (UID: "7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:27:06.995337 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:06.995303    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe-tls-certs" pod "7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe" (UID: "7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:27:06.996068 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:06.995980    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe-kube-api-access-ggh4c" pod "7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe" (UID: "7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe"). InnerVolumeSpecName "kube-api-access-ggh4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:27:07.078831 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:07.075936    6224 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe-dshm\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:27:07.078831 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:07.075972    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggh4c\" (UniqueName: \"kubernetes.io/projected/7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe-kube-api-access-ggh4c\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:27:07.078831 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:07.075989    6224 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe-tls-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:27:07.078831 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:07.076005    6224 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe-model-cache\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:27:07.078831 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:07.076017    6224 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe-home\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:27:07.078831 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:07.076029    6224 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe-tmp-dir\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:27:07.109954 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:07.109921    6224 generic.go:356] "Generic (PLEG): container finished" podID="7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe" containerID="c7cfd2c1cb5c8083267c6e9c7793bb353a6551411f090d3427cad8f27dc7fe6c" exitCode=0
Oct 30 07:27:07.110093 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:07.109967    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-04aede/autores-model-04aede-kserve-5547c4789c-d5v66"
Oct 30 07:27:07.110093 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:07.110011    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-04aede/autores-model-04aede-kserve-5547c4789c-d5v66" event={"ID":"7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe","Type":"ContainerDied","Data":"c7cfd2c1cb5c8083267c6e9c7793bb353a6551411f090d3427cad8f27dc7fe6c"}
Oct 30 07:27:07.110093 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:07.110065    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-04aede/autores-model-04aede-kserve-5547c4789c-d5v66" event={"ID":"7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe","Type":"ContainerDied","Data":"12e84b7608d8b4cedc8073fc8935f6cb2e3f845d66c2c1b9794cfe6491a081a4"}
Oct 30 07:27:07.110093 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:07.110089    6224 scope.go:122] "RemoveContainer" containerID="c7cfd2c1cb5c8083267c6e9c7793bb353a6551411f090d3427cad8f27dc7fe6c"
Oct 30 07:27:07.112634 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:07.112528    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b5998f-57685c68cb-xts98" event={"ID":"afe4bac2-032c-41ce-90a8-34d3619eebb1","Type":"ContainerStarted","Data":"0bdfc61a316f1e327e490ae86907ede7a04590ad0870e18d50e2c8bccc4281a6"}
Oct 30 07:27:07.112634 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:07.112601    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b5998f-57685c68cb-xts98"
Oct 30 07:27:07.121515 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:07.121500    6224 scope.go:122] "RemoveContainer" containerID="c7cfd2c1cb5c8083267c6e9c7793bb353a6551411f090d3427cad8f27dc7fe6c"
Oct 30 07:27:07.121747 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:27:07.121726    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7cfd2c1cb5c8083267c6e9c7793bb353a6551411f090d3427cad8f27dc7fe6c\": container with ID starting with c7cfd2c1cb5c8083267c6e9c7793bb353a6551411f090d3427cad8f27dc7fe6c not found: ID does not exist" containerID="c7cfd2c1cb5c8083267c6e9c7793bb353a6551411f090d3427cad8f27dc7fe6c"
Oct 30 07:27:07.121801 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:07.121773    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7cfd2c1cb5c8083267c6e9c7793bb353a6551411f090d3427cad8f27dc7fe6c"} err="failed to get container status \"c7cfd2c1cb5c8083267c6e9c7793bb353a6551411f090d3427cad8f27dc7fe6c\": rpc error: code = NotFound desc = could not find container \"c7cfd2c1cb5c8083267c6e9c7793bb353a6551411f090d3427cad8f27dc7fe6c\": container with ID starting with c7cfd2c1cb5c8083267c6e9c7793bb353a6551411f090d3427cad8f27dc7fe6c not found: ID does not exist"
Oct 30 07:27:07.190934 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:07.190888    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-04aede/autores-model-04aede-kserve-5547c4789c-d5v66"]
Oct 30 07:27:07.194639 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:07.194600    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-04aede/autores-model-04aede-kserve-5547c4789c-d5v66"]
Oct 30 07:27:07.232029 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:07.231980    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b5998f-57685c68cb-xts98" podStartSLOduration=2.83875784 podStartE2EDuration="3.231965655s" totalImagesPullingTime="393.207815ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:27:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:27:07.22803814 +0000 UTC m=+1785.837445110" watchObservedRunningTime="2026-10-30 07:27:07.231965655 +0000 UTC m=+1785.841372659"
Oct 30 07:27:07.292547 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:07.292515    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-04aede-5cd4c88478-kbvnt" podUID="bf4f18e1-ad1c-4315-9c07-d756d6d70476" containerName="payload-processing" containerID="cri-o://390329941ea55222838eb235936ba87674826be3cf0a03aa5e901ad9b6034968" gracePeriod=25
Oct 30 07:27:07.359385 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:07.359351    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-04aede-776cc889f-47jhw" podUID="c23e777f-2b9b-45fb-8110-62912dcbbaa6" containerName="payload-pre-processing" containerID="cri-o://8d20287cb61e3868c7478b45b7d527b6067091566f9a07d7d2725b8fbc99c50e" gracePeriod=25
Oct 30 07:27:07.389702 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:07.389264    6224 patch_prober.go:29] interesting pod/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Oct 30 07:27:07.389702 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:07.389326    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc" podUID="50bc25e6-c53f-4d16-b1fd-738cdd8c6a10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Oct 30 07:27:07.506626 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:07.506594    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe" path="/var/lib/kubelet/pods/7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe/volumes"
Oct 30 07:27:07.723769 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:07.723735    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-04aede-5cd4c88478-kbvnt"
Oct 30 07:27:07.820903 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:07.816634    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-04aede-776cc889f-47jhw"
Oct 30 07:27:07.907832 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:07.907335    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf4f18e1-ad1c-4315-9c07-d756d6d70476-kube-api-access-d2grh\" (UniqueName: \"kubernetes.io/projected/bf4f18e1-ad1c-4315-9c07-d756d6d70476-kube-api-access-d2grh\") pod \"bf4f18e1-ad1c-4315-9c07-d756d6d70476\" (UID: \"bf4f18e1-ad1c-4315-9c07-d756d6d70476\") "
Oct 30 07:27:07.907832 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:07.907529    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf4f18e1-ad1c-4315-9c07-d756d6d70476-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf4f18e1-ad1c-4315-9c07-d756d6d70476-plugins-config-volume\") pod \"bf4f18e1-ad1c-4315-9c07-d756d6d70476\" (UID: \"bf4f18e1-ad1c-4315-9c07-d756d6d70476\") "
Oct 30 07:27:07.909875 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:07.908947    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf4f18e1-ad1c-4315-9c07-d756d6d70476-plugins-config-volume" pod "bf4f18e1-ad1c-4315-9c07-d756d6d70476" (UID: "bf4f18e1-ad1c-4315-9c07-d756d6d70476"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:27:07.944634 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:07.944590    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf4f18e1-ad1c-4315-9c07-d756d6d70476-kube-api-access-d2grh" pod "bf4f18e1-ad1c-4315-9c07-d756d6d70476" (UID: "bf4f18e1-ad1c-4315-9c07-d756d6d70476"). InnerVolumeSpecName "kube-api-access-d2grh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:27:08.009003 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:08.008968    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c23e777f-2b9b-45fb-8110-62912dcbbaa6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c23e777f-2b9b-45fb-8110-62912dcbbaa6-plugins-config-volume\") pod \"c23e777f-2b9b-45fb-8110-62912dcbbaa6\" (UID: \"c23e777f-2b9b-45fb-8110-62912dcbbaa6\") "
Oct 30 07:27:08.009293 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:08.009269    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c23e777f-2b9b-45fb-8110-62912dcbbaa6-kube-api-access-qdfxw\" (UniqueName: \"kubernetes.io/projected/c23e777f-2b9b-45fb-8110-62912dcbbaa6-kube-api-access-qdfxw\") pod \"c23e777f-2b9b-45fb-8110-62912dcbbaa6\" (UID: \"c23e777f-2b9b-45fb-8110-62912dcbbaa6\") "
Oct 30 07:27:08.009419 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:08.009395    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c23e777f-2b9b-45fb-8110-62912dcbbaa6-plugins-config-volume" pod "c23e777f-2b9b-45fb-8110-62912dcbbaa6" (UID: "c23e777f-2b9b-45fb-8110-62912dcbbaa6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:27:08.009845 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:08.009824    6224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c23e777f-2b9b-45fb-8110-62912dcbbaa6-plugins-config-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:27:08.009845 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:08.009844    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2grh\" (UniqueName: \"kubernetes.io/projected/bf4f18e1-ad1c-4315-9c07-d756d6d70476-kube-api-access-d2grh\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:27:08.009971 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:08.009854    6224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf4f18e1-ad1c-4315-9c07-d756d6d70476-plugins-config-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:27:08.011841 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:08.011779    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c23e777f-2b9b-45fb-8110-62912dcbbaa6-kube-api-access-qdfxw" pod "c23e777f-2b9b-45fb-8110-62912dcbbaa6" (UID: "c23e777f-2b9b-45fb-8110-62912dcbbaa6"). InnerVolumeSpecName "kube-api-access-qdfxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:27:08.110969 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:08.110889    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdfxw\" (UniqueName: \"kubernetes.io/projected/c23e777f-2b9b-45fb-8110-62912dcbbaa6-kube-api-access-qdfxw\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:27:08.122689 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:08.122658    6224 generic.go:356] "Generic (PLEG): container finished" podID="c23e777f-2b9b-45fb-8110-62912dcbbaa6" containerID="8d20287cb61e3868c7478b45b7d527b6067091566f9a07d7d2725b8fbc99c50e" exitCode=0
Oct 30 07:27:08.122956 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:08.122710    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-04aede-776cc889f-47jhw"
Oct 30 07:27:08.123037 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:08.122725    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-04aede-776cc889f-47jhw" event={"ID":"c23e777f-2b9b-45fb-8110-62912dcbbaa6","Type":"ContainerDied","Data":"8d20287cb61e3868c7478b45b7d527b6067091566f9a07d7d2725b8fbc99c50e"}
Oct 30 07:27:08.123202 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:08.123183    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-04aede-776cc889f-47jhw" event={"ID":"c23e777f-2b9b-45fb-8110-62912dcbbaa6","Type":"ContainerDied","Data":"e6d7bfaebf65798ce017d133c83d3eae7fb6c0178a19056d15dd8759abb3a24d"}
Oct 30 07:27:08.123327 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:08.123314    6224 scope.go:122] "RemoveContainer" containerID="8d20287cb61e3868c7478b45b7d527b6067091566f9a07d7d2725b8fbc99c50e"
Oct 30 07:27:08.124637 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:08.124616    6224 generic.go:356] "Generic (PLEG): container finished" podID="bf4f18e1-ad1c-4315-9c07-d756d6d70476" containerID="390329941ea55222838eb235936ba87674826be3cf0a03aa5e901ad9b6034968" exitCode=0
Oct 30 07:27:08.124780 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:08.124676    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-04aede-5cd4c88478-kbvnt"
Oct 30 07:27:08.124875 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:08.124693    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-04aede-5cd4c88478-kbvnt" event={"ID":"bf4f18e1-ad1c-4315-9c07-d756d6d70476","Type":"ContainerDied","Data":"390329941ea55222838eb235936ba87674826be3cf0a03aa5e901ad9b6034968"}
Oct 30 07:27:08.124875 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:08.124863    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-04aede-5cd4c88478-kbvnt" event={"ID":"bf4f18e1-ad1c-4315-9c07-d756d6d70476","Type":"ContainerDied","Data":"a7741dead3d13aaafc67c7aa1f760bfb678e708e550f806b8e13500e5d7150eb"}
Oct 30 07:27:08.135445 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:08.135059    6224 scope.go:122] "RemoveContainer" containerID="8d20287cb61e3868c7478b45b7d527b6067091566f9a07d7d2725b8fbc99c50e"
Oct 30 07:27:08.135445 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:27:08.135309    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d20287cb61e3868c7478b45b7d527b6067091566f9a07d7d2725b8fbc99c50e\": container with ID starting with 8d20287cb61e3868c7478b45b7d527b6067091566f9a07d7d2725b8fbc99c50e not found: ID does not exist" containerID="8d20287cb61e3868c7478b45b7d527b6067091566f9a07d7d2725b8fbc99c50e"
Oct 30 07:27:08.135445 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:08.135334    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d20287cb61e3868c7478b45b7d527b6067091566f9a07d7d2725b8fbc99c50e"} err="failed to get container status \"8d20287cb61e3868c7478b45b7d527b6067091566f9a07d7d2725b8fbc99c50e\": rpc error: code = NotFound desc = could not find container \"8d20287cb61e3868c7478b45b7d527b6067091566f9a07d7d2725b8fbc99c50e\": container with ID starting with 8d20287cb61e3868c7478b45b7d527b6067091566f9a07d7d2725b8fbc99c50e not found: ID does not exist"
Oct 30 07:27:08.135445 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:08.135357    6224 scope.go:122] "RemoveContainer" containerID="390329941ea55222838eb235936ba87674826be3cf0a03aa5e901ad9b6034968"
Oct 30 07:27:08.144488 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:08.144461    6224 scope.go:122] "RemoveContainer" containerID="390329941ea55222838eb235936ba87674826be3cf0a03aa5e901ad9b6034968"
Oct 30 07:27:08.144740 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:27:08.144725    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"390329941ea55222838eb235936ba87674826be3cf0a03aa5e901ad9b6034968\": container with ID starting with 390329941ea55222838eb235936ba87674826be3cf0a03aa5e901ad9b6034968 not found: ID does not exist" containerID="390329941ea55222838eb235936ba87674826be3cf0a03aa5e901ad9b6034968"
Oct 30 07:27:08.144792 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:08.144746    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"390329941ea55222838eb235936ba87674826be3cf0a03aa5e901ad9b6034968"} err="failed to get container status \"390329941ea55222838eb235936ba87674826be3cf0a03aa5e901ad9b6034968\": rpc error: code = NotFound desc = could not find container \"390329941ea55222838eb235936ba87674826be3cf0a03aa5e901ad9b6034968\": container with ID starting with 390329941ea55222838eb235936ba87674826be3cf0a03aa5e901ad9b6034968 not found: ID does not exist"
Oct 30 07:27:08.195925 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:08.195856    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-04aede-5cd4c88478-kbvnt"]
Oct 30 07:27:08.203551 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:08.203521    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-04aede-5cd4c88478-kbvnt"]
Oct 30 07:27:08.242788 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:08.242760    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-04aede-776cc889f-47jhw"]
Oct 30 07:27:08.246398 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:08.246376    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-04aede-776cc889f-47jhw"]
Oct 30 07:27:08.389512 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:08.389403    6224 patch_prober.go:29] interesting pod/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Oct 30 07:27:08.389512 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:08.389470    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc" podUID="50bc25e6-c53f-4d16-b1fd-738cdd8c6a10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Oct 30 07:27:09.393897 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.393863    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc"
Oct 30 07:27:09.394326 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.394260    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc"
Oct 30 07:27:09.394943 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.394922    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc"
Oct 30 07:27:09.475524 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.475495    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9"]
Oct 30 07:27:09.476151 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.476134    6224 cpu_manager.go:414] "containerMap: removing container" podUID="c23e777f-2b9b-45fb-8110-62912dcbbaa6" containerName="payload-pre-processing"
Oct 30 07:27:09.476233 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.476152    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c23e777f-2b9b-45fb-8110-62912dcbbaa6" containerName="payload-pre-processing"
Oct 30 07:27:09.476233 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.476181    6224 cpu_manager.go:414] "containerMap: removing container" podUID="7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe" containerName="main"
Oct 30 07:27:09.476233 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.476186    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe" containerName="main"
Oct 30 07:27:09.476233 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.476205    6224 cpu_manager.go:414] "containerMap: removing container" podUID="bf4f18e1-ad1c-4315-9c07-d756d6d70476" containerName="payload-processing"
Oct 30 07:27:09.476233 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.476214    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf4f18e1-ad1c-4315-9c07-d756d6d70476" containerName="payload-processing"
Oct 30 07:27:09.476453 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.476354    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="c23e777f-2b9b-45fb-8110-62912dcbbaa6" containerName="payload-pre-processing"
Oct 30 07:27:09.476453 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.476367    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c7cda87-d4ae-45d4-b56b-5c71e8c32dbe" containerName="main"
Oct 30 07:27:09.476453 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.476376    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf4f18e1-ad1c-4315-9c07-d756d6d70476" containerName="payload-processing"
Oct 30 07:27:09.483389 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.483364    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9"]
Oct 30 07:27:09.483542 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.483506    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9"
Oct 30 07:27:09.488802 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.488780    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9"
Oct 30 07:27:09.513212 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.513180    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf4f18e1-ad1c-4315-9c07-d756d6d70476" path="/var/lib/kubelet/pods/bf4f18e1-ad1c-4315-9c07-d756d6d70476/volumes"
Oct 30 07:27:09.514019 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.513986    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c23e777f-2b9b-45fb-8110-62912dcbbaa6" path="/var/lib/kubelet/pods/c23e777f-2b9b-45fb-8110-62912dcbbaa6/volumes"
Oct 30 07:27:09.519377 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.514588    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9"]
Oct 30 07:27:09.519377 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.516718    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9"]
Oct 30 07:27:09.529483 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.529453    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3d25e61-890d-45bd-957a-6690c5f26a08-workload-certs\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9\" (UID: \"a3d25e61-890d-45bd-957a-6690c5f26a08\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9"
Oct 30 07:27:09.529611 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.529497    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee43b9a-1501-4a88-842e-564d9f1851f7-istiod-ca-cert\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9\" (UID: \"7ee43b9a-1501-4a88-842e-564d9f1851f7\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9"
Oct 30 07:27:09.529972 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.529535    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3d25e61-890d-45bd-957a-6690c5f26a08-istio-data\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9\" (UID: \"a3d25e61-890d-45bd-957a-6690c5f26a08\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9"
Oct 30 07:27:09.530092 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.530066    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee43b9a-1501-4a88-842e-564d9f1851f7-credential-socket\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9\" (UID: \"7ee43b9a-1501-4a88-842e-564d9f1851f7\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9"
Oct 30 07:27:09.530161 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.530117    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3d25e61-890d-45bd-957a-6690c5f26a08-istio-token\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9\" (UID: \"a3d25e61-890d-45bd-957a-6690c5f26a08\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9"
Oct 30 07:27:09.530220 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.530159    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52w5d\" (UniqueName: \"kubernetes.io/projected/7ee43b9a-1501-4a88-842e-564d9f1851f7-kube-api-access-52w5d\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9\" (UID: \"7ee43b9a-1501-4a88-842e-564d9f1851f7\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9"
Oct 30 07:27:09.530220 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.530210    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee43b9a-1501-4a88-842e-564d9f1851f7-istio-podinfo\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9\" (UID: \"7ee43b9a-1501-4a88-842e-564d9f1851f7\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9"
Oct 30 07:27:09.530334 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.530238    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee43b9a-1501-4a88-842e-564d9f1851f7-workload-socket\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9\" (UID: \"7ee43b9a-1501-4a88-842e-564d9f1851f7\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9"
Oct 30 07:27:09.530334 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.530288    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee43b9a-1501-4a88-842e-564d9f1851f7-workload-certs\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9\" (UID: \"7ee43b9a-1501-4a88-842e-564d9f1851f7\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9"
Oct 30 07:27:09.530334 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.530311    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3d25e61-890d-45bd-957a-6690c5f26a08-istio-podinfo\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9\" (UID: \"a3d25e61-890d-45bd-957a-6690c5f26a08\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9"
Oct 30 07:27:09.530490 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.530367    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5rn5\" (UniqueName: \"kubernetes.io/projected/a3d25e61-890d-45bd-957a-6690c5f26a08-kube-api-access-g5rn5\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9\" (UID: \"a3d25e61-890d-45bd-957a-6690c5f26a08\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9"
Oct 30 07:27:09.530490 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.530392    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3d25e61-890d-45bd-957a-6690c5f26a08-credential-socket\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9\" (UID: \"a3d25e61-890d-45bd-957a-6690c5f26a08\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9"
Oct 30 07:27:09.530605 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.530589    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee43b9a-1501-4a88-842e-564d9f1851f7-istio-data\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9\" (UID: \"7ee43b9a-1501-4a88-842e-564d9f1851f7\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9"
Oct 30 07:27:09.530669 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.530653    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ee43b9a-1501-4a88-842e-564d9f1851f7-istio-token\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9\" (UID: \"7ee43b9a-1501-4a88-842e-564d9f1851f7\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9"
Oct 30 07:27:09.530731 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.530691    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3d25e61-890d-45bd-957a-6690c5f26a08-istiod-ca-cert\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9\" (UID: \"a3d25e61-890d-45bd-957a-6690c5f26a08\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9"
Oct 30 07:27:09.530731 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.530725    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee43b9a-1501-4a88-842e-564d9f1851f7-istio-envoy\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9\" (UID: \"7ee43b9a-1501-4a88-842e-564d9f1851f7\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9"
Oct 30 07:27:09.530884 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.530762    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3d25e61-890d-45bd-957a-6690c5f26a08-workload-socket\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9\" (UID: \"a3d25e61-890d-45bd-957a-6690c5f26a08\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9"
Oct 30 07:27:09.530992 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.530967    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3d25e61-890d-45bd-957a-6690c5f26a08-istio-envoy\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9\" (UID: \"a3d25e61-890d-45bd-957a-6690c5f26a08\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9"
Oct 30 07:27:09.631947 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.631913    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3d25e61-890d-45bd-957a-6690c5f26a08-istio-envoy\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9\" (UID: \"a3d25e61-890d-45bd-957a-6690c5f26a08\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9"
Oct 30 07:27:09.632115 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.631966    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3d25e61-890d-45bd-957a-6690c5f26a08-workload-certs\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9\" (UID: \"a3d25e61-890d-45bd-957a-6690c5f26a08\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9"
Oct 30 07:27:09.632155 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.632131    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee43b9a-1501-4a88-842e-564d9f1851f7-istiod-ca-cert\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9\" (UID: \"7ee43b9a-1501-4a88-842e-564d9f1851f7\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9"
Oct 30 07:27:09.632191 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.632173    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3d25e61-890d-45bd-957a-6690c5f26a08-istio-data\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9\" (UID: \"a3d25e61-890d-45bd-957a-6690c5f26a08\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9"
Oct 30 07:27:09.632254 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.632238    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee43b9a-1501-4a88-842e-564d9f1851f7-credential-socket\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9\" (UID: \"7ee43b9a-1501-4a88-842e-564d9f1851f7\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9"
Oct 30 07:27:09.632301 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.632287    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3d25e61-890d-45bd-957a-6690c5f26a08-istio-token\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9\" (UID: \"a3d25e61-890d-45bd-957a-6690c5f26a08\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9"
Oct 30 07:27:09.632417 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.632332    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3d25e61-890d-45bd-957a-6690c5f26a08-workload-certs\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9\" (UID: \"a3d25e61-890d-45bd-957a-6690c5f26a08\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9"
Oct 30 07:27:09.632417 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.632332    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52w5d\" (UniqueName: \"kubernetes.io/projected/7ee43b9a-1501-4a88-842e-564d9f1851f7-kube-api-access-52w5d\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9\" (UID: \"7ee43b9a-1501-4a88-842e-564d9f1851f7\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9"
Oct 30 07:27:09.632417 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.632405    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee43b9a-1501-4a88-842e-564d9f1851f7-istio-podinfo\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9\" (UID: \"7ee43b9a-1501-4a88-842e-564d9f1851f7\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9"
Oct 30 07:27:09.632583 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.632430    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee43b9a-1501-4a88-842e-564d9f1851f7-workload-socket\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9\" (UID: \"7ee43b9a-1501-4a88-842e-564d9f1851f7\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9"
Oct 30 07:27:09.632583 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.632515    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee43b9a-1501-4a88-842e-564d9f1851f7-workload-certs\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9\" (UID: \"7ee43b9a-1501-4a88-842e-564d9f1851f7\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9"
Oct 30 07:27:09.632583 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.632545    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3d25e61-890d-45bd-957a-6690c5f26a08-istio-podinfo\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9\" (UID: \"a3d25e61-890d-45bd-957a-6690c5f26a08\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9"
Oct 30 07:27:09.632747 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.632601    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3d25e61-890d-45bd-957a-6690c5f26a08-istio-data\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9\" (UID: \"a3d25e61-890d-45bd-957a-6690c5f26a08\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9"
Oct 30 07:27:09.632924 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.632900    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee43b9a-1501-4a88-842e-564d9f1851f7-credential-socket\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9\" (UID: \"7ee43b9a-1501-4a88-842e-564d9f1851f7\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9"
Oct 30 07:27:09.633016 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.632933    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee43b9a-1501-4a88-842e-564d9f1851f7-workload-socket\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9\" (UID: \"7ee43b9a-1501-4a88-842e-564d9f1851f7\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9"
Oct 30 07:27:09.633016 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.632938    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee43b9a-1501-4a88-842e-564d9f1851f7-istiod-ca-cert\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9\" (UID: \"7ee43b9a-1501-4a88-842e-564d9f1851f7\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9"
Oct 30 07:27:09.633171 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.632608    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5rn5\" (UniqueName: \"kubernetes.io/projected/a3d25e61-890d-45bd-957a-6690c5f26a08-kube-api-access-g5rn5\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9\" (UID: \"a3d25e61-890d-45bd-957a-6690c5f26a08\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9"
Oct 30 07:27:09.633239 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.633197    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3d25e61-890d-45bd-957a-6690c5f26a08-credential-socket\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9\" (UID: \"a3d25e61-890d-45bd-957a-6690c5f26a08\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9"
Oct 30 07:27:09.633239 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.633220    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee43b9a-1501-4a88-842e-564d9f1851f7-workload-certs\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9\" (UID: \"7ee43b9a-1501-4a88-842e-564d9f1851f7\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9"
Oct 30 07:27:09.633348 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.633265    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee43b9a-1501-4a88-842e-564d9f1851f7-istio-data\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9\" (UID: \"7ee43b9a-1501-4a88-842e-564d9f1851f7\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9"
Oct 30 07:27:09.633348 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.633290    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ee43b9a-1501-4a88-842e-564d9f1851f7-istio-token\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9\" (UID: \"7ee43b9a-1501-4a88-842e-564d9f1851f7\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9"
Oct 30 07:27:09.633348 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.633329    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3d25e61-890d-45bd-957a-6690c5f26a08-istiod-ca-cert\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9\" (UID: \"a3d25e61-890d-45bd-957a-6690c5f26a08\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9"
Oct 30 07:27:09.633511 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.633357    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee43b9a-1501-4a88-842e-564d9f1851f7-istio-envoy\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9\" (UID: \"7ee43b9a-1501-4a88-842e-564d9f1851f7\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9"
Oct 30 07:27:09.633511 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.633403    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3d25e61-890d-45bd-957a-6690c5f26a08-workload-socket\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9\" (UID: \"a3d25e61-890d-45bd-957a-6690c5f26a08\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9"
Oct 30 07:27:09.633511 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.633442    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3d25e61-890d-45bd-957a-6690c5f26a08-credential-socket\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9\" (UID: \"a3d25e61-890d-45bd-957a-6690c5f26a08\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9"
Oct 30 07:27:09.633688 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.633674    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee43b9a-1501-4a88-842e-564d9f1851f7-istio-data\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9\" (UID: \"7ee43b9a-1501-4a88-842e-564d9f1851f7\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9"
Oct 30 07:27:09.634156 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.634115    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3d25e61-890d-45bd-957a-6690c5f26a08-workload-socket\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9\" (UID: \"a3d25e61-890d-45bd-957a-6690c5f26a08\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9"
Oct 30 07:27:09.644164 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.644031    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee43b9a-1501-4a88-842e-564d9f1851f7-istio-envoy\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9\" (UID: \"7ee43b9a-1501-4a88-842e-564d9f1851f7\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9"
Oct 30 07:27:09.648950 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.645700    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3d25e61-890d-45bd-957a-6690c5f26a08-istio-envoy\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9\" (UID: \"a3d25e61-890d-45bd-957a-6690c5f26a08\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9"
Oct 30 07:27:09.648950 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.645846    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3d25e61-890d-45bd-957a-6690c5f26a08-istio-podinfo\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9\" (UID: \"a3d25e61-890d-45bd-957a-6690c5f26a08\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9"
Oct 30 07:27:09.648950 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.646303    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3d25e61-890d-45bd-957a-6690c5f26a08-istiod-ca-cert\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9\" (UID: \"a3d25e61-890d-45bd-957a-6690c5f26a08\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9"
Oct 30 07:27:09.653426 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.653394    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee43b9a-1501-4a88-842e-564d9f1851f7-istio-podinfo\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9\" (UID: \"7ee43b9a-1501-4a88-842e-564d9f1851f7\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9"
Oct 30 07:27:09.657500 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.657473    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52w5d\" (UniqueName: \"kubernetes.io/projected/7ee43b9a-1501-4a88-842e-564d9f1851f7-kube-api-access-52w5d\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9\" (UID: \"7ee43b9a-1501-4a88-842e-564d9f1851f7\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9"
Oct 30 07:27:09.662543 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.662519    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ee43b9a-1501-4a88-842e-564d9f1851f7-istio-token\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9\" (UID: \"7ee43b9a-1501-4a88-842e-564d9f1851f7\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9"
Oct 30 07:27:09.662641 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.662547    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3d25e61-890d-45bd-957a-6690c5f26a08-istio-token\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9\" (UID: \"a3d25e61-890d-45bd-957a-6690c5f26a08\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9"
Oct 30 07:27:09.663846 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.663825    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5rn5\" (UniqueName: \"kubernetes.io/projected/a3d25e61-890d-45bd-957a-6690c5f26a08-kube-api-access-g5rn5\") pod \"e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9\" (UID: \"a3d25e61-890d-45bd-957a-6690c5f26a08\") " pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9"
Oct 30 07:27:09.822108 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.822079    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9"
Oct 30 07:27:09.830520 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:09.830493    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9"
Oct 30 07:27:10.158323 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:27:10.158290    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3d25e61_890d_45bd_957a_6690c5f26a08.slice/crio-7c53bed9ecdf33a3285a4579ee9714c6bba180040e46f2cfae17060395e6b115 WatchSource:0}: Error finding container 7c53bed9ecdf33a3285a4579ee9714c6bba180040e46f2cfae17060395e6b115: Status 404 returned error can't find the container with id 7c53bed9ecdf33a3285a4579ee9714c6bba180040e46f2cfae17060395e6b115
Oct 30 07:27:10.159285 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:10.159261    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9"]
Oct 30 07:27:10.161020 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:10.160985    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:27:10.161144 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:10.161048    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:27:10.161144 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:10.161078    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:27:10.163872 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:10.163847    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9"]
Oct 30 07:27:10.171847 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:27:10.171824    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ee43b9a_1501_4a88_842e_564d9f1851f7.slice/crio-a4c9d81d2c6a227307ac350089422c58e8fa0ad12527a957b752806ab8b7f600 WatchSource:0}: Error finding container a4c9d81d2c6a227307ac350089422c58e8fa0ad12527a957b752806ab8b7f600: Status 404 returned error can't find the container with id a4c9d81d2c6a227307ac350089422c58e8fa0ad12527a957b752806ab8b7f600
Oct 30 07:27:10.174745 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:10.174709    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:27:10.174845 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:10.174794    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:27:10.174893 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:10.174877    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:27:11.150564 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:11.150525    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9" event={"ID":"a3d25e61-890d-45bd-957a-6690c5f26a08","Type":"ContainerStarted","Data":"7c12c72888c54b6f84c4bd64842231316058fc7987d9af92fb1f91c229c4ddaa"}
Oct 30 07:27:11.151066 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:11.150599    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9" event={"ID":"a3d25e61-890d-45bd-957a-6690c5f26a08","Type":"ContainerStarted","Data":"7c53bed9ecdf33a3285a4579ee9714c6bba180040e46f2cfae17060395e6b115"}
Oct 30 07:27:11.152571 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:11.152543    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9" event={"ID":"7ee43b9a-1501-4a88-842e-564d9f1851f7","Type":"ContainerStarted","Data":"c61bb1da292be9fd9bb682d9ac0b1d2d3d91f9b401e7af65068ceb460c2ef6ea"}
Oct 30 07:27:11.152661 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:11.152594    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9" event={"ID":"7ee43b9a-1501-4a88-842e-564d9f1851f7","Type":"ContainerStarted","Data":"a4c9d81d2c6a227307ac350089422c58e8fa0ad12527a957b752806ab8b7f600"}
Oct 30 07:27:11.222828 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:11.222672    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9" podStartSLOduration=2.222657396 podStartE2EDuration="2.222657396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:27:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:27:11.221981879 +0000 UTC m=+1789.831388862" watchObservedRunningTime="2026-10-30 07:27:11.222657396 +0000 UTC m=+1789.832064365"
Oct 30 07:27:11.823075 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:11.823044    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9"
Oct 30 07:27:11.825398 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:11.825362    6224 patch_prober.go:29] interesting pod/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Oct 30 07:27:11.825547 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:11.825427    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9" podUID="7ee43b9a-1501-4a88-842e-564d9f1851f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Oct 30 07:27:11.831636 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:11.831609    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9"
Oct 30 07:27:11.832901 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:11.832879    6224 patch_prober.go:29] interesting pod/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Oct 30 07:27:11.833004 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:11.832934    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9" podUID="a3d25e61-890d-45bd-957a-6690c5f26a08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Oct 30 07:27:12.839829 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:12.837394    6224 patch_prober.go:29] interesting pod/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Oct 30 07:27:12.839829 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:12.837459    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9" podUID="a3d25e61-890d-45bd-957a-6690c5f26a08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Oct 30 07:27:12.843825 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:12.842292    6224 patch_prober.go:29] interesting pod/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Oct 30 07:27:12.843825 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:12.842350    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9" podUID="7ee43b9a-1501-4a88-842e-564d9f1851f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Oct 30 07:27:13.133230 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:13.133152    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b5998f-57685c68cb-xts98"
Oct 30 07:27:13.226700 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:13.226621    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9" podStartSLOduration=4.226600788 podStartE2EDuration="4.226600788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:27:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:27:11.284315122 +0000 UTC m=+1789.893722085" watchObservedRunningTime="2026-10-30 07:27:13.226600788 +0000 UTC m=+1791.836007759"
Oct 30 07:27:13.825750 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:13.825244    6224 patch_prober.go:29] interesting pod/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Oct 30 07:27:13.825750 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:13.825311    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9" podUID="7ee43b9a-1501-4a88-842e-564d9f1851f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Oct 30 07:27:13.832172 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:13.832142    6224 patch_prober.go:29] interesting pod/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Oct 30 07:27:13.832387 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:13.832366    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9" podUID="a3d25e61-890d-45bd-957a-6690c5f26a08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Oct 30 07:27:14.823261 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:14.823218    6224 patch_prober.go:29] interesting pod/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Oct 30 07:27:14.823955 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:14.823917    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9" podUID="7ee43b9a-1501-4a88-842e-564d9f1851f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Oct 30 07:27:14.831374 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:14.831347    6224 patch_prober.go:29] interesting pod/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Oct 30 07:27:14.831512 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:14.831417    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9" podUID="a3d25e61-890d-45bd-957a-6690c5f26a08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Oct 30 07:27:15.823225 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:15.823189    6224 patch_prober.go:29] interesting pod/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Oct 30 07:27:15.823535 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:15.823496    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9" podUID="7ee43b9a-1501-4a88-842e-564d9f1851f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Oct 30 07:27:15.831298 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:15.831274    6224 patch_prober.go:29] interesting pod/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Oct 30 07:27:15.831432 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:15.831411    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9" podUID="a3d25e61-890d-45bd-957a-6690c5f26a08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Oct 30 07:27:16.450840 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:16.449087    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-14271d/auth-test-model-14271d-kserve-6c67747f77-k8psp"]
Oct 30 07:27:16.477375 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:16.472044    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-14271d/auth-test-model-14271d-kserve-6c67747f77-k8psp"
Oct 30 07:27:16.490586 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:16.490530    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-14271d/auth-test-model-14271d-kserve-6c67747f77-k8psp"]
Oct 30 07:27:16.504288 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:16.504264    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-14271d\"/\"openshift-service-ca.crt\""
Oct 30 07:27:16.504482 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:16.504457    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-14271d\"/\"auth-test-model-14271d-kserve-self-signed-certs\""
Oct 30 07:27:16.504604 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:16.504342    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-14271d\"/\"kube-root-ca.crt\""
Oct 30 07:27:16.504665 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:16.504267    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-14271d\"/\"default-dockercfg-qtpw7\""
Oct 30 07:27:16.543831 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:16.543765    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5bf8b84-73bd-4729-b5a7-1776c09f23f4-home\") pod \"auth-test-model-14271d-kserve-6c67747f77-k8psp\" (UID: \"f5bf8b84-73bd-4729-b5a7-1776c09f23f4\") " pod="ai-tenant-e2e-shared-a-w3-14271d/auth-test-model-14271d-kserve-6c67747f77-k8psp"
Oct 30 07:27:16.544197 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:16.544174    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnnr8\" (UniqueName: \"kubernetes.io/projected/f5bf8b84-73bd-4729-b5a7-1776c09f23f4-kube-api-access-fnnr8\") pod \"auth-test-model-14271d-kserve-6c67747f77-k8psp\" (UID: \"f5bf8b84-73bd-4729-b5a7-1776c09f23f4\") " pod="ai-tenant-e2e-shared-a-w3-14271d/auth-test-model-14271d-kserve-6c67747f77-k8psp"
Oct 30 07:27:16.544440 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:16.544422    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5bf8b84-73bd-4729-b5a7-1776c09f23f4-model-cache\") pod \"auth-test-model-14271d-kserve-6c67747f77-k8psp\" (UID: \"f5bf8b84-73bd-4729-b5a7-1776c09f23f4\") " pod="ai-tenant-e2e-shared-a-w3-14271d/auth-test-model-14271d-kserve-6c67747f77-k8psp"
Oct 30 07:27:16.544642 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:16.544623    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5bf8b84-73bd-4729-b5a7-1776c09f23f4-tls-certs\") pod \"auth-test-model-14271d-kserve-6c67747f77-k8psp\" (UID: \"f5bf8b84-73bd-4729-b5a7-1776c09f23f4\") " pod="ai-tenant-e2e-shared-a-w3-14271d/auth-test-model-14271d-kserve-6c67747f77-k8psp"
Oct 30 07:27:16.544776 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:16.544760    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5bf8b84-73bd-4729-b5a7-1776c09f23f4-dshm\") pod \"auth-test-model-14271d-kserve-6c67747f77-k8psp\" (UID: \"f5bf8b84-73bd-4729-b5a7-1776c09f23f4\") " pod="ai-tenant-e2e-shared-a-w3-14271d/auth-test-model-14271d-kserve-6c67747f77-k8psp"
Oct 30 07:27:16.544924 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:16.544907    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5bf8b84-73bd-4729-b5a7-1776c09f23f4-tmp-dir\") pod \"auth-test-model-14271d-kserve-6c67747f77-k8psp\" (UID: \"f5bf8b84-73bd-4729-b5a7-1776c09f23f4\") " pod="ai-tenant-e2e-shared-a-w3-14271d/auth-test-model-14271d-kserve-6c67747f77-k8psp"
Oct 30 07:27:16.646610 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:16.646568    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5bf8b84-73bd-4729-b5a7-1776c09f23f4-home\") pod \"auth-test-model-14271d-kserve-6c67747f77-k8psp\" (UID: \"f5bf8b84-73bd-4729-b5a7-1776c09f23f4\") " pod="ai-tenant-e2e-shared-a-w3-14271d/auth-test-model-14271d-kserve-6c67747f77-k8psp"
Oct 30 07:27:16.646818 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:16.646659    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnnr8\" (UniqueName: \"kubernetes.io/projected/f5bf8b84-73bd-4729-b5a7-1776c09f23f4-kube-api-access-fnnr8\") pod \"auth-test-model-14271d-kserve-6c67747f77-k8psp\" (UID: \"f5bf8b84-73bd-4729-b5a7-1776c09f23f4\") " pod="ai-tenant-e2e-shared-a-w3-14271d/auth-test-model-14271d-kserve-6c67747f77-k8psp"
Oct 30 07:27:16.646818 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:16.646739    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5bf8b84-73bd-4729-b5a7-1776c09f23f4-model-cache\") pod \"auth-test-model-14271d-kserve-6c67747f77-k8psp\" (UID: \"f5bf8b84-73bd-4729-b5a7-1776c09f23f4\") " pod="ai-tenant-e2e-shared-a-w3-14271d/auth-test-model-14271d-kserve-6c67747f77-k8psp"
Oct 30 07:27:16.646959 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:16.646877    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5bf8b84-73bd-4729-b5a7-1776c09f23f4-tls-certs\") pod \"auth-test-model-14271d-kserve-6c67747f77-k8psp\" (UID: \"f5bf8b84-73bd-4729-b5a7-1776c09f23f4\") " pod="ai-tenant-e2e-shared-a-w3-14271d/auth-test-model-14271d-kserve-6c67747f77-k8psp"
Oct 30 07:27:16.646959 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:16.646921    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5bf8b84-73bd-4729-b5a7-1776c09f23f4-dshm\") pod \"auth-test-model-14271d-kserve-6c67747f77-k8psp\" (UID: \"f5bf8b84-73bd-4729-b5a7-1776c09f23f4\") " pod="ai-tenant-e2e-shared-a-w3-14271d/auth-test-model-14271d-kserve-6c67747f77-k8psp"
Oct 30 07:27:16.646959 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:16.646954    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5bf8b84-73bd-4729-b5a7-1776c09f23f4-tmp-dir\") pod \"auth-test-model-14271d-kserve-6c67747f77-k8psp\" (UID: \"f5bf8b84-73bd-4729-b5a7-1776c09f23f4\") " pod="ai-tenant-e2e-shared-a-w3-14271d/auth-test-model-14271d-kserve-6c67747f77-k8psp"
Oct 30 07:27:16.647440 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:16.647411    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5bf8b84-73bd-4729-b5a7-1776c09f23f4-tmp-dir\") pod \"auth-test-model-14271d-kserve-6c67747f77-k8psp\" (UID: \"f5bf8b84-73bd-4729-b5a7-1776c09f23f4\") " pod="ai-tenant-e2e-shared-a-w3-14271d/auth-test-model-14271d-kserve-6c67747f77-k8psp"
Oct 30 07:27:16.647712 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:16.647691    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5bf8b84-73bd-4729-b5a7-1776c09f23f4-home\") pod \"auth-test-model-14271d-kserve-6c67747f77-k8psp\" (UID: \"f5bf8b84-73bd-4729-b5a7-1776c09f23f4\") " pod="ai-tenant-e2e-shared-a-w3-14271d/auth-test-model-14271d-kserve-6c67747f77-k8psp"
Oct 30 07:27:16.648286 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:16.648262    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5bf8b84-73bd-4729-b5a7-1776c09f23f4-model-cache\") pod \"auth-test-model-14271d-kserve-6c67747f77-k8psp\" (UID: \"f5bf8b84-73bd-4729-b5a7-1776c09f23f4\") " pod="ai-tenant-e2e-shared-a-w3-14271d/auth-test-model-14271d-kserve-6c67747f77-k8psp"
Oct 30 07:27:16.658957 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:16.653873    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5bf8b84-73bd-4729-b5a7-1776c09f23f4-tls-certs\") pod \"auth-test-model-14271d-kserve-6c67747f77-k8psp\" (UID: \"f5bf8b84-73bd-4729-b5a7-1776c09f23f4\") " pod="ai-tenant-e2e-shared-a-w3-14271d/auth-test-model-14271d-kserve-6c67747f77-k8psp"
Oct 30 07:27:16.658957 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:16.655320    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5bf8b84-73bd-4729-b5a7-1776c09f23f4-dshm\") pod \"auth-test-model-14271d-kserve-6c67747f77-k8psp\" (UID: \"f5bf8b84-73bd-4729-b5a7-1776c09f23f4\") " pod="ai-tenant-e2e-shared-a-w3-14271d/auth-test-model-14271d-kserve-6c67747f77-k8psp"
Oct 30 07:27:16.673854 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:16.673789    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnnr8\" (UniqueName: \"kubernetes.io/projected/f5bf8b84-73bd-4729-b5a7-1776c09f23f4-kube-api-access-fnnr8\") pod \"auth-test-model-14271d-kserve-6c67747f77-k8psp\" (UID: \"f5bf8b84-73bd-4729-b5a7-1776c09f23f4\") " pod="ai-tenant-e2e-shared-a-w3-14271d/auth-test-model-14271d-kserve-6c67747f77-k8psp"
Oct 30 07:27:16.804114 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:16.804080    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-14271d/auth-test-model-14271d-kserve-6c67747f77-k8psp"
Oct 30 07:27:16.827905 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:16.827871    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9"
Oct 30 07:27:16.828558 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:16.828448    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9"
Oct 30 07:27:16.829888 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:16.829854    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9"
Oct 30 07:27:16.838182 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:16.838162    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9"
Oct 30 07:27:16.838558 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:16.838538    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9"
Oct 30 07:27:16.839550 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:16.839529    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9"
Oct 30 07:27:17.339718 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:17.339550    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-14271d/auth-test-model-14271d-kserve-6c67747f77-k8psp"]
Oct 30 07:27:17.344088 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:27:17.344058    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf5bf8b84_73bd_4729_b5a7_1776c09f23f4.slice/crio-47415a471c0bfd4b8eaf8655be240736de41b58ecc83ab7a1b9bcc1b10daf939 WatchSource:0}: Error finding container 47415a471c0bfd4b8eaf8655be240736de41b58ecc83ab7a1b9bcc1b10daf939: Status 404 returned error can't find the container with id 47415a471c0bfd4b8eaf8655be240736de41b58ecc83ab7a1b9bcc1b10daf939
Oct 30 07:27:18.357189 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:18.357153    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-14271d/auth-test-model-14271d-kserve-6c67747f77-k8psp" event={"ID":"f5bf8b84-73bd-4729-b5a7-1776c09f23f4","Type":"ContainerStarted","Data":"9a844d9df856143fa2fbc735216cacf392e12846a578fef95d92684876aa2c7e"}
Oct 30 07:27:18.357628 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:18.357201    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-14271d/auth-test-model-14271d-kserve-6c67747f77-k8psp" event={"ID":"f5bf8b84-73bd-4729-b5a7-1776c09f23f4","Type":"ContainerStarted","Data":"47415a471c0bfd4b8eaf8655be240736de41b58ecc83ab7a1b9bcc1b10daf939"}
Oct 30 07:27:18.401779 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:18.401726    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-14271d/auth-test-model-14271d-kserve-6c67747f77-k8psp" podStartSLOduration=2.401713457 podStartE2EDuration="2.401713457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:27:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:27:18.401599301 +0000 UTC m=+1797.011006287" watchObservedRunningTime="2026-10-30 07:27:18.401713457 +0000 UTC m=+1797.011120427"
Oct 30 07:27:18.805207 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:18.805172    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-14271d/auth-test-model-14271d-kserve-6c67747f77-k8psp"
Oct 30 07:27:22.329841 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:22.324700    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:27:26.804974 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:26.804936    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-14271d/auth-test-model-14271d-kserve-6c67747f77-k8psp"
Oct 30 07:27:26.820091 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:26.820062    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-14271d/auth-test-model-14271d-kserve-6c67747f77-k8psp"
Oct 30 07:27:27.449048 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:27.449016    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-14271d/auth-test-model-14271d-kserve-6c67747f77-k8psp"
Oct 30 07:27:31.859796 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:27:31.859754    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:27:32.782996 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:32.782921    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f84d9e-78664b96c-lswqs"]
Oct 30 07:27:32.789206 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:32.789181    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f84d9e-78664b96c-lswqs"
Oct 30 07:27:32.805737 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:32.805708    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f84d9e-78664b96c-lswqs"]
Oct 30 07:27:32.807785 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:32.807760    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec5be7a5-4988-4f68-9e20-68c467379762-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-f84d9e-78664b96c-lswqs\" (UID: \"ec5be7a5-4988-4f68-9e20-68c467379762\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f84d9e-78664b96c-lswqs"
Oct 30 07:27:32.807949 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:32.807886    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47fp5\" (UniqueName: \"kubernetes.io/projected/ec5be7a5-4988-4f68-9e20-68c467379762-kube-api-access-47fp5\") pod \"payload-pre-processing-e2e-explicit-f84d9e-78664b96c-lswqs\" (UID: \"ec5be7a5-4988-4f68-9e20-68c467379762\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f84d9e-78664b96c-lswqs"
Oct 30 07:27:32.815394 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:32.815370    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-f84d9e\""
Oct 30 07:27:32.815826 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:32.815770    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-f84d9e-dockercfg-sfncn\""
Oct 30 07:27:32.942829 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:32.922367    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec5be7a5-4988-4f68-9e20-68c467379762-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-f84d9e-78664b96c-lswqs\" (UID: \"ec5be7a5-4988-4f68-9e20-68c467379762\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f84d9e-78664b96c-lswqs"
Oct 30 07:27:32.942829 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:32.922448    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47fp5\" (UniqueName: \"kubernetes.io/projected/ec5be7a5-4988-4f68-9e20-68c467379762-kube-api-access-47fp5\") pod \"payload-pre-processing-e2e-explicit-f84d9e-78664b96c-lswqs\" (UID: \"ec5be7a5-4988-4f68-9e20-68c467379762\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f84d9e-78664b96c-lswqs"
Oct 30 07:27:32.942829 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:32.924000    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec5be7a5-4988-4f68-9e20-68c467379762-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-f84d9e-78664b96c-lswqs\" (UID: \"ec5be7a5-4988-4f68-9e20-68c467379762\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f84d9e-78664b96c-lswqs"
Oct 30 07:27:32.995033 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:32.994688    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47fp5\" (UniqueName: \"kubernetes.io/projected/ec5be7a5-4988-4f68-9e20-68c467379762-kube-api-access-47fp5\") pod \"payload-pre-processing-e2e-explicit-f84d9e-78664b96c-lswqs\" (UID: \"ec5be7a5-4988-4f68-9e20-68c467379762\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f84d9e-78664b96c-lswqs"
Oct 30 07:27:33.105633 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:33.105471    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f84d9e-78664b96c-lswqs"
Oct 30 07:27:33.420233 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:27:33.420198    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec5be7a5_4988_4f68_9e20_68c467379762.slice/crio-c1a7729ec407386391c24a416db7ee804299b4ee81b68aa51ec3ab1eb2758f57 WatchSource:0}: Error finding container c1a7729ec407386391c24a416db7ee804299b4ee81b68aa51ec3ab1eb2758f57: Status 404 returned error can't find the container with id c1a7729ec407386391c24a416db7ee804299b4ee81b68aa51ec3ab1eb2758f57
Oct 30 07:27:33.421192 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:33.421163    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f84d9e-78664b96c-lswqs"]
Oct 30 07:27:33.482054 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:33.482027    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f84d9e-78664b96c-lswqs" event={"ID":"ec5be7a5-4988-4f68-9e20-68c467379762","Type":"ContainerStarted","Data":"c1a7729ec407386391c24a416db7ee804299b4ee81b68aa51ec3ab1eb2758f57"}
Oct 30 07:27:34.490085 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:34.490048    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f84d9e-78664b96c-lswqs" event={"ID":"ec5be7a5-4988-4f68-9e20-68c467379762","Type":"ContainerStarted","Data":"08eed846764f86a908bfe9f10c62efb0b9369e42b45e19fdd911c906a1ae68a1"}
Oct 30 07:27:34.490507 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:34.490138    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f84d9e-78664b96c-lswqs"
Oct 30 07:27:34.538709 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:34.538652    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f84d9e-78664b96c-lswqs" podStartSLOduration=2.538633736 podStartE2EDuration="2.538633736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:27:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:27:34.53474294 +0000 UTC m=+1813.144149910" watchObservedRunningTime="2026-10-30 07:27:34.538633736 +0000 UTC m=+1813.148040706"
Oct 30 07:27:38.881304 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:38.881272    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-265384/auth-test-model-265384-kserve-556c8f946f-7l2tb"]
Oct 30 07:27:38.885819 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:38.885788    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-265384/auth-test-model-265384-kserve-556c8f946f-7l2tb"
Oct 30 07:27:38.939603 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:38.939567    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-265384/auth-test-model-265384-kserve-556c8f946f-7l2tb"]
Oct 30 07:27:38.942001 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:38.941973    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-265384\"/\"kube-root-ca.crt\""
Oct 30 07:27:38.942168 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:38.942038    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-265384\"/\"openshift-service-ca.crt\""
Oct 30 07:27:38.947287 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:38.947263    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-265384\"/\"default-dockercfg-dh4nj\""
Oct 30 07:27:38.947435 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:38.947304    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-265384\"/\"auth-test-model-265384-kserve-self-signed-certs\""
Oct 30 07:27:39.016276 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:39.016231    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/012fc6e1-7b44-448e-89d4-848cede712f0-tls-certs\") pod \"auth-test-model-265384-kserve-556c8f946f-7l2tb\" (UID: \"012fc6e1-7b44-448e-89d4-848cede712f0\") " pod="ai-tenant-e2e-shared-a-w1-265384/auth-test-model-265384-kserve-556c8f946f-7l2tb"
Oct 30 07:27:39.016497 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:39.016360    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/012fc6e1-7b44-448e-89d4-848cede712f0-tmp-dir\") pod \"auth-test-model-265384-kserve-556c8f946f-7l2tb\" (UID: \"012fc6e1-7b44-448e-89d4-848cede712f0\") " pod="ai-tenant-e2e-shared-a-w1-265384/auth-test-model-265384-kserve-556c8f946f-7l2tb"
Oct 30 07:27:39.016497 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:39.016415    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/012fc6e1-7b44-448e-89d4-848cede712f0-home\") pod \"auth-test-model-265384-kserve-556c8f946f-7l2tb\" (UID: \"012fc6e1-7b44-448e-89d4-848cede712f0\") " pod="ai-tenant-e2e-shared-a-w1-265384/auth-test-model-265384-kserve-556c8f946f-7l2tb"
Oct 30 07:27:39.016497 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:39.016439    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2sdc\" (UniqueName: \"kubernetes.io/projected/012fc6e1-7b44-448e-89d4-848cede712f0-kube-api-access-s2sdc\") pod \"auth-test-model-265384-kserve-556c8f946f-7l2tb\" (UID: \"012fc6e1-7b44-448e-89d4-848cede712f0\") " pod="ai-tenant-e2e-shared-a-w1-265384/auth-test-model-265384-kserve-556c8f946f-7l2tb"
Oct 30 07:27:39.016684 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:39.016665    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/012fc6e1-7b44-448e-89d4-848cede712f0-dshm\") pod \"auth-test-model-265384-kserve-556c8f946f-7l2tb\" (UID: \"012fc6e1-7b44-448e-89d4-848cede712f0\") " pod="ai-tenant-e2e-shared-a-w1-265384/auth-test-model-265384-kserve-556c8f946f-7l2tb"
Oct 30 07:27:39.016742 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:39.016704    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/012fc6e1-7b44-448e-89d4-848cede712f0-model-cache\") pod \"auth-test-model-265384-kserve-556c8f946f-7l2tb\" (UID: \"012fc6e1-7b44-448e-89d4-848cede712f0\") " pod="ai-tenant-e2e-shared-a-w1-265384/auth-test-model-265384-kserve-556c8f946f-7l2tb"
Oct 30 07:27:39.118487 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:39.118452    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/012fc6e1-7b44-448e-89d4-848cede712f0-dshm\") pod \"auth-test-model-265384-kserve-556c8f946f-7l2tb\" (UID: \"012fc6e1-7b44-448e-89d4-848cede712f0\") " pod="ai-tenant-e2e-shared-a-w1-265384/auth-test-model-265384-kserve-556c8f946f-7l2tb"
Oct 30 07:27:39.118717 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:39.118515    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/012fc6e1-7b44-448e-89d4-848cede712f0-model-cache\") pod \"auth-test-model-265384-kserve-556c8f946f-7l2tb\" (UID: \"012fc6e1-7b44-448e-89d4-848cede712f0\") " pod="ai-tenant-e2e-shared-a-w1-265384/auth-test-model-265384-kserve-556c8f946f-7l2tb"
Oct 30 07:27:39.118717 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:39.118566    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/012fc6e1-7b44-448e-89d4-848cede712f0-tls-certs\") pod \"auth-test-model-265384-kserve-556c8f946f-7l2tb\" (UID: \"012fc6e1-7b44-448e-89d4-848cede712f0\") " pod="ai-tenant-e2e-shared-a-w1-265384/auth-test-model-265384-kserve-556c8f946f-7l2tb"
Oct 30 07:27:39.118717 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:39.118653    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/012fc6e1-7b44-448e-89d4-848cede712f0-tmp-dir\") pod \"auth-test-model-265384-kserve-556c8f946f-7l2tb\" (UID: \"012fc6e1-7b44-448e-89d4-848cede712f0\") " pod="ai-tenant-e2e-shared-a-w1-265384/auth-test-model-265384-kserve-556c8f946f-7l2tb"
Oct 30 07:27:39.118717 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:39.118702    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/012fc6e1-7b44-448e-89d4-848cede712f0-home\") pod \"auth-test-model-265384-kserve-556c8f946f-7l2tb\" (UID: \"012fc6e1-7b44-448e-89d4-848cede712f0\") " pod="ai-tenant-e2e-shared-a-w1-265384/auth-test-model-265384-kserve-556c8f946f-7l2tb"
Oct 30 07:27:39.118978 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:39.118734    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2sdc\" (UniqueName: \"kubernetes.io/projected/012fc6e1-7b44-448e-89d4-848cede712f0-kube-api-access-s2sdc\") pod \"auth-test-model-265384-kserve-556c8f946f-7l2tb\" (UID: \"012fc6e1-7b44-448e-89d4-848cede712f0\") " pod="ai-tenant-e2e-shared-a-w1-265384/auth-test-model-265384-kserve-556c8f946f-7l2tb"
Oct 30 07:27:39.119129 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:39.119053    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/012fc6e1-7b44-448e-89d4-848cede712f0-model-cache\") pod \"auth-test-model-265384-kserve-556c8f946f-7l2tb\" (UID: \"012fc6e1-7b44-448e-89d4-848cede712f0\") " pod="ai-tenant-e2e-shared-a-w1-265384/auth-test-model-265384-kserve-556c8f946f-7l2tb"
Oct 30 07:27:39.119286 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:39.119262    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/012fc6e1-7b44-448e-89d4-848cede712f0-tmp-dir\") pod \"auth-test-model-265384-kserve-556c8f946f-7l2tb\" (UID: \"012fc6e1-7b44-448e-89d4-848cede712f0\") " pod="ai-tenant-e2e-shared-a-w1-265384/auth-test-model-265384-kserve-556c8f946f-7l2tb"
Oct 30 07:27:39.119349 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:39.119329    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/012fc6e1-7b44-448e-89d4-848cede712f0-home\") pod \"auth-test-model-265384-kserve-556c8f946f-7l2tb\" (UID: \"012fc6e1-7b44-448e-89d4-848cede712f0\") " pod="ai-tenant-e2e-shared-a-w1-265384/auth-test-model-265384-kserve-556c8f946f-7l2tb"
Oct 30 07:27:39.121168 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:39.121146    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/012fc6e1-7b44-448e-89d4-848cede712f0-dshm\") pod \"auth-test-model-265384-kserve-556c8f946f-7l2tb\" (UID: \"012fc6e1-7b44-448e-89d4-848cede712f0\") " pod="ai-tenant-e2e-shared-a-w1-265384/auth-test-model-265384-kserve-556c8f946f-7l2tb"
Oct 30 07:27:39.121605 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:39.121590    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/012fc6e1-7b44-448e-89d4-848cede712f0-tls-certs\") pod \"auth-test-model-265384-kserve-556c8f946f-7l2tb\" (UID: \"012fc6e1-7b44-448e-89d4-848cede712f0\") " pod="ai-tenant-e2e-shared-a-w1-265384/auth-test-model-265384-kserve-556c8f946f-7l2tb"
Oct 30 07:27:39.171898 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:39.170499    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2sdc\" (UniqueName: \"kubernetes.io/projected/012fc6e1-7b44-448e-89d4-848cede712f0-kube-api-access-s2sdc\") pod \"auth-test-model-265384-kserve-556c8f946f-7l2tb\" (UID: \"012fc6e1-7b44-448e-89d4-848cede712f0\") " pod="ai-tenant-e2e-shared-a-w1-265384/auth-test-model-265384-kserve-556c8f946f-7l2tb"
Oct 30 07:27:39.206829 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:39.200097    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-265384/auth-test-model-265384-kserve-556c8f946f-7l2tb"
Oct 30 07:27:39.545592 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:27:39.545557    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod012fc6e1_7b44_448e_89d4_848cede712f0.slice/crio-4c19044692a32a6fd6ed57b64570e90ddf0105df782f464ffc631a18428a6279 WatchSource:0}: Error finding container 4c19044692a32a6fd6ed57b64570e90ddf0105df782f464ffc631a18428a6279: Status 404 returned error can't find the container with id 4c19044692a32a6fd6ed57b64570e90ddf0105df782f464ffc631a18428a6279
Oct 30 07:27:39.547702 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:39.547678    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-265384/auth-test-model-265384-kserve-556c8f946f-7l2tb"]
Oct 30 07:27:40.537720 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:40.537685    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-265384/auth-test-model-265384-kserve-556c8f946f-7l2tb" event={"ID":"012fc6e1-7b44-448e-89d4-848cede712f0","Type":"ContainerStarted","Data":"5db83520c42f8087319353fbcd8981372df1cae5c97e1a7fad01f4d19a268af5"}
Oct 30 07:27:40.537720 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:40.537733    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-265384/auth-test-model-265384-kserve-556c8f946f-7l2tb" event={"ID":"012fc6e1-7b44-448e-89d4-848cede712f0","Type":"ContainerStarted","Data":"4c19044692a32a6fd6ed57b64570e90ddf0105df782f464ffc631a18428a6279"}
Oct 30 07:27:40.587941 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:40.587895    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-265384/auth-test-model-265384-kserve-556c8f946f-7l2tb" podStartSLOduration=2.587882015 podStartE2EDuration="2.587882015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:27:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:27:40.585298053 +0000 UTC m=+1819.194705022" watchObservedRunningTime="2026-10-30 07:27:40.587882015 +0000 UTC m=+1819.197289021"
Oct 30 07:27:41.204042 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:41.204005    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-265384/auth-test-model-265384-kserve-556c8f946f-7l2tb"
Oct 30 07:27:44.155092 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:44.155060    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-932c00/auth-test-model-932c00-kserve-5496946777-b8b4x"]
Oct 30 07:27:44.161106 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:44.161077    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-932c00/auth-test-model-932c00-kserve-5496946777-b8b4x"
Oct 30 07:27:44.183595 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:44.183558    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-932c00/auth-test-model-932c00-kserve-5496946777-b8b4x"]
Oct 30 07:27:44.188821 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:44.188782    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-932c00\"/\"openshift-service-ca.crt\""
Oct 30 07:27:44.188967 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:44.188925    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-932c00\"/\"kube-root-ca.crt\""
Oct 30 07:27:44.189038 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:44.189019    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-932c00\"/\"default-dockercfg-9fsbl\""
Oct 30 07:27:44.192089 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:44.192069    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-932c00\"/\"auth-test-model-932c00-kserve-self-signed-certs\""
Oct 30 07:27:44.310050 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:44.306393    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72f1915d-8b19-4459-8450-436b4c30ba38-model-cache\") pod \"auth-test-model-932c00-kserve-5496946777-b8b4x\" (UID: \"72f1915d-8b19-4459-8450-436b4c30ba38\") " pod="ai-tenant-e2e-shared-a-w2-932c00/auth-test-model-932c00-kserve-5496946777-b8b4x"
Oct 30 07:27:44.310050 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:44.306591    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72f1915d-8b19-4459-8450-436b4c30ba38-dshm\") pod \"auth-test-model-932c00-kserve-5496946777-b8b4x\" (UID: \"72f1915d-8b19-4459-8450-436b4c30ba38\") " pod="ai-tenant-e2e-shared-a-w2-932c00/auth-test-model-932c00-kserve-5496946777-b8b4x"
Oct 30 07:27:44.310050 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:44.306634    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72f1915d-8b19-4459-8450-436b4c30ba38-tls-certs\") pod \"auth-test-model-932c00-kserve-5496946777-b8b4x\" (UID: \"72f1915d-8b19-4459-8450-436b4c30ba38\") " pod="ai-tenant-e2e-shared-a-w2-932c00/auth-test-model-932c00-kserve-5496946777-b8b4x"
Oct 30 07:27:44.310050 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:44.306678    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72f1915d-8b19-4459-8450-436b4c30ba38-tmp-dir\") pod \"auth-test-model-932c00-kserve-5496946777-b8b4x\" (UID: \"72f1915d-8b19-4459-8450-436b4c30ba38\") " pod="ai-tenant-e2e-shared-a-w2-932c00/auth-test-model-932c00-kserve-5496946777-b8b4x"
Oct 30 07:27:44.310050 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:44.306767    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxxqq\" (UniqueName: \"kubernetes.io/projected/72f1915d-8b19-4459-8450-436b4c30ba38-kube-api-access-qxxqq\") pod \"auth-test-model-932c00-kserve-5496946777-b8b4x\" (UID: \"72f1915d-8b19-4459-8450-436b4c30ba38\") " pod="ai-tenant-e2e-shared-a-w2-932c00/auth-test-model-932c00-kserve-5496946777-b8b4x"
Oct 30 07:27:44.310050 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:44.306842    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72f1915d-8b19-4459-8450-436b4c30ba38-home\") pod \"auth-test-model-932c00-kserve-5496946777-b8b4x\" (UID: \"72f1915d-8b19-4459-8450-436b4c30ba38\") " pod="ai-tenant-e2e-shared-a-w2-932c00/auth-test-model-932c00-kserve-5496946777-b8b4x"
Oct 30 07:27:44.408062 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:44.407963    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxxqq\" (UniqueName: \"kubernetes.io/projected/72f1915d-8b19-4459-8450-436b4c30ba38-kube-api-access-qxxqq\") pod \"auth-test-model-932c00-kserve-5496946777-b8b4x\" (UID: \"72f1915d-8b19-4459-8450-436b4c30ba38\") " pod="ai-tenant-e2e-shared-a-w2-932c00/auth-test-model-932c00-kserve-5496946777-b8b4x"
Oct 30 07:27:44.408062 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:44.408044    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72f1915d-8b19-4459-8450-436b4c30ba38-home\") pod \"auth-test-model-932c00-kserve-5496946777-b8b4x\" (UID: \"72f1915d-8b19-4459-8450-436b4c30ba38\") " pod="ai-tenant-e2e-shared-a-w2-932c00/auth-test-model-932c00-kserve-5496946777-b8b4x"
Oct 30 07:27:44.408293 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:44.408198    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72f1915d-8b19-4459-8450-436b4c30ba38-model-cache\") pod \"auth-test-model-932c00-kserve-5496946777-b8b4x\" (UID: \"72f1915d-8b19-4459-8450-436b4c30ba38\") " pod="ai-tenant-e2e-shared-a-w2-932c00/auth-test-model-932c00-kserve-5496946777-b8b4x"
Oct 30 07:27:44.408402 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:44.408375    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72f1915d-8b19-4459-8450-436b4c30ba38-dshm\") pod \"auth-test-model-932c00-kserve-5496946777-b8b4x\" (UID: \"72f1915d-8b19-4459-8450-436b4c30ba38\") " pod="ai-tenant-e2e-shared-a-w2-932c00/auth-test-model-932c00-kserve-5496946777-b8b4x"
Oct 30 07:27:44.408489 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:44.408408    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72f1915d-8b19-4459-8450-436b4c30ba38-tls-certs\") pod \"auth-test-model-932c00-kserve-5496946777-b8b4x\" (UID: \"72f1915d-8b19-4459-8450-436b4c30ba38\") " pod="ai-tenant-e2e-shared-a-w2-932c00/auth-test-model-932c00-kserve-5496946777-b8b4x"
Oct 30 07:27:44.408489 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:44.408446    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72f1915d-8b19-4459-8450-436b4c30ba38-tmp-dir\") pod \"auth-test-model-932c00-kserve-5496946777-b8b4x\" (UID: \"72f1915d-8b19-4459-8450-436b4c30ba38\") " pod="ai-tenant-e2e-shared-a-w2-932c00/auth-test-model-932c00-kserve-5496946777-b8b4x"
Oct 30 07:27:44.408606 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:44.408567    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72f1915d-8b19-4459-8450-436b4c30ba38-home\") pod \"auth-test-model-932c00-kserve-5496946777-b8b4x\" (UID: \"72f1915d-8b19-4459-8450-436b4c30ba38\") " pod="ai-tenant-e2e-shared-a-w2-932c00/auth-test-model-932c00-kserve-5496946777-b8b4x"
Oct 30 07:27:44.408696 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:44.408667    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72f1915d-8b19-4459-8450-436b4c30ba38-model-cache\") pod \"auth-test-model-932c00-kserve-5496946777-b8b4x\" (UID: \"72f1915d-8b19-4459-8450-436b4c30ba38\") " pod="ai-tenant-e2e-shared-a-w2-932c00/auth-test-model-932c00-kserve-5496946777-b8b4x"
Oct 30 07:27:44.409093 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:44.409072    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72f1915d-8b19-4459-8450-436b4c30ba38-tmp-dir\") pod \"auth-test-model-932c00-kserve-5496946777-b8b4x\" (UID: \"72f1915d-8b19-4459-8450-436b4c30ba38\") " pod="ai-tenant-e2e-shared-a-w2-932c00/auth-test-model-932c00-kserve-5496946777-b8b4x"
Oct 30 07:27:44.423851 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:44.414515    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72f1915d-8b19-4459-8450-436b4c30ba38-dshm\") pod \"auth-test-model-932c00-kserve-5496946777-b8b4x\" (UID: \"72f1915d-8b19-4459-8450-436b4c30ba38\") " pod="ai-tenant-e2e-shared-a-w2-932c00/auth-test-model-932c00-kserve-5496946777-b8b4x"
Oct 30 07:27:44.423851 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:44.416358    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72f1915d-8b19-4459-8450-436b4c30ba38-tls-certs\") pod \"auth-test-model-932c00-kserve-5496946777-b8b4x\" (UID: \"72f1915d-8b19-4459-8450-436b4c30ba38\") " pod="ai-tenant-e2e-shared-a-w2-932c00/auth-test-model-932c00-kserve-5496946777-b8b4x"
Oct 30 07:27:44.450845 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:44.448557    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxxqq\" (UniqueName: \"kubernetes.io/projected/72f1915d-8b19-4459-8450-436b4c30ba38-kube-api-access-qxxqq\") pod \"auth-test-model-932c00-kserve-5496946777-b8b4x\" (UID: \"72f1915d-8b19-4459-8450-436b4c30ba38\") " pod="ai-tenant-e2e-shared-a-w2-932c00/auth-test-model-932c00-kserve-5496946777-b8b4x"
Oct 30 07:27:44.502842 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:44.489189    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-932c00/auth-test-model-932c00-kserve-5496946777-b8b4x"
Oct 30 07:27:44.899659 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:44.899631    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-932c00/auth-test-model-932c00-kserve-5496946777-b8b4x"]
Oct 30 07:27:44.900073 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:27:44.900049    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod72f1915d_8b19_4459_8450_436b4c30ba38.slice/crio-058dae9ad3a94d88212f2dfa0c2b6030f4398ba8054dcc65940f2a3165a7bb86 WatchSource:0}: Error finding container 058dae9ad3a94d88212f2dfa0c2b6030f4398ba8054dcc65940f2a3165a7bb86: Status 404 returned error can't find the container with id 058dae9ad3a94d88212f2dfa0c2b6030f4398ba8054dcc65940f2a3165a7bb86
Oct 30 07:27:45.504550 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:45.504522    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f84d9e-78664b96c-lswqs"
Oct 30 07:27:45.602976 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:45.602944    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-932c00/auth-test-model-932c00-kserve-5496946777-b8b4x" event={"ID":"72f1915d-8b19-4459-8450-436b4c30ba38","Type":"ContainerStarted","Data":"f2accd87f99618c7f295c4bc73161d4fb2083de40a5fc22369e8daa6173f3f1a"}
Oct 30 07:27:45.603151 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:45.602991    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-932c00/auth-test-model-932c00-kserve-5496946777-b8b4x" event={"ID":"72f1915d-8b19-4459-8450-436b4c30ba38","Type":"ContainerStarted","Data":"058dae9ad3a94d88212f2dfa0c2b6030f4398ba8054dcc65940f2a3165a7bb86"}
Oct 30 07:27:45.649698 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:45.649646    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-932c00/auth-test-model-932c00-kserve-5496946777-b8b4x" podStartSLOduration=1.649633796 podStartE2EDuration="1.649633796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:27:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:27:45.647144156 +0000 UTC m=+1824.256551127" watchObservedRunningTime="2026-10-30 07:27:45.649633796 +0000 UTC m=+1824.259040790"
Oct 30 07:27:46.490093 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:46.490060    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-932c00/auth-test-model-932c00-kserve-5496946777-b8b4x"
Oct 30 07:27:49.204258 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:49.204223    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-265384/auth-test-model-265384-kserve-556c8f946f-7l2tb"
Oct 30 07:27:49.216589 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:49.216566    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-265384/auth-test-model-265384-kserve-556c8f946f-7l2tb"
Oct 30 07:27:49.646049 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:49.646021    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-265384/auth-test-model-265384-kserve-556c8f946f-7l2tb"
Oct 30 07:27:54.490162 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:54.490130    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-932c00/auth-test-model-932c00-kserve-5496946777-b8b4x"
Oct 30 07:27:54.502761 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:54.502732    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-932c00/auth-test-model-932c00-kserve-5496946777-b8b4x"
Oct 30 07:27:54.684769 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:27:54.684743    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-932c00/auth-test-model-932c00-kserve-5496946777-b8b4x"
Oct 30 07:28:01.857325 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:28:01.857273    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:28:03.370675 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:03.370640    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:28:14.816332 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:14.816284    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-06e19a/auth-test-model-06e19a-kserve-6947fb644f-szn9m"]
Oct 30 07:28:14.819505 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:14.819480    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-06e19a/auth-test-model-06e19a-kserve-6947fb644f-szn9m"
Oct 30 07:28:14.841357 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:14.841329    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-06e19a/auth-test-model-06e19a-kserve-6947fb644f-szn9m"]
Oct 30 07:28:14.850452 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:14.850425    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-06e19a\"/\"kube-root-ca.crt\""
Oct 30 07:28:14.854371 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:14.854347    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-06e19a\"/\"default-dockercfg-qkr97\""
Oct 30 07:28:14.854506 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:14.854434    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-06e19a\"/\"openshift-service-ca.crt\""
Oct 30 07:28:14.854506 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:14.854435    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-06e19a\"/\"auth-test-model-06e19a-kserve-self-signed-certs\""
Oct 30 07:28:14.929435 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:14.929404    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8925605-f5fb-43cd-ac97-157c5a5973c9-dshm\") pod \"auth-test-model-06e19a-kserve-6947fb644f-szn9m\" (UID: \"b8925605-f5fb-43cd-ac97-157c5a5973c9\") " pod="ai-tenant-e2e-shared-b-w1-06e19a/auth-test-model-06e19a-kserve-6947fb644f-szn9m"
Oct 30 07:28:14.929638 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:14.929616    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8925605-f5fb-43cd-ac97-157c5a5973c9-model-cache\") pod \"auth-test-model-06e19a-kserve-6947fb644f-szn9m\" (UID: \"b8925605-f5fb-43cd-ac97-157c5a5973c9\") " pod="ai-tenant-e2e-shared-b-w1-06e19a/auth-test-model-06e19a-kserve-6947fb644f-szn9m"
Oct 30 07:28:14.929700 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:14.929653    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djjh4\" (UniqueName: \"kubernetes.io/projected/b8925605-f5fb-43cd-ac97-157c5a5973c9-kube-api-access-djjh4\") pod \"auth-test-model-06e19a-kserve-6947fb644f-szn9m\" (UID: \"b8925605-f5fb-43cd-ac97-157c5a5973c9\") " pod="ai-tenant-e2e-shared-b-w1-06e19a/auth-test-model-06e19a-kserve-6947fb644f-szn9m"
Oct 30 07:28:14.929856 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:14.929834    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8925605-f5fb-43cd-ac97-157c5a5973c9-home\") pod \"auth-test-model-06e19a-kserve-6947fb644f-szn9m\" (UID: \"b8925605-f5fb-43cd-ac97-157c5a5973c9\") " pod="ai-tenant-e2e-shared-b-w1-06e19a/auth-test-model-06e19a-kserve-6947fb644f-szn9m"
Oct 30 07:28:14.929961 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:14.929945    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8925605-f5fb-43cd-ac97-157c5a5973c9-tmp-dir\") pod \"auth-test-model-06e19a-kserve-6947fb644f-szn9m\" (UID: \"b8925605-f5fb-43cd-ac97-157c5a5973c9\") " pod="ai-tenant-e2e-shared-b-w1-06e19a/auth-test-model-06e19a-kserve-6947fb644f-szn9m"
Oct 30 07:28:14.930037 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:14.930015    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8925605-f5fb-43cd-ac97-157c5a5973c9-tls-certs\") pod \"auth-test-model-06e19a-kserve-6947fb644f-szn9m\" (UID: \"b8925605-f5fb-43cd-ac97-157c5a5973c9\") " pod="ai-tenant-e2e-shared-b-w1-06e19a/auth-test-model-06e19a-kserve-6947fb644f-szn9m"
Oct 30 07:28:15.031035 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:15.031004    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8925605-f5fb-43cd-ac97-157c5a5973c9-home\") pod \"auth-test-model-06e19a-kserve-6947fb644f-szn9m\" (UID: \"b8925605-f5fb-43cd-ac97-157c5a5973c9\") " pod="ai-tenant-e2e-shared-b-w1-06e19a/auth-test-model-06e19a-kserve-6947fb644f-szn9m"
Oct 30 07:28:15.031198 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:15.031068    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8925605-f5fb-43cd-ac97-157c5a5973c9-tmp-dir\") pod \"auth-test-model-06e19a-kserve-6947fb644f-szn9m\" (UID: \"b8925605-f5fb-43cd-ac97-157c5a5973c9\") " pod="ai-tenant-e2e-shared-b-w1-06e19a/auth-test-model-06e19a-kserve-6947fb644f-szn9m"
Oct 30 07:28:15.031198 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:15.031085    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8925605-f5fb-43cd-ac97-157c5a5973c9-tls-certs\") pod \"auth-test-model-06e19a-kserve-6947fb644f-szn9m\" (UID: \"b8925605-f5fb-43cd-ac97-157c5a5973c9\") " pod="ai-tenant-e2e-shared-b-w1-06e19a/auth-test-model-06e19a-kserve-6947fb644f-szn9m"
Oct 30 07:28:15.031198 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:15.031147    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8925605-f5fb-43cd-ac97-157c5a5973c9-dshm\") pod \"auth-test-model-06e19a-kserve-6947fb644f-szn9m\" (UID: \"b8925605-f5fb-43cd-ac97-157c5a5973c9\") " pod="ai-tenant-e2e-shared-b-w1-06e19a/auth-test-model-06e19a-kserve-6947fb644f-szn9m"
Oct 30 07:28:15.031345 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:15.031292    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8925605-f5fb-43cd-ac97-157c5a5973c9-model-cache\") pod \"auth-test-model-06e19a-kserve-6947fb644f-szn9m\" (UID: \"b8925605-f5fb-43cd-ac97-157c5a5973c9\") " pod="ai-tenant-e2e-shared-b-w1-06e19a/auth-test-model-06e19a-kserve-6947fb644f-szn9m"
Oct 30 07:28:15.031345 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:15.031319    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djjh4\" (UniqueName: \"kubernetes.io/projected/b8925605-f5fb-43cd-ac97-157c5a5973c9-kube-api-access-djjh4\") pod \"auth-test-model-06e19a-kserve-6947fb644f-szn9m\" (UID: \"b8925605-f5fb-43cd-ac97-157c5a5973c9\") " pod="ai-tenant-e2e-shared-b-w1-06e19a/auth-test-model-06e19a-kserve-6947fb644f-szn9m"
Oct 30 07:28:15.031518 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:15.031488    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8925605-f5fb-43cd-ac97-157c5a5973c9-home\") pod \"auth-test-model-06e19a-kserve-6947fb644f-szn9m\" (UID: \"b8925605-f5fb-43cd-ac97-157c5a5973c9\") " pod="ai-tenant-e2e-shared-b-w1-06e19a/auth-test-model-06e19a-kserve-6947fb644f-szn9m"
Oct 30 07:28:15.031825 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:15.031784    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8925605-f5fb-43cd-ac97-157c5a5973c9-tmp-dir\") pod \"auth-test-model-06e19a-kserve-6947fb644f-szn9m\" (UID: \"b8925605-f5fb-43cd-ac97-157c5a5973c9\") " pod="ai-tenant-e2e-shared-b-w1-06e19a/auth-test-model-06e19a-kserve-6947fb644f-szn9m"
Oct 30 07:28:15.031938 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:15.031800    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8925605-f5fb-43cd-ac97-157c5a5973c9-model-cache\") pod \"auth-test-model-06e19a-kserve-6947fb644f-szn9m\" (UID: \"b8925605-f5fb-43cd-ac97-157c5a5973c9\") " pod="ai-tenant-e2e-shared-b-w1-06e19a/auth-test-model-06e19a-kserve-6947fb644f-szn9m"
Oct 30 07:28:15.034083 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:15.034057    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8925605-f5fb-43cd-ac97-157c5a5973c9-dshm\") pod \"auth-test-model-06e19a-kserve-6947fb644f-szn9m\" (UID: \"b8925605-f5fb-43cd-ac97-157c5a5973c9\") " pod="ai-tenant-e2e-shared-b-w1-06e19a/auth-test-model-06e19a-kserve-6947fb644f-szn9m"
Oct 30 07:28:15.034299 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:15.034281    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8925605-f5fb-43cd-ac97-157c5a5973c9-tls-certs\") pod \"auth-test-model-06e19a-kserve-6947fb644f-szn9m\" (UID: \"b8925605-f5fb-43cd-ac97-157c5a5973c9\") " pod="ai-tenant-e2e-shared-b-w1-06e19a/auth-test-model-06e19a-kserve-6947fb644f-szn9m"
Oct 30 07:28:15.091690 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:15.091609    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djjh4\" (UniqueName: \"kubernetes.io/projected/b8925605-f5fb-43cd-ac97-157c5a5973c9-kube-api-access-djjh4\") pod \"auth-test-model-06e19a-kserve-6947fb644f-szn9m\" (UID: \"b8925605-f5fb-43cd-ac97-157c5a5973c9\") " pod="ai-tenant-e2e-shared-b-w1-06e19a/auth-test-model-06e19a-kserve-6947fb644f-szn9m"
Oct 30 07:28:15.148835 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:15.146024    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-06e19a/auth-test-model-06e19a-kserve-6947fb644f-szn9m"
Oct 30 07:28:15.494676 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:15.494605    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-06e19a/auth-test-model-06e19a-kserve-6947fb644f-szn9m"]
Oct 30 07:28:15.502393 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:28:15.502367    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb8925605_f5fb_43cd_ac97_157c5a5973c9.slice/crio-48c6ab1b00576964317b2a59f38ea730ae739dda42fb3eac2f160ebcef6bd060 WatchSource:0}: Error finding container 48c6ab1b00576964317b2a59f38ea730ae739dda42fb3eac2f160ebcef6bd060: Status 404 returned error can't find the container with id 48c6ab1b00576964317b2a59f38ea730ae739dda42fb3eac2f160ebcef6bd060
Oct 30 07:28:15.883186 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:15.883150    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-06e19a/auth-test-model-06e19a-kserve-6947fb644f-szn9m" event={"ID":"b8925605-f5fb-43cd-ac97-157c5a5973c9","Type":"ContainerStarted","Data":"c768138fecaf83d7177db5e91958dabd201311696c6f20e382ac0af01e99444b"}
Oct 30 07:28:15.883186 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:15.883192    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-06e19a/auth-test-model-06e19a-kserve-6947fb644f-szn9m" event={"ID":"b8925605-f5fb-43cd-ac97-157c5a5973c9","Type":"ContainerStarted","Data":"48c6ab1b00576964317b2a59f38ea730ae739dda42fb3eac2f160ebcef6bd060"}
Oct 30 07:28:15.932626 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:15.932575    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-06e19a/auth-test-model-06e19a-kserve-6947fb644f-szn9m" podStartSLOduration=1.932561106 podStartE2EDuration="1.932561106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:28:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:28:15.930211104 +0000 UTC m=+1854.539618086" watchObservedRunningTime="2026-10-30 07:28:15.932561106 +0000 UTC m=+1854.541968119"
Oct 30 07:28:16.146980 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:16.146876    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-06e19a/auth-test-model-06e19a-kserve-6947fb644f-szn9m"
Oct 30 07:28:25.147443 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:25.147404    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-06e19a/auth-test-model-06e19a-kserve-6947fb644f-szn9m"
Oct 30 07:28:25.160105 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:25.160082    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-06e19a/auth-test-model-06e19a-kserve-6947fb644f-szn9m"
Oct 30 07:28:25.973386 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:25.973358    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-06e19a/auth-test-model-06e19a-kserve-6947fb644f-szn9m"
Oct 30 07:28:27.035225 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:27.035189    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b387aa31-pzrjz"]
Oct 30 07:28:27.048594 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:27.048560    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b387aa31-pzrjz"]
Oct 30 07:28:27.505137 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:27.505045    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0846f9fd-b067-4762-8590-944ecd142425" path="/var/lib/kubelet/pods/0846f9fd-b067-4762-8590-944ecd142425/volumes"
Oct 30 07:28:31.861302 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:28:31.861266    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:28:39.850707 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:39.850664    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:28:47.356440 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:47.356404    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-b5998f/auth-test-model-b5998f-kserve-79b66ddd9c-56qmh"]
Oct 30 07:28:47.360781 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:47.360747    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-b5998f/auth-test-model-b5998f-kserve-79b66ddd9c-56qmh"
Oct 30 07:28:47.382137 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:47.382112    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-b5998f/auth-test-model-b5998f-kserve-79b66ddd9c-56qmh"]
Oct 30 07:28:47.394874 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:47.390262    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-b5998f\"/\"default-dockercfg-9tb9r\""
Oct 30 07:28:47.394874 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:47.390992    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-b5998f\"/\"auth-test-model-b5998f-kserve-self-signed-certs\""
Oct 30 07:28:47.394874 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:47.392209    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-b5998f\"/\"kube-root-ca.crt\""
Oct 30 07:28:47.394874 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:47.392286    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-b5998f\"/\"openshift-service-ca.crt\""
Oct 30 07:28:47.432950 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:47.432924    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a61cd57-5e12-4851-b981-3dd6c7ba0a66-dshm\") pod \"auth-test-model-b5998f-kserve-79b66ddd9c-56qmh\" (UID: \"2a61cd57-5e12-4851-b981-3dd6c7ba0a66\") " pod="ai-tenant-e2e-shared-b-w0-b5998f/auth-test-model-b5998f-kserve-79b66ddd9c-56qmh"
Oct 30 07:28:47.433084 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:47.432980    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a61cd57-5e12-4851-b981-3dd6c7ba0a66-tmp-dir\") pod \"auth-test-model-b5998f-kserve-79b66ddd9c-56qmh\" (UID: \"2a61cd57-5e12-4851-b981-3dd6c7ba0a66\") " pod="ai-tenant-e2e-shared-b-w0-b5998f/auth-test-model-b5998f-kserve-79b66ddd9c-56qmh"
Oct 30 07:28:47.433159 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:47.433135    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a61cd57-5e12-4851-b981-3dd6c7ba0a66-tls-certs\") pod \"auth-test-model-b5998f-kserve-79b66ddd9c-56qmh\" (UID: \"2a61cd57-5e12-4851-b981-3dd6c7ba0a66\") " pod="ai-tenant-e2e-shared-b-w0-b5998f/auth-test-model-b5998f-kserve-79b66ddd9c-56qmh"
Oct 30 07:28:47.433293 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:47.433277    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wbfm\" (UniqueName: \"kubernetes.io/projected/2a61cd57-5e12-4851-b981-3dd6c7ba0a66-kube-api-access-5wbfm\") pod \"auth-test-model-b5998f-kserve-79b66ddd9c-56qmh\" (UID: \"2a61cd57-5e12-4851-b981-3dd6c7ba0a66\") " pod="ai-tenant-e2e-shared-b-w0-b5998f/auth-test-model-b5998f-kserve-79b66ddd9c-56qmh"
Oct 30 07:28:47.433415 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:47.433393    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a61cd57-5e12-4851-b981-3dd6c7ba0a66-home\") pod \"auth-test-model-b5998f-kserve-79b66ddd9c-56qmh\" (UID: \"2a61cd57-5e12-4851-b981-3dd6c7ba0a66\") " pod="ai-tenant-e2e-shared-b-w0-b5998f/auth-test-model-b5998f-kserve-79b66ddd9c-56qmh"
Oct 30 07:28:47.433473 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:47.433457    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a61cd57-5e12-4851-b981-3dd6c7ba0a66-model-cache\") pod \"auth-test-model-b5998f-kserve-79b66ddd9c-56qmh\" (UID: \"2a61cd57-5e12-4851-b981-3dd6c7ba0a66\") " pod="ai-tenant-e2e-shared-b-w0-b5998f/auth-test-model-b5998f-kserve-79b66ddd9c-56qmh"
Oct 30 07:28:47.534883 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:47.534833    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wbfm\" (UniqueName: \"kubernetes.io/projected/2a61cd57-5e12-4851-b981-3dd6c7ba0a66-kube-api-access-5wbfm\") pod \"auth-test-model-b5998f-kserve-79b66ddd9c-56qmh\" (UID: \"2a61cd57-5e12-4851-b981-3dd6c7ba0a66\") " pod="ai-tenant-e2e-shared-b-w0-b5998f/auth-test-model-b5998f-kserve-79b66ddd9c-56qmh"
Oct 30 07:28:47.535077 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:47.534928    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a61cd57-5e12-4851-b981-3dd6c7ba0a66-home\") pod \"auth-test-model-b5998f-kserve-79b66ddd9c-56qmh\" (UID: \"2a61cd57-5e12-4851-b981-3dd6c7ba0a66\") " pod="ai-tenant-e2e-shared-b-w0-b5998f/auth-test-model-b5998f-kserve-79b66ddd9c-56qmh"
Oct 30 07:28:47.535077 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:47.534985    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a61cd57-5e12-4851-b981-3dd6c7ba0a66-model-cache\") pod \"auth-test-model-b5998f-kserve-79b66ddd9c-56qmh\" (UID: \"2a61cd57-5e12-4851-b981-3dd6c7ba0a66\") " pod="ai-tenant-e2e-shared-b-w0-b5998f/auth-test-model-b5998f-kserve-79b66ddd9c-56qmh"
Oct 30 07:28:47.535207 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:47.535133    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a61cd57-5e12-4851-b981-3dd6c7ba0a66-dshm\") pod \"auth-test-model-b5998f-kserve-79b66ddd9c-56qmh\" (UID: \"2a61cd57-5e12-4851-b981-3dd6c7ba0a66\") " pod="ai-tenant-e2e-shared-b-w0-b5998f/auth-test-model-b5998f-kserve-79b66ddd9c-56qmh"
Oct 30 07:28:47.535263 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:47.535204    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a61cd57-5e12-4851-b981-3dd6c7ba0a66-tmp-dir\") pod \"auth-test-model-b5998f-kserve-79b66ddd9c-56qmh\" (UID: \"2a61cd57-5e12-4851-b981-3dd6c7ba0a66\") " pod="ai-tenant-e2e-shared-b-w0-b5998f/auth-test-model-b5998f-kserve-79b66ddd9c-56qmh"
Oct 30 07:28:47.535321 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:47.535284    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a61cd57-5e12-4851-b981-3dd6c7ba0a66-tls-certs\") pod \"auth-test-model-b5998f-kserve-79b66ddd9c-56qmh\" (UID: \"2a61cd57-5e12-4851-b981-3dd6c7ba0a66\") " pod="ai-tenant-e2e-shared-b-w0-b5998f/auth-test-model-b5998f-kserve-79b66ddd9c-56qmh"
Oct 30 07:28:47.535411 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:47.535386    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a61cd57-5e12-4851-b981-3dd6c7ba0a66-home\") pod \"auth-test-model-b5998f-kserve-79b66ddd9c-56qmh\" (UID: \"2a61cd57-5e12-4851-b981-3dd6c7ba0a66\") " pod="ai-tenant-e2e-shared-b-w0-b5998f/auth-test-model-b5998f-kserve-79b66ddd9c-56qmh"
Oct 30 07:28:47.535492 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:47.535465    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a61cd57-5e12-4851-b981-3dd6c7ba0a66-model-cache\") pod \"auth-test-model-b5998f-kserve-79b66ddd9c-56qmh\" (UID: \"2a61cd57-5e12-4851-b981-3dd6c7ba0a66\") " pod="ai-tenant-e2e-shared-b-w0-b5998f/auth-test-model-b5998f-kserve-79b66ddd9c-56qmh"
Oct 30 07:28:47.535758 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:47.535734    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a61cd57-5e12-4851-b981-3dd6c7ba0a66-tmp-dir\") pod \"auth-test-model-b5998f-kserve-79b66ddd9c-56qmh\" (UID: \"2a61cd57-5e12-4851-b981-3dd6c7ba0a66\") " pod="ai-tenant-e2e-shared-b-w0-b5998f/auth-test-model-b5998f-kserve-79b66ddd9c-56qmh"
Oct 30 07:28:47.538133 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:47.538110    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a61cd57-5e12-4851-b981-3dd6c7ba0a66-dshm\") pod \"auth-test-model-b5998f-kserve-79b66ddd9c-56qmh\" (UID: \"2a61cd57-5e12-4851-b981-3dd6c7ba0a66\") " pod="ai-tenant-e2e-shared-b-w0-b5998f/auth-test-model-b5998f-kserve-79b66ddd9c-56qmh"
Oct 30 07:28:47.538409 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:47.538387    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a61cd57-5e12-4851-b981-3dd6c7ba0a66-tls-certs\") pod \"auth-test-model-b5998f-kserve-79b66ddd9c-56qmh\" (UID: \"2a61cd57-5e12-4851-b981-3dd6c7ba0a66\") " pod="ai-tenant-e2e-shared-b-w0-b5998f/auth-test-model-b5998f-kserve-79b66ddd9c-56qmh"
Oct 30 07:28:47.560658 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:47.560631    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wbfm\" (UniqueName: \"kubernetes.io/projected/2a61cd57-5e12-4851-b981-3dd6c7ba0a66-kube-api-access-5wbfm\") pod \"auth-test-model-b5998f-kserve-79b66ddd9c-56qmh\" (UID: \"2a61cd57-5e12-4851-b981-3dd6c7ba0a66\") " pod="ai-tenant-e2e-shared-b-w0-b5998f/auth-test-model-b5998f-kserve-79b66ddd9c-56qmh"
Oct 30 07:28:47.688458 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:47.687656    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-b5998f/auth-test-model-b5998f-kserve-79b66ddd9c-56qmh"
Oct 30 07:28:48.086078 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:48.086050    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-b5998f/auth-test-model-b5998f-kserve-79b66ddd9c-56qmh"]
Oct 30 07:28:48.086633 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:28:48.086605    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a61cd57_5e12_4851_b981_3dd6c7ba0a66.slice/crio-72d5f979daea9da44f4ce0d222a3065658a520af9fa7a89e191d5ec173bd3b74 WatchSource:0}: Error finding container 72d5f979daea9da44f4ce0d222a3065658a520af9fa7a89e191d5ec173bd3b74: Status 404 returned error can't find the container with id 72d5f979daea9da44f4ce0d222a3065658a520af9fa7a89e191d5ec173bd3b74
Oct 30 07:28:48.135569 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:48.135536    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-b5998f/auth-test-model-b5998f-kserve-79b66ddd9c-56qmh" event={"ID":"2a61cd57-5e12-4851-b981-3dd6c7ba0a66","Type":"ContainerStarted","Data":"72d5f979daea9da44f4ce0d222a3065658a520af9fa7a89e191d5ec173bd3b74"}
Oct 30 07:28:49.143629 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:49.143589    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-b5998f/auth-test-model-b5998f-kserve-79b66ddd9c-56qmh" event={"ID":"2a61cd57-5e12-4851-b981-3dd6c7ba0a66","Type":"ContainerStarted","Data":"af99855d52056775fbfebb5e33e0ff272df9d42739a38fb2b3ac5df01bb6a2ad"}
Oct 30 07:28:49.191107 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:49.191056    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-b5998f/auth-test-model-b5998f-kserve-79b66ddd9c-56qmh" podStartSLOduration=2.191042277 podStartE2EDuration="2.191042277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:28:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:28:49.18909638 +0000 UTC m=+1887.798503351" watchObservedRunningTime="2026-10-30 07:28:49.191042277 +0000 UTC m=+1887.800449248"
Oct 30 07:28:49.687998 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:49.687968    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-b5998f/auth-test-model-b5998f-kserve-79b66ddd9c-56qmh"
Oct 30 07:28:52.041136 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:52.041092    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6eee8b4c-f28rj"]
Oct 30 07:28:52.055088 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:52.055051    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6eee8b4c-f28rj"]
Oct 30 07:28:53.504488 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:53.504453    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e7ee039-3a41-4271-8335-9cd1c1796949" path="/var/lib/kubelet/pods/0e7ee039-3a41-4271-8335-9cd1c1796949/volumes"
Oct 30 07:28:57.687850 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:57.687820    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-b5998f/auth-test-model-b5998f-kserve-79b66ddd9c-56qmh"
Oct 30 07:28:57.700741 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:57.700717    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-b5998f/auth-test-model-b5998f-kserve-79b66ddd9c-56qmh"
Oct 30 07:28:58.220497 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:28:58.220220    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-b5998f/auth-test-model-b5998f-kserve-79b66ddd9c-56qmh"
Oct 30 07:29:01.858414 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:29:01.858376    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:29:02.048655 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:02.048622    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-490cf3a0-96jsg"]
Oct 30 07:29:02.061921 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:02.061895    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-490cf3a0-96jsg"]
Oct 30 07:29:03.504650 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:03.504571    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b952e3e-75e2-42f1-9dbc-55b5a46dbc16" path="/var/lib/kubelet/pods/3b952e3e-75e2-42f1-9dbc-55b5a46dbc16/volumes"
Oct 30 07:29:03.598615 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:03.598578    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-f84d9e/explicit-model-f84d9e-kserve-7969668679-gr6n2"]
Oct 30 07:29:03.603483 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:03.603458    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-f84d9e/explicit-model-f84d9e-kserve-7969668679-gr6n2"
Oct 30 07:29:03.619907 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:03.619879    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-f84d9e/explicit-model-f84d9e-kserve-7969668679-gr6n2"]
Oct 30 07:29:03.629594 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:03.629568    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-f84d9e\"/\"openshift-service-ca.crt\""
Oct 30 07:29:03.629753 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:03.629732    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-f84d9e\"/\"kube-root-ca.crt\""
Oct 30 07:29:03.637489 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:03.637449    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-f84d9e\"/\"explicit-model-f84d9e-kserve-self-signed-certs\""
Oct 30 07:29:03.637604 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:03.637572    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-f84d9e\"/\"default-dockercfg-hsdwc\""
Oct 30 07:29:03.694031 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:03.693997    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhkvt\" (UniqueName: \"kubernetes.io/projected/dc345afa-49c1-4bd2-be1d-d3a54a92a949-kube-api-access-rhkvt\") pod \"explicit-model-f84d9e-kserve-7969668679-gr6n2\" (UID: \"dc345afa-49c1-4bd2-be1d-d3a54a92a949\") " pod="ai-tenant-e2e-explicit-f84d9e/explicit-model-f84d9e-kserve-7969668679-gr6n2"
Oct 30 07:29:03.694227 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:03.694049    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc345afa-49c1-4bd2-be1d-d3a54a92a949-dshm\") pod \"explicit-model-f84d9e-kserve-7969668679-gr6n2\" (UID: \"dc345afa-49c1-4bd2-be1d-d3a54a92a949\") " pod="ai-tenant-e2e-explicit-f84d9e/explicit-model-f84d9e-kserve-7969668679-gr6n2"
Oct 30 07:29:03.694227 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:03.694157    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc345afa-49c1-4bd2-be1d-d3a54a92a949-model-cache\") pod \"explicit-model-f84d9e-kserve-7969668679-gr6n2\" (UID: \"dc345afa-49c1-4bd2-be1d-d3a54a92a949\") " pod="ai-tenant-e2e-explicit-f84d9e/explicit-model-f84d9e-kserve-7969668679-gr6n2"
Oct 30 07:29:03.694344 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:03.694226    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc345afa-49c1-4bd2-be1d-d3a54a92a949-tls-certs\") pod \"explicit-model-f84d9e-kserve-7969668679-gr6n2\" (UID: \"dc345afa-49c1-4bd2-be1d-d3a54a92a949\") " pod="ai-tenant-e2e-explicit-f84d9e/explicit-model-f84d9e-kserve-7969668679-gr6n2"
Oct 30 07:29:03.694344 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:03.694306    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc345afa-49c1-4bd2-be1d-d3a54a92a949-tmp-dir\") pod \"explicit-model-f84d9e-kserve-7969668679-gr6n2\" (UID: \"dc345afa-49c1-4bd2-be1d-d3a54a92a949\") " pod="ai-tenant-e2e-explicit-f84d9e/explicit-model-f84d9e-kserve-7969668679-gr6n2"
Oct 30 07:29:03.694496 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:03.694481    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc345afa-49c1-4bd2-be1d-d3a54a92a949-home\") pod \"explicit-model-f84d9e-kserve-7969668679-gr6n2\" (UID: \"dc345afa-49c1-4bd2-be1d-d3a54a92a949\") " pod="ai-tenant-e2e-explicit-f84d9e/explicit-model-f84d9e-kserve-7969668679-gr6n2"
Oct 30 07:29:03.796270 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:03.796233    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc345afa-49c1-4bd2-be1d-d3a54a92a949-model-cache\") pod \"explicit-model-f84d9e-kserve-7969668679-gr6n2\" (UID: \"dc345afa-49c1-4bd2-be1d-d3a54a92a949\") " pod="ai-tenant-e2e-explicit-f84d9e/explicit-model-f84d9e-kserve-7969668679-gr6n2"
Oct 30 07:29:03.796453 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:03.796315    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc345afa-49c1-4bd2-be1d-d3a54a92a949-tls-certs\") pod \"explicit-model-f84d9e-kserve-7969668679-gr6n2\" (UID: \"dc345afa-49c1-4bd2-be1d-d3a54a92a949\") " pod="ai-tenant-e2e-explicit-f84d9e/explicit-model-f84d9e-kserve-7969668679-gr6n2"
Oct 30 07:29:03.796453 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:03.796391    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc345afa-49c1-4bd2-be1d-d3a54a92a949-tmp-dir\") pod \"explicit-model-f84d9e-kserve-7969668679-gr6n2\" (UID: \"dc345afa-49c1-4bd2-be1d-d3a54a92a949\") " pod="ai-tenant-e2e-explicit-f84d9e/explicit-model-f84d9e-kserve-7969668679-gr6n2"
Oct 30 07:29:03.796576 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:03.796562    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc345afa-49c1-4bd2-be1d-d3a54a92a949-home\") pod \"explicit-model-f84d9e-kserve-7969668679-gr6n2\" (UID: \"dc345afa-49c1-4bd2-be1d-d3a54a92a949\") " pod="ai-tenant-e2e-explicit-f84d9e/explicit-model-f84d9e-kserve-7969668679-gr6n2"
Oct 30 07:29:03.796702 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:03.796684    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc345afa-49c1-4bd2-be1d-d3a54a92a949-model-cache\") pod \"explicit-model-f84d9e-kserve-7969668679-gr6n2\" (UID: \"dc345afa-49c1-4bd2-be1d-d3a54a92a949\") " pod="ai-tenant-e2e-explicit-f84d9e/explicit-model-f84d9e-kserve-7969668679-gr6n2"
Oct 30 07:29:03.796800 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:03.796772    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc345afa-49c1-4bd2-be1d-d3a54a92a949-tmp-dir\") pod \"explicit-model-f84d9e-kserve-7969668679-gr6n2\" (UID: \"dc345afa-49c1-4bd2-be1d-d3a54a92a949\") " pod="ai-tenant-e2e-explicit-f84d9e/explicit-model-f84d9e-kserve-7969668679-gr6n2"
Oct 30 07:29:03.796948 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:03.796863    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc345afa-49c1-4bd2-be1d-d3a54a92a949-home\") pod \"explicit-model-f84d9e-kserve-7969668679-gr6n2\" (UID: \"dc345afa-49c1-4bd2-be1d-d3a54a92a949\") " pod="ai-tenant-e2e-explicit-f84d9e/explicit-model-f84d9e-kserve-7969668679-gr6n2"
Oct 30 07:29:03.797080 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:03.797059    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhkvt\" (UniqueName: \"kubernetes.io/projected/dc345afa-49c1-4bd2-be1d-d3a54a92a949-kube-api-access-rhkvt\") pod \"explicit-model-f84d9e-kserve-7969668679-gr6n2\" (UID: \"dc345afa-49c1-4bd2-be1d-d3a54a92a949\") " pod="ai-tenant-e2e-explicit-f84d9e/explicit-model-f84d9e-kserve-7969668679-gr6n2"
Oct 30 07:29:03.797149 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:03.797134    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc345afa-49c1-4bd2-be1d-d3a54a92a949-dshm\") pod \"explicit-model-f84d9e-kserve-7969668679-gr6n2\" (UID: \"dc345afa-49c1-4bd2-be1d-d3a54a92a949\") " pod="ai-tenant-e2e-explicit-f84d9e/explicit-model-f84d9e-kserve-7969668679-gr6n2"
Oct 30 07:29:03.799167 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:03.799143    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc345afa-49c1-4bd2-be1d-d3a54a92a949-tls-certs\") pod \"explicit-model-f84d9e-kserve-7969668679-gr6n2\" (UID: \"dc345afa-49c1-4bd2-be1d-d3a54a92a949\") " pod="ai-tenant-e2e-explicit-f84d9e/explicit-model-f84d9e-kserve-7969668679-gr6n2"
Oct 30 07:29:03.799266 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:03.799227    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc345afa-49c1-4bd2-be1d-d3a54a92a949-dshm\") pod \"explicit-model-f84d9e-kserve-7969668679-gr6n2\" (UID: \"dc345afa-49c1-4bd2-be1d-d3a54a92a949\") " pod="ai-tenant-e2e-explicit-f84d9e/explicit-model-f84d9e-kserve-7969668679-gr6n2"
Oct 30 07:29:03.823199 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:03.823171    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhkvt\" (UniqueName: \"kubernetes.io/projected/dc345afa-49c1-4bd2-be1d-d3a54a92a949-kube-api-access-rhkvt\") pod \"explicit-model-f84d9e-kserve-7969668679-gr6n2\" (UID: \"dc345afa-49c1-4bd2-be1d-d3a54a92a949\") " pod="ai-tenant-e2e-explicit-f84d9e/explicit-model-f84d9e-kserve-7969668679-gr6n2"
Oct 30 07:29:03.918167 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:03.918117    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-f84d9e/explicit-model-f84d9e-kserve-7969668679-gr6n2"
Oct 30 07:29:04.273189 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:29:04.273140    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc345afa_49c1_4bd2_be1d_d3a54a92a949.slice/crio-ce8880c6f094473c2ae70f9d6c06e048ff02435b56b8aa0f06e00484fa895741 WatchSource:0}: Error finding container ce8880c6f094473c2ae70f9d6c06e048ff02435b56b8aa0f06e00484fa895741: Status 404 returned error can't find the container with id ce8880c6f094473c2ae70f9d6c06e048ff02435b56b8aa0f06e00484fa895741
Oct 30 07:29:04.274367 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:04.274339    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-f84d9e/explicit-model-f84d9e-kserve-7969668679-gr6n2"]
Oct 30 07:29:05.262017 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:05.261983    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-f84d9e/explicit-model-f84d9e-kserve-7969668679-gr6n2" event={"ID":"dc345afa-49c1-4bd2-be1d-d3a54a92a949","Type":"ContainerStarted","Data":"ce39eb37471e6ba52257c4dc520461ce2aa11f26f8dfb9e803788f5119c8fb45"}
Oct 30 07:29:05.262504 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:05.262031    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-f84d9e/explicit-model-f84d9e-kserve-7969668679-gr6n2" event={"ID":"dc345afa-49c1-4bd2-be1d-d3a54a92a949","Type":"ContainerStarted","Data":"ce8880c6f094473c2ae70f9d6c06e048ff02435b56b8aa0f06e00484fa895741"}
Oct 30 07:29:05.310609 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:05.310561    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-f84d9e/explicit-model-f84d9e-kserve-7969668679-gr6n2" podStartSLOduration=2.310547068 podStartE2EDuration="2.310547068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:29:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:29:05.307046936 +0000 UTC m=+1903.916453906" watchObservedRunningTime="2026-10-30 07:29:05.310547068 +0000 UTC m=+1903.919954073"
Oct 30 07:29:05.918911 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:05.918873    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-f84d9e/explicit-model-f84d9e-kserve-7969668679-gr6n2"
Oct 30 07:29:09.902120 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:09.902081    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-f84d9e/explicit-model-f84d9e-kserve-7969668679-gr6n2"]
Oct 30 07:29:20.995692 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:20.995659    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f84d9e-78664b96c-lswqs"]
Oct 30 07:29:24.931963 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:24.931921    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-f84d9e/explicit-model-f84d9e-kserve-7969668679-gr6n2" podUID="dc345afa-49c1-4bd2-be1d-d3a54a92a949" containerName="main" containerID="cri-o://ce39eb37471e6ba52257c4dc520461ce2aa11f26f8dfb9e803788f5119c8fb45" gracePeriod=45
Oct 30 07:29:25.216364 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:25.216329    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-f84d9e/explicit-model-f84d9e-kserve-7969668679-gr6n2"
Oct 30 07:29:25.360489 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:25.360456    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc345afa-49c1-4bd2-be1d-d3a54a92a949-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc345afa-49c1-4bd2-be1d-d3a54a92a949-tmp-dir\") pod \"dc345afa-49c1-4bd2-be1d-d3a54a92a949\" (UID: \"dc345afa-49c1-4bd2-be1d-d3a54a92a949\") "
Oct 30 07:29:25.360678 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:25.360599    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc345afa-49c1-4bd2-be1d-d3a54a92a949-home\" (UniqueName: \"kubernetes.io/empty-dir/dc345afa-49c1-4bd2-be1d-d3a54a92a949-home\") pod \"dc345afa-49c1-4bd2-be1d-d3a54a92a949\" (UID: \"dc345afa-49c1-4bd2-be1d-d3a54a92a949\") "
Oct 30 07:29:25.360750 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:25.360723    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc345afa-49c1-4bd2-be1d-d3a54a92a949-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc345afa-49c1-4bd2-be1d-d3a54a92a949-model-cache\") pod \"dc345afa-49c1-4bd2-be1d-d3a54a92a949\" (UID: \"dc345afa-49c1-4bd2-be1d-d3a54a92a949\") "
Oct 30 07:29:25.360822 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:25.360753    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc345afa-49c1-4bd2-be1d-d3a54a92a949-tmp-dir" pod "dc345afa-49c1-4bd2-be1d-d3a54a92a949" (UID: "dc345afa-49c1-4bd2-be1d-d3a54a92a949"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:29:25.360896 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:25.360796    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc345afa-49c1-4bd2-be1d-d3a54a92a949-kube-api-access-rhkvt\" (UniqueName: \"kubernetes.io/projected/dc345afa-49c1-4bd2-be1d-d3a54a92a949-kube-api-access-rhkvt\") pod \"dc345afa-49c1-4bd2-be1d-d3a54a92a949\" (UID: \"dc345afa-49c1-4bd2-be1d-d3a54a92a949\") "
Oct 30 07:29:25.360952 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:25.360894    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc345afa-49c1-4bd2-be1d-d3a54a92a949-home" pod "dc345afa-49c1-4bd2-be1d-d3a54a92a949" (UID: "dc345afa-49c1-4bd2-be1d-d3a54a92a949"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:29:25.361008 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:25.360954    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc345afa-49c1-4bd2-be1d-d3a54a92a949-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc345afa-49c1-4bd2-be1d-d3a54a92a949-dshm\") pod \"dc345afa-49c1-4bd2-be1d-d3a54a92a949\" (UID: \"dc345afa-49c1-4bd2-be1d-d3a54a92a949\") "
Oct 30 07:29:25.361061 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:25.361020    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc345afa-49c1-4bd2-be1d-d3a54a92a949-model-cache" pod "dc345afa-49c1-4bd2-be1d-d3a54a92a949" (UID: "dc345afa-49c1-4bd2-be1d-d3a54a92a949"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:29:25.361125 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:25.361092    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc345afa-49c1-4bd2-be1d-d3a54a92a949-tls-certs\" (UniqueName: \"kubernetes.io/secret/dc345afa-49c1-4bd2-be1d-d3a54a92a949-tls-certs\") pod \"dc345afa-49c1-4bd2-be1d-d3a54a92a949\" (UID: \"dc345afa-49c1-4bd2-be1d-d3a54a92a949\") "
Oct 30 07:29:25.361821 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:25.361788    6224 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc345afa-49c1-4bd2-be1d-d3a54a92a949-tmp-dir\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:29:25.361960 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:25.361830    6224 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc345afa-49c1-4bd2-be1d-d3a54a92a949-home\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:29:25.361960 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:25.361845    6224 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc345afa-49c1-4bd2-be1d-d3a54a92a949-model-cache\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:29:25.363127 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:25.363106    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc345afa-49c1-4bd2-be1d-d3a54a92a949-kube-api-access-rhkvt" pod "dc345afa-49c1-4bd2-be1d-d3a54a92a949" (UID: "dc345afa-49c1-4bd2-be1d-d3a54a92a949"). InnerVolumeSpecName "kube-api-access-rhkvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:29:25.363205 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:25.363157    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc345afa-49c1-4bd2-be1d-d3a54a92a949-tls-certs" pod "dc345afa-49c1-4bd2-be1d-d3a54a92a949" (UID: "dc345afa-49c1-4bd2-be1d-d3a54a92a949"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:29:25.363205 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:25.363162    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc345afa-49c1-4bd2-be1d-d3a54a92a949-dshm" pod "dc345afa-49c1-4bd2-be1d-d3a54a92a949" (UID: "dc345afa-49c1-4bd2-be1d-d3a54a92a949"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:29:25.411885 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:25.411853    6224 generic.go:356] "Generic (PLEG): container finished" podID="dc345afa-49c1-4bd2-be1d-d3a54a92a949" containerID="ce39eb37471e6ba52257c4dc520461ce2aa11f26f8dfb9e803788f5119c8fb45" exitCode=0
Oct 30 07:29:25.412022 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:25.411889    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-f84d9e/explicit-model-f84d9e-kserve-7969668679-gr6n2" event={"ID":"dc345afa-49c1-4bd2-be1d-d3a54a92a949","Type":"ContainerDied","Data":"ce39eb37471e6ba52257c4dc520461ce2aa11f26f8dfb9e803788f5119c8fb45"}
Oct 30 07:29:25.412022 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:25.411924    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-f84d9e/explicit-model-f84d9e-kserve-7969668679-gr6n2"
Oct 30 07:29:25.412022 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:25.411940    6224 scope.go:122] "RemoveContainer" containerID="ce39eb37471e6ba52257c4dc520461ce2aa11f26f8dfb9e803788f5119c8fb45"
Oct 30 07:29:25.412354 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:25.412318    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-f84d9e/explicit-model-f84d9e-kserve-7969668679-gr6n2" event={"ID":"dc345afa-49c1-4bd2-be1d-d3a54a92a949","Type":"ContainerDied","Data":"ce8880c6f094473c2ae70f9d6c06e048ff02435b56b8aa0f06e00484fa895741"}
Oct 30 07:29:25.423471 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:25.423454    6224 scope.go:122] "RemoveContainer" containerID="ce39eb37471e6ba52257c4dc520461ce2aa11f26f8dfb9e803788f5119c8fb45"
Oct 30 07:29:25.423737 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:29:25.423720    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce39eb37471e6ba52257c4dc520461ce2aa11f26f8dfb9e803788f5119c8fb45\": container with ID starting with ce39eb37471e6ba52257c4dc520461ce2aa11f26f8dfb9e803788f5119c8fb45 not found: ID does not exist" containerID="ce39eb37471e6ba52257c4dc520461ce2aa11f26f8dfb9e803788f5119c8fb45"
Oct 30 07:29:25.423783 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:25.423742    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce39eb37471e6ba52257c4dc520461ce2aa11f26f8dfb9e803788f5119c8fb45"} err="failed to get container status \"ce39eb37471e6ba52257c4dc520461ce2aa11f26f8dfb9e803788f5119c8fb45\": rpc error: code = NotFound desc = could not find container \"ce39eb37471e6ba52257c4dc520461ce2aa11f26f8dfb9e803788f5119c8fb45\": container with ID starting with ce39eb37471e6ba52257c4dc520461ce2aa11f26f8dfb9e803788f5119c8fb45 not found: ID does not exist"
Oct 30 07:29:25.463364 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:25.463312    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhkvt\" (UniqueName: \"kubernetes.io/projected/dc345afa-49c1-4bd2-be1d-d3a54a92a949-kube-api-access-rhkvt\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:29:25.463364 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:25.463335    6224 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc345afa-49c1-4bd2-be1d-d3a54a92a949-dshm\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:29:25.463364 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:25.463346    6224 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc345afa-49c1-4bd2-be1d-d3a54a92a949-tls-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:29:25.470995 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:25.470971    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-f84d9e/explicit-model-f84d9e-kserve-7969668679-gr6n2"]
Oct 30 07:29:25.475183 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:25.475164    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-f84d9e/explicit-model-f84d9e-kserve-7969668679-gr6n2"]
Oct 30 07:29:25.504140 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:25.504117    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc345afa-49c1-4bd2-be1d-d3a54a92a949" path="/var/lib/kubelet/pods/dc345afa-49c1-4bd2-be1d-d3a54a92a949/volumes"
Oct 30 07:29:25.996290 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:25.996207    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f84d9e-78664b96c-lswqs" podUID="ec5be7a5-4988-4f68-9e20-68c467379762" containerName="payload-pre-processing" containerID="cri-o://08eed846764f86a908bfe9f10c62efb0b9369e42b45e19fdd911c906a1ae68a1" gracePeriod=25
Oct 30 07:29:26.347310 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:26.346149    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f84d9e-78664b96c-lswqs"
Oct 30 07:29:26.453980 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:26.453935    6224 generic.go:356] "Generic (PLEG): container finished" podID="ec5be7a5-4988-4f68-9e20-68c467379762" containerID="08eed846764f86a908bfe9f10c62efb0b9369e42b45e19fdd911c906a1ae68a1" exitCode=0
Oct 30 07:29:26.454303 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:26.454157    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f84d9e-78664b96c-lswqs"
Oct 30 07:29:26.454744 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:26.454175    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f84d9e-78664b96c-lswqs" event={"ID":"ec5be7a5-4988-4f68-9e20-68c467379762","Type":"ContainerDied","Data":"08eed846764f86a908bfe9f10c62efb0b9369e42b45e19fdd911c906a1ae68a1"}
Oct 30 07:29:26.454866 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:26.454782    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f84d9e-78664b96c-lswqs" event={"ID":"ec5be7a5-4988-4f68-9e20-68c467379762","Type":"ContainerDied","Data":"c1a7729ec407386391c24a416db7ee804299b4ee81b68aa51ec3ab1eb2758f57"}
Oct 30 07:29:26.454866 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:26.454842    6224 scope.go:122] "RemoveContainer" containerID="08eed846764f86a908bfe9f10c62efb0b9369e42b45e19fdd911c906a1ae68a1"
Oct 30 07:29:26.478554 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:26.478519    6224 scope.go:122] "RemoveContainer" containerID="08eed846764f86a908bfe9f10c62efb0b9369e42b45e19fdd911c906a1ae68a1"
Oct 30 07:29:26.478973 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:29:26.478941    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08eed846764f86a908bfe9f10c62efb0b9369e42b45e19fdd911c906a1ae68a1\": container with ID starting with 08eed846764f86a908bfe9f10c62efb0b9369e42b45e19fdd911c906a1ae68a1 not found: ID does not exist" containerID="08eed846764f86a908bfe9f10c62efb0b9369e42b45e19fdd911c906a1ae68a1"
Oct 30 07:29:26.479058 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:26.478985    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08eed846764f86a908bfe9f10c62efb0b9369e42b45e19fdd911c906a1ae68a1"} err="failed to get container status \"08eed846764f86a908bfe9f10c62efb0b9369e42b45e19fdd911c906a1ae68a1\": rpc error: code = NotFound desc = could not find container \"08eed846764f86a908bfe9f10c62efb0b9369e42b45e19fdd911c906a1ae68a1\": container with ID starting with 08eed846764f86a908bfe9f10c62efb0b9369e42b45e19fdd911c906a1ae68a1 not found: ID does not exist"
Oct 30 07:29:26.482450 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:26.482427    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec5be7a5-4988-4f68-9e20-68c467379762-kube-api-access-47fp5\" (UniqueName: \"kubernetes.io/projected/ec5be7a5-4988-4f68-9e20-68c467379762-kube-api-access-47fp5\") pod \"ec5be7a5-4988-4f68-9e20-68c467379762\" (UID: \"ec5be7a5-4988-4f68-9e20-68c467379762\") "
Oct 30 07:29:26.482730 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:26.482713    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec5be7a5-4988-4f68-9e20-68c467379762-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec5be7a5-4988-4f68-9e20-68c467379762-plugins-config-volume\") pod \"ec5be7a5-4988-4f68-9e20-68c467379762\" (UID: \"ec5be7a5-4988-4f68-9e20-68c467379762\") "
Oct 30 07:29:26.483113 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:26.483093    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec5be7a5-4988-4f68-9e20-68c467379762-plugins-config-volume" pod "ec5be7a5-4988-4f68-9e20-68c467379762" (UID: "ec5be7a5-4988-4f68-9e20-68c467379762"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:29:26.483265 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:26.483251    6224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec5be7a5-4988-4f68-9e20-68c467379762-plugins-config-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:29:26.484570 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:26.484547    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec5be7a5-4988-4f68-9e20-68c467379762-kube-api-access-47fp5" pod "ec5be7a5-4988-4f68-9e20-68c467379762" (UID: "ec5be7a5-4988-4f68-9e20-68c467379762"). InnerVolumeSpecName "kube-api-access-47fp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:29:26.584604 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:26.584533    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47fp5\" (UniqueName: \"kubernetes.io/projected/ec5be7a5-4988-4f68-9e20-68c467379762-kube-api-access-47fp5\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:29:26.805579 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:26.805547    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f84d9e-78664b96c-lswqs"]
Oct 30 07:29:26.819325 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:26.819296    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f84d9e-78664b96c-lswqs"]
Oct 30 07:29:27.504726 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:27.504691    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec5be7a5-4988-4f68-9e20-68c467379762" path="/var/lib/kubelet/pods/ec5be7a5-4988-4f68-9e20-68c467379762/volumes"
Oct 30 07:29:31.415275 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:31.415237    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:29:31.876266 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:29:31.876221    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:29:41.462563 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:41.462525    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qbltv"]
Oct 30 07:29:41.463115 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:41.462714    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qbltv" podUID="7177df4a-a897-4dc3-b71b-c70431bb12fd" containerName="istio-proxy" containerID="cri-o://5e1f4df81215c3b4bad30111114830b75524755b468a49e32f71ff036375e93e" gracePeriod=30
Oct 30 07:29:46.631281 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:46.631244    6224 generic.go:356] "Generic (PLEG): container finished" podID="7177df4a-a897-4dc3-b71b-c70431bb12fd" containerID="5e1f4df81215c3b4bad30111114830b75524755b468a49e32f71ff036375e93e" exitCode=0
Oct 30 07:29:46.631752 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:46.631324    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qbltv" event={"ID":"7177df4a-a897-4dc3-b71b-c70431bb12fd","Type":"ContainerDied","Data":"5e1f4df81215c3b4bad30111114830b75524755b468a49e32f71ff036375e93e"}
Oct 30 07:29:46.735531 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:46.735474    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qbltv"
Oct 30 07:29:46.798217 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:46.798172    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7177df4a-a897-4dc3-b71b-c70431bb12fd-istio-token\" (UniqueName: \"kubernetes.io/projected/7177df4a-a897-4dc3-b71b-c70431bb12fd-istio-token\") pod \"7177df4a-a897-4dc3-b71b-c70431bb12fd\" (UID: \"7177df4a-a897-4dc3-b71b-c70431bb12fd\") "
Oct 30 07:29:46.798379 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:46.798276    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7177df4a-a897-4dc3-b71b-c70431bb12fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7177df4a-a897-4dc3-b71b-c70431bb12fd-workload-socket\") pod \"7177df4a-a897-4dc3-b71b-c70431bb12fd\" (UID: \"7177df4a-a897-4dc3-b71b-c70431bb12fd\") "
Oct 30 07:29:46.798379 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:46.798317    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7177df4a-a897-4dc3-b71b-c70431bb12fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7177df4a-a897-4dc3-b71b-c70431bb12fd-istiod-ca-cert\") pod \"7177df4a-a897-4dc3-b71b-c70431bb12fd\" (UID: \"7177df4a-a897-4dc3-b71b-c70431bb12fd\") "
Oct 30 07:29:46.798379 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:46.798349    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7177df4a-a897-4dc3-b71b-c70431bb12fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7177df4a-a897-4dc3-b71b-c70431bb12fd-istio-podinfo\") pod \"7177df4a-a897-4dc3-b71b-c70431bb12fd\" (UID: \"7177df4a-a897-4dc3-b71b-c70431bb12fd\") "
Oct 30 07:29:46.798549 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:46.798411    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7177df4a-a897-4dc3-b71b-c70431bb12fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7177df4a-a897-4dc3-b71b-c70431bb12fd-istio-data\") pod \"7177df4a-a897-4dc3-b71b-c70431bb12fd\" (UID: \"7177df4a-a897-4dc3-b71b-c70431bb12fd\") "
Oct 30 07:29:46.798605 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:46.798589    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7177df4a-a897-4dc3-b71b-c70431bb12fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7177df4a-a897-4dc3-b71b-c70431bb12fd-credential-socket\") pod \"7177df4a-a897-4dc3-b71b-c70431bb12fd\" (UID: \"7177df4a-a897-4dc3-b71b-c70431bb12fd\") "
Oct 30 07:29:46.798664 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:46.798648    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7177df4a-a897-4dc3-b71b-c70431bb12fd-kube-api-access-6t98c\" (UniqueName: \"kubernetes.io/projected/7177df4a-a897-4dc3-b71b-c70431bb12fd-kube-api-access-6t98c\") pod \"7177df4a-a897-4dc3-b71b-c70431bb12fd\" (UID: \"7177df4a-a897-4dc3-b71b-c70431bb12fd\") "
Oct 30 07:29:46.798721 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:46.798709    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7177df4a-a897-4dc3-b71b-c70431bb12fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7177df4a-a897-4dc3-b71b-c70431bb12fd-workload-certs\") pod \"7177df4a-a897-4dc3-b71b-c70431bb12fd\" (UID: \"7177df4a-a897-4dc3-b71b-c70431bb12fd\") "
Oct 30 07:29:46.798779 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:46.798756    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7177df4a-a897-4dc3-b71b-c70431bb12fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7177df4a-a897-4dc3-b71b-c70431bb12fd-istio-envoy\") pod \"7177df4a-a897-4dc3-b71b-c70431bb12fd\" (UID: \"7177df4a-a897-4dc3-b71b-c70431bb12fd\") "
Oct 30 07:29:46.798947 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:46.798910    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7177df4a-a897-4dc3-b71b-c70431bb12fd-istiod-ca-cert" pod "7177df4a-a897-4dc3-b71b-c70431bb12fd" (UID: "7177df4a-a897-4dc3-b71b-c70431bb12fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:29:46.799151 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:46.799132    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7177df4a-a897-4dc3-b71b-c70431bb12fd-workload-socket" pod "7177df4a-a897-4dc3-b71b-c70431bb12fd" (UID: "7177df4a-a897-4dc3-b71b-c70431bb12fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:29:46.799473 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:46.799453    6224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7177df4a-a897-4dc3-b71b-c70431bb12fd-workload-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:29:46.799473 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:46.799470    6224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7177df4a-a897-4dc3-b71b-c70431bb12fd-istiod-ca-cert\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:29:46.799640 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:46.799628    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7177df4a-a897-4dc3-b71b-c70431bb12fd-credential-socket" pod "7177df4a-a897-4dc3-b71b-c70431bb12fd" (UID: "7177df4a-a897-4dc3-b71b-c70431bb12fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:29:46.800156 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:46.800113    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7177df4a-a897-4dc3-b71b-c70431bb12fd-workload-certs" pod "7177df4a-a897-4dc3-b71b-c70431bb12fd" (UID: "7177df4a-a897-4dc3-b71b-c70431bb12fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:29:46.800156 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:46.800129    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7177df4a-a897-4dc3-b71b-c70431bb12fd-istio-data" pod "7177df4a-a897-4dc3-b71b-c70431bb12fd" (UID: "7177df4a-a897-4dc3-b71b-c70431bb12fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:29:46.802042 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:46.801979    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7177df4a-a897-4dc3-b71b-c70431bb12fd-istio-envoy" pod "7177df4a-a897-4dc3-b71b-c70431bb12fd" (UID: "7177df4a-a897-4dc3-b71b-c70431bb12fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:29:46.802145 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:46.802104    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7177df4a-a897-4dc3-b71b-c70431bb12fd-istio-podinfo" pod "7177df4a-a897-4dc3-b71b-c70431bb12fd" (UID: "7177df4a-a897-4dc3-b71b-c70431bb12fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:29:46.802501 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:46.802470    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7177df4a-a897-4dc3-b71b-c70431bb12fd-istio-token" pod "7177df4a-a897-4dc3-b71b-c70431bb12fd" (UID: "7177df4a-a897-4dc3-b71b-c70431bb12fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:29:46.808943 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:46.808916    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7177df4a-a897-4dc3-b71b-c70431bb12fd-kube-api-access-6t98c" pod "7177df4a-a897-4dc3-b71b-c70431bb12fd" (UID: "7177df4a-a897-4dc3-b71b-c70431bb12fd"). InnerVolumeSpecName "kube-api-access-6t98c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:29:46.900415 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:46.900379    6224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7177df4a-a897-4dc3-b71b-c70431bb12fd-credential-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:29:46.900415 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:46.900410    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6t98c\" (UniqueName: \"kubernetes.io/projected/7177df4a-a897-4dc3-b71b-c70431bb12fd-kube-api-access-6t98c\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:29:46.900415 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:46.900420    6224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7177df4a-a897-4dc3-b71b-c70431bb12fd-workload-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:29:46.900645 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:46.900431    6224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7177df4a-a897-4dc3-b71b-c70431bb12fd-istio-envoy\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:29:46.900645 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:46.900439    6224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7177df4a-a897-4dc3-b71b-c70431bb12fd-istio-token\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:29:46.900645 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:46.900447    6224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7177df4a-a897-4dc3-b71b-c70431bb12fd-istio-podinfo\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:29:46.900645 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:46.900455    6224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7177df4a-a897-4dc3-b71b-c70431bb12fd-istio-data\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:29:47.800610 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:47.790653    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qbltv" event={"ID":"7177df4a-a897-4dc3-b71b-c70431bb12fd","Type":"ContainerDied","Data":"351d714d6a34fced82e646937a8f3a6c43a43c1fe5aef5e283928168e90fb8c5"}
Oct 30 07:29:47.800610 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:47.790723    6224 scope.go:122] "RemoveContainer" containerID="5e1f4df81215c3b4bad30111114830b75524755b468a49e32f71ff036375e93e"
Oct 30 07:29:47.800610 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:47.790950    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qbltv"
Oct 30 07:29:47.866984 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:47.866955    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qbltv"]
Oct 30 07:29:47.879789 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:47.879764    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qbltv"]
Oct 30 07:29:49.504866 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:29:49.504828    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7177df4a-a897-4dc3-b71b-c70431bb12fd" path="/var/lib/kubelet/pods/7177df4a-a897-4dc3-b71b-c70431bb12fd/volumes"
Oct 30 07:30:00.193760 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:00.193724    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889090-jn62g"]
Oct 30 07:30:00.194619 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:00.194596    6224 cpu_manager.go:414] "containerMap: removing container" podUID="ec5be7a5-4988-4f68-9e20-68c467379762" containerName="payload-pre-processing"
Oct 30 07:30:00.194733 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:00.194620    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec5be7a5-4988-4f68-9e20-68c467379762" containerName="payload-pre-processing"
Oct 30 07:30:00.194733 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:00.194683    6224 cpu_manager.go:414] "containerMap: removing container" podUID="dc345afa-49c1-4bd2-be1d-d3a54a92a949" containerName="main"
Oct 30 07:30:00.194733 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:00.194691    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc345afa-49c1-4bd2-be1d-d3a54a92a949" containerName="main"
Oct 30 07:30:00.194733 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:00.194723    6224 cpu_manager.go:414] "containerMap: removing container" podUID="7177df4a-a897-4dc3-b71b-c70431bb12fd" containerName="istio-proxy"
Oct 30 07:30:00.194733 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:00.194730    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7177df4a-a897-4dc3-b71b-c70431bb12fd" containerName="istio-proxy"
Oct 30 07:30:00.195003 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:00.194926    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="7177df4a-a897-4dc3-b71b-c70431bb12fd" containerName="istio-proxy"
Oct 30 07:30:00.195003 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:00.194946    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec5be7a5-4988-4f68-9e20-68c467379762" containerName="payload-pre-processing"
Oct 30 07:30:00.195003 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:00.194964    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc345afa-49c1-4bd2-be1d-d3a54a92a949" containerName="main"
Oct 30 07:30:00.199979 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:00.199958    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889090-jn62g"
Oct 30 07:30:00.225523 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:00.225491    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889090-fzvfn"]
Oct 30 07:30:00.233097 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:00.233069    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889090-fzvfn"
Oct 30 07:30:00.237255 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:00.237227    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889090-jn62g"]
Oct 30 07:30:00.251484 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:00.251460    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889090-9ccls"]
Oct 30 07:30:00.256417 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:00.256397    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889090-28b2b"]
Oct 30 07:30:00.256529 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:00.256498    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889090-9ccls"
Oct 30 07:30:00.260903 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:00.260884    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889090-28b2b"
Oct 30 07:30:00.278387 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:00.278364    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbwtz\" (UniqueName: \"kubernetes.io/projected/d87fa68a-a7e0-41c1-92c3-65c1faec9821-kube-api-access-dbwtz\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889090-jn62g\" (UID: \"d87fa68a-a7e0-41c1-92c3-65c1faec9821\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889090-jn62g"
Oct 30 07:30:00.278458 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:00.278403    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b67837f-7acb-4ec6-86a5-dbe1263b46e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889090-fzvfn\" (UID: \"7b67837f-7acb-4ec6-86a5-dbe1263b46e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889090-fzvfn"
Oct 30 07:30:00.278518 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:00.278501    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-927tk\" (UniqueName: \"kubernetes.io/projected/7b67837f-7acb-4ec6-86a5-dbe1263b46e2-kube-api-access-927tk\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889090-fzvfn\" (UID: \"7b67837f-7acb-4ec6-86a5-dbe1263b46e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889090-fzvfn"
Oct 30 07:30:00.278596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:00.278582    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d87fa68a-a7e0-41c1-92c3-65c1faec9821-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889090-jn62g\" (UID: \"d87fa68a-a7e0-41c1-92c3-65c1faec9821\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889090-jn62g"
Oct 30 07:30:00.286074 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:00.286028    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889090-6fr4m"]
Oct 30 07:30:00.290783 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:00.290762    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889090-6fr4m"
Oct 30 07:30:00.295593 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:00.295566    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:30:00.304174 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:00.304152    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889090-9ccls"]
Oct 30 07:30:00.310019 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:00.309993    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889090-fzvfn"]
Oct 30 07:30:00.320889 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:00.320856    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889090-6fr4m"]
Oct 30 07:30:00.324930 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:00.324906    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889090-28b2b"]
Oct 30 07:30:00.380267 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:00.380238    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-927tk\" (UniqueName: \"kubernetes.io/projected/7b67837f-7acb-4ec6-86a5-dbe1263b46e2-kube-api-access-927tk\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889090-fzvfn\" (UID: \"7b67837f-7acb-4ec6-86a5-dbe1263b46e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889090-fzvfn"
Oct 30 07:30:00.380412 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:00.380297    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plm8k\" (UniqueName: \"kubernetes.io/projected/59474a91-6ad5-4a1b-9220-30356c14f255-kube-api-access-plm8k\") pod \"maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889090-28b2b\" (UID: \"59474a91-6ad5-4a1b-9220-30356c14f255\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889090-28b2b"
Oct 30 07:30:00.380412 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:00.380338    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d87fa68a-a7e0-41c1-92c3-65c1faec9821-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889090-jn62g\" (UID: \"d87fa68a-a7e0-41c1-92c3-65c1faec9821\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889090-jn62g"
Oct 30 07:30:00.380412 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:00.380357    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94c9d1b4-2f77-4e91-aa74-8bbb2bc04713-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889090-6fr4m\" (UID: \"94c9d1b4-2f77-4e91-aa74-8bbb2bc04713\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889090-6fr4m"
Oct 30 07:30:00.380528 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:00.380465    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59474a91-6ad5-4a1b-9220-30356c14f255-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889090-28b2b\" (UID: \"59474a91-6ad5-4a1b-9220-30356c14f255\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889090-28b2b"
Oct 30 07:30:00.380566 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:00.380539    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rq5f\" (UniqueName: \"kubernetes.io/projected/cd765ec1-29ac-4173-9a95-17099c37379a-kube-api-access-6rq5f\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-265384-29889090-9ccls\" (UID: \"cd765ec1-29ac-4173-9a95-17099c37379a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889090-9ccls"
Oct 30 07:30:00.380610 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:00.380578    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd765ec1-29ac-4173-9a95-17099c37379a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-265384-29889090-9ccls\" (UID: \"cd765ec1-29ac-4173-9a95-17099c37379a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889090-9ccls"
Oct 30 07:30:00.380716 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:00.380703    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfhtb\" (UniqueName: \"kubernetes.io/projected/94c9d1b4-2f77-4e91-aa74-8bbb2bc04713-kube-api-access-hfhtb\") pod \"maas-api-key-cleanup-29889090-6fr4m\" (UID: \"94c9d1b4-2f77-4e91-aa74-8bbb2bc04713\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889090-6fr4m"
Oct 30 07:30:00.380752 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:00.380736    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbwtz\" (UniqueName: \"kubernetes.io/projected/d87fa68a-a7e0-41c1-92c3-65c1faec9821-kube-api-access-dbwtz\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889090-jn62g\" (UID: \"d87fa68a-a7e0-41c1-92c3-65c1faec9821\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889090-jn62g"
Oct 30 07:30:00.380797 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:00.380755    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b67837f-7acb-4ec6-86a5-dbe1263b46e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889090-fzvfn\" (UID: \"7b67837f-7acb-4ec6-86a5-dbe1263b46e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889090-fzvfn"
Oct 30 07:30:00.381051 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:00.381030    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d87fa68a-a7e0-41c1-92c3-65c1faec9821-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889090-jn62g\" (UID: \"d87fa68a-a7e0-41c1-92c3-65c1faec9821\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889090-jn62g"
Oct 30 07:30:00.381379 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:00.381361    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b67837f-7acb-4ec6-86a5-dbe1263b46e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889090-fzvfn\" (UID: \"7b67837f-7acb-4ec6-86a5-dbe1263b46e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889090-fzvfn"
Oct 30 07:30:00.406943 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:00.406920    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-927tk\" (UniqueName: \"kubernetes.io/projected/7b67837f-7acb-4ec6-86a5-dbe1263b46e2-kube-api-access-927tk\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889090-fzvfn\" (UID: \"7b67837f-7acb-4ec6-86a5-dbe1263b46e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889090-fzvfn"
Oct 30 07:30:00.407039 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:00.406950    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbwtz\" (UniqueName: \"kubernetes.io/projected/d87fa68a-a7e0-41c1-92c3-65c1faec9821-kube-api-access-dbwtz\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889090-jn62g\" (UID: \"d87fa68a-a7e0-41c1-92c3-65c1faec9821\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889090-jn62g"
Oct 30 07:30:00.486405 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:00.484206    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94c9d1b4-2f77-4e91-aa74-8bbb2bc04713-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889090-6fr4m\" (UID: \"94c9d1b4-2f77-4e91-aa74-8bbb2bc04713\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889090-6fr4m"
Oct 30 07:30:00.486405 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:00.484333    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59474a91-6ad5-4a1b-9220-30356c14f255-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889090-28b2b\" (UID: \"59474a91-6ad5-4a1b-9220-30356c14f255\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889090-28b2b"
Oct 30 07:30:00.486405 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:00.484388    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rq5f\" (UniqueName: \"kubernetes.io/projected/cd765ec1-29ac-4173-9a95-17099c37379a-kube-api-access-6rq5f\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-265384-29889090-9ccls\" (UID: \"cd765ec1-29ac-4173-9a95-17099c37379a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889090-9ccls"
Oct 30 07:30:00.486405 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:00.484420    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd765ec1-29ac-4173-9a95-17099c37379a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-265384-29889090-9ccls\" (UID: \"cd765ec1-29ac-4173-9a95-17099c37379a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889090-9ccls"
Oct 30 07:30:00.486405 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:00.484580    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfhtb\" (UniqueName: \"kubernetes.io/projected/94c9d1b4-2f77-4e91-aa74-8bbb2bc04713-kube-api-access-hfhtb\") pod \"maas-api-key-cleanup-29889090-6fr4m\" (UID: \"94c9d1b4-2f77-4e91-aa74-8bbb2bc04713\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889090-6fr4m"
Oct 30 07:30:00.486405 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:00.484776    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plm8k\" (UniqueName: \"kubernetes.io/projected/59474a91-6ad5-4a1b-9220-30356c14f255-kube-api-access-plm8k\") pod \"maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889090-28b2b\" (UID: \"59474a91-6ad5-4a1b-9220-30356c14f255\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889090-28b2b"
Oct 30 07:30:00.486405 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:00.485979    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94c9d1b4-2f77-4e91-aa74-8bbb2bc04713-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889090-6fr4m\" (UID: \"94c9d1b4-2f77-4e91-aa74-8bbb2bc04713\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889090-6fr4m"
Oct 30 07:30:00.486934 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:00.486752    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59474a91-6ad5-4a1b-9220-30356c14f255-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889090-28b2b\" (UID: \"59474a91-6ad5-4a1b-9220-30356c14f255\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889090-28b2b"
Oct 30 07:30:00.487791 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:00.487763    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd765ec1-29ac-4173-9a95-17099c37379a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-265384-29889090-9ccls\" (UID: \"cd765ec1-29ac-4173-9a95-17099c37379a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889090-9ccls"
Oct 30 07:30:00.514714 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:00.514665    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rq5f\" (UniqueName: \"kubernetes.io/projected/cd765ec1-29ac-4173-9a95-17099c37379a-kube-api-access-6rq5f\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-265384-29889090-9ccls\" (UID: \"cd765ec1-29ac-4173-9a95-17099c37379a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889090-9ccls"
Oct 30 07:30:00.516936 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:00.516908    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfhtb\" (UniqueName: \"kubernetes.io/projected/94c9d1b4-2f77-4e91-aa74-8bbb2bc04713-kube-api-access-hfhtb\") pod \"maas-api-key-cleanup-29889090-6fr4m\" (UID: \"94c9d1b4-2f77-4e91-aa74-8bbb2bc04713\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889090-6fr4m"
Oct 30 07:30:00.517062 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:00.516917    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plm8k\" (UniqueName: \"kubernetes.io/projected/59474a91-6ad5-4a1b-9220-30356c14f255-kube-api-access-plm8k\") pod \"maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889090-28b2b\" (UID: \"59474a91-6ad5-4a1b-9220-30356c14f255\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889090-28b2b"
Oct 30 07:30:00.527779 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:00.527748    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889090-jn62g"
Oct 30 07:30:00.554751 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:00.554722    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889090-fzvfn"
Oct 30 07:30:00.577352 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:00.577233    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889090-9ccls"
Oct 30 07:30:00.589131 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:00.589096    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889090-28b2b"
Oct 30 07:30:00.623604 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:00.623071    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889090-6fr4m"
Oct 30 07:30:00.847731 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:30:00.847697    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd87fa68a_a7e0_41c1_92c3_65c1faec9821.slice/crio-f08a54aa589bca0f93e643c2d48c137981d7cd8818510c97105011d503fe8295 WatchSource:0}: Error finding container f08a54aa589bca0f93e643c2d48c137981d7cd8818510c97105011d503fe8295: Status 404 returned error can't find the container with id f08a54aa589bca0f93e643c2d48c137981d7cd8818510c97105011d503fe8295
Oct 30 07:30:00.848994 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:00.848970    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889090-jn62g"]
Oct 30 07:30:00.895064 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:00.895023    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889090-fzvfn"]
Oct 30 07:30:00.924233 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:00.924195    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889090-fzvfn" event={"ID":"7b67837f-7acb-4ec6-86a5-dbe1263b46e2","Type":"ContainerStarted","Data":"6e21d6a00488afbf58e0fc7b4c36b9147f29d3b27c07651f3216121ad2a41d10"}
Oct 30 07:30:00.925582 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:00.925557    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889090-jn62g" event={"ID":"d87fa68a-a7e0-41c1-92c3-65c1faec9821","Type":"ContainerStarted","Data":"f08a54aa589bca0f93e643c2d48c137981d7cd8818510c97105011d503fe8295"}
Oct 30 07:30:00.945454 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:30:00.945420    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd765ec1_29ac_4173_9a95_17099c37379a.slice/crio-e4dbbae415c2f78095d2850a5f38e5c9b0613439803014cd7683a8e7140ba671 WatchSource:0}: Error finding container e4dbbae415c2f78095d2850a5f38e5c9b0613439803014cd7683a8e7140ba671: Status 404 returned error can't find the container with id e4dbbae415c2f78095d2850a5f38e5c9b0613439803014cd7683a8e7140ba671
Oct 30 07:30:00.947626 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:00.947602    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889090-9ccls"]
Oct 30 07:30:00.956818 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:00.956779    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889090-28b2b"]
Oct 30 07:30:01.006538 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:01.006515    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889090-6fr4m"]
Oct 30 07:30:01.069252 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:30:01.068383    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94c9d1b4_2f77_4e91_aa74_8bbb2bc04713.slice/crio-c4486f98b5e4d5983c5273a4f6a12ca6d976f1a318cbf627de6adf04cf29e63b WatchSource:0}: Error finding container c4486f98b5e4d5983c5273a4f6a12ca6d976f1a318cbf627de6adf04cf29e63b: Status 404 returned error can't find the container with id c4486f98b5e4d5983c5273a4f6a12ca6d976f1a318cbf627de6adf04cf29e63b
Oct 30 07:30:01.860955 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:30:01.860913    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:30:01.934125 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:01.934035    6224 generic.go:356] "Generic (PLEG): container finished" podID="7b67837f-7acb-4ec6-86a5-dbe1263b46e2" containerID="b43195bfeb2bb8057947f2035711430be1e6e4f89a0d75875c4d4af4e9ab3461" exitCode=60
Oct 30 07:30:01.934289 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:01.934113    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889090-fzvfn" event={"ID":"7b67837f-7acb-4ec6-86a5-dbe1263b46e2","Type":"ContainerDied","Data":"b43195bfeb2bb8057947f2035711430be1e6e4f89a0d75875c4d4af4e9ab3461"}
Oct 30 07:30:01.934436 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:01.934418    6224 scope.go:122] "RemoveContainer" containerID="b43195bfeb2bb8057947f2035711430be1e6e4f89a0d75875c4d4af4e9ab3461"
Oct 30 07:30:01.935818 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:01.935783    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889090-28b2b" event={"ID":"59474a91-6ad5-4a1b-9220-30356c14f255","Type":"ContainerStarted","Data":"4d4a4da6a1bc6863152cec647db415105c19a5f094dbe248df503052ce728669"}
Oct 30 07:30:01.935929 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:01.935854    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889090-28b2b" event={"ID":"59474a91-6ad5-4a1b-9220-30356c14f255","Type":"ContainerStarted","Data":"654d6f4c1e2e8c4b13cdd69624af75c856b6e4d3e5baabf8838eee74b7376674"}
Oct 30 07:30:01.937907 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:01.937878    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889090-jn62g" event={"ID":"d87fa68a-a7e0-41c1-92c3-65c1faec9821","Type":"ContainerStarted","Data":"5c910463f311bcf9003119aba6716798f9acfc37056b8096213763e440f198cc"}
Oct 30 07:30:01.939419 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:01.939395    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889090-6fr4m" event={"ID":"94c9d1b4-2f77-4e91-aa74-8bbb2bc04713","Type":"ContainerStarted","Data":"484474c092c0dec64b786f026716045c2e6fa013abe009e15cf30fb8b15c32ec"}
Oct 30 07:30:01.939523 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:01.939436    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889090-6fr4m" event={"ID":"94c9d1b4-2f77-4e91-aa74-8bbb2bc04713","Type":"ContainerStarted","Data":"c4486f98b5e4d5983c5273a4f6a12ca6d976f1a318cbf627de6adf04cf29e63b"}
Oct 30 07:30:01.941112 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:01.941083    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889090-9ccls" event={"ID":"cd765ec1-29ac-4173-9a95-17099c37379a","Type":"ContainerStarted","Data":"d5157cf45160241602e0ea8ff20ff8538073fbc6c8443288ecb43ae9b4698e54"}
Oct 30 07:30:01.941224 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:01.941139    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889090-9ccls" event={"ID":"cd765ec1-29ac-4173-9a95-17099c37379a","Type":"ContainerStarted","Data":"e4dbbae415c2f78095d2850a5f38e5c9b0613439803014cd7683a8e7140ba671"}
Oct 30 07:30:02.027292 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:02.027234    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889090-28b2b" podStartSLOduration=2.027221766 podStartE2EDuration="2.027221766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:30:02.025864738 +0000 UTC m=+1960.635271707" watchObservedRunningTime="2026-10-30 07:30:02.027221766 +0000 UTC m=+1960.636628766"
Oct 30 07:30:02.053771 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:02.053741    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a71a224-wsc72"]
Oct 30 07:30:02.068839 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:02.068795    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a71a224-wsc72"]
Oct 30 07:30:02.078433 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:02.078382    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889090-jn62g" podStartSLOduration=2.07836811 podStartE2EDuration="2.07836811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:30:02.076221707 +0000 UTC m=+1960.685628678" watchObservedRunningTime="2026-10-30 07:30:02.07836811 +0000 UTC m=+1960.687775079"
Oct 30 07:30:02.126195 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:02.126142    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889090-6fr4m" podStartSLOduration=2.126129897 podStartE2EDuration="2.126129897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:30:02.124166014 +0000 UTC m=+1960.733572983" watchObservedRunningTime="2026-10-30 07:30:02.126129897 +0000 UTC m=+1960.735536924"
Oct 30 07:30:02.177001 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:02.176951    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889090-9ccls" podStartSLOduration=2.176936673 podStartE2EDuration="2.176936673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:30:02.173781112 +0000 UTC m=+1960.783188082" watchObservedRunningTime="2026-10-30 07:30:02.176936673 +0000 UTC m=+1960.786343689"
Oct 30 07:30:02.950331 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:02.950296    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889090-fzvfn" event={"ID":"7b67837f-7acb-4ec6-86a5-dbe1263b46e2","Type":"ContainerStarted","Data":"9b97288d534cbc679ccbe2c863bdc9e3246fe56a62058efe4539f6bf25e7b142"}
Oct 30 07:30:02.951962 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:02.951934    6224 generic.go:356] "Generic (PLEG): container finished" podID="59474a91-6ad5-4a1b-9220-30356c14f255" containerID="4d4a4da6a1bc6863152cec647db415105c19a5f094dbe248df503052ce728669" exitCode=60
Oct 30 07:30:02.952069 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:02.952017    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889090-28b2b" event={"ID":"59474a91-6ad5-4a1b-9220-30356c14f255","Type":"ContainerDied","Data":"4d4a4da6a1bc6863152cec647db415105c19a5f094dbe248df503052ce728669"}
Oct 30 07:30:02.952255 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:02.952239    6224 scope.go:122] "RemoveContainer" containerID="4d4a4da6a1bc6863152cec647db415105c19a5f094dbe248df503052ce728669"
Oct 30 07:30:02.953711 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:02.953690    6224 generic.go:356] "Generic (PLEG): container finished" podID="d87fa68a-a7e0-41c1-92c3-65c1faec9821" containerID="5c910463f311bcf9003119aba6716798f9acfc37056b8096213763e440f198cc" exitCode=60
Oct 30 07:30:02.953840 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:02.953764    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889090-jn62g" event={"ID":"d87fa68a-a7e0-41c1-92c3-65c1faec9821","Type":"ContainerDied","Data":"5c910463f311bcf9003119aba6716798f9acfc37056b8096213763e440f198cc"}
Oct 30 07:30:02.953972 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:02.953962    6224 scope.go:122] "RemoveContainer" containerID="5c910463f311bcf9003119aba6716798f9acfc37056b8096213763e440f198cc"
Oct 30 07:30:02.955672 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:02.955646    6224 generic.go:356] "Generic (PLEG): container finished" podID="94c9d1b4-2f77-4e91-aa74-8bbb2bc04713" containerID="484474c092c0dec64b786f026716045c2e6fa013abe009e15cf30fb8b15c32ec" exitCode=60
Oct 30 07:30:02.955920 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:02.955754    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889090-6fr4m" event={"ID":"94c9d1b4-2f77-4e91-aa74-8bbb2bc04713","Type":"ContainerDied","Data":"484474c092c0dec64b786f026716045c2e6fa013abe009e15cf30fb8b15c32ec"}
Oct 30 07:30:02.955959 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:02.955944    6224 scope.go:122] "RemoveContainer" containerID="484474c092c0dec64b786f026716045c2e6fa013abe009e15cf30fb8b15c32ec"
Oct 30 07:30:02.957585 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:02.957562    6224 generic.go:356] "Generic (PLEG): container finished" podID="cd765ec1-29ac-4173-9a95-17099c37379a" containerID="d5157cf45160241602e0ea8ff20ff8538073fbc6c8443288ecb43ae9b4698e54" exitCode=60
Oct 30 07:30:02.957666 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:02.957642    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889090-9ccls" event={"ID":"cd765ec1-29ac-4173-9a95-17099c37379a","Type":"ContainerDied","Data":"d5157cf45160241602e0ea8ff20ff8538073fbc6c8443288ecb43ae9b4698e54"}
Oct 30 07:30:02.957799 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:02.957786    6224 scope.go:122] "RemoveContainer" containerID="d5157cf45160241602e0ea8ff20ff8538073fbc6c8443288ecb43ae9b4698e54"
Oct 30 07:30:03.004743 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:03.004694    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889090-fzvfn" podStartSLOduration=3.00468064 podStartE2EDuration="3.00468064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:30:03.001514409 +0000 UTC m=+1961.610921404" watchObservedRunningTime="2026-10-30 07:30:03.00468064 +0000 UTC m=+1961.614087650"
Oct 30 07:30:03.505430 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:03.505402    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ce27afe-fbcc-4c82-9986-ab4aa4f575ee" path="/var/lib/kubelet/pods/4ce27afe-fbcc-4c82-9986-ab4aa4f575ee/volumes"
Oct 30 07:30:03.972669 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:03.972631    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889090-jn62g" event={"ID":"d87fa68a-a7e0-41c1-92c3-65c1faec9821","Type":"ContainerStarted","Data":"30d6b47c1cb23702dfe61d2fa1cd1915341537c40940d067c37bef81edd903c2"}
Oct 30 07:30:03.980877 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:03.980839    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889090-6fr4m" event={"ID":"94c9d1b4-2f77-4e91-aa74-8bbb2bc04713","Type":"ContainerStarted","Data":"970e4def9622835f4b87dfdf87cb1ce9362179eb700aa69adc8482d4a103b9c1"}
Oct 30 07:30:03.989372 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:03.989341    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889090-9ccls" event={"ID":"cd765ec1-29ac-4173-9a95-17099c37379a","Type":"ContainerStarted","Data":"58eb2521f0b8cabb69dab628f0afd044a7e0085b0c3a48a591a2ffadbb2dde3c"}
Oct 30 07:30:03.989683 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:03.989663    6224 scope.go:122] "RemoveContainer" containerID="58eb2521f0b8cabb69dab628f0afd044a7e0085b0c3a48a591a2ffadbb2dde3c"
Oct 30 07:30:03.990073 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:30:03.990044    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-265384-29889090-9ccls_odh-ai-gateway-infra(cd765ec1-29ac-4173-9a95-17099c37379a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889090-9ccls" podUID="cd765ec1-29ac-4173-9a95-17099c37379a"
Oct 30 07:30:03.991645 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:03.991622    6224 generic.go:356] "Generic (PLEG): container finished" podID="7b67837f-7acb-4ec6-86a5-dbe1263b46e2" containerID="9b97288d534cbc679ccbe2c863bdc9e3246fe56a62058efe4539f6bf25e7b142" exitCode=60
Oct 30 07:30:03.991759 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:03.991697    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889090-fzvfn" event={"ID":"7b67837f-7acb-4ec6-86a5-dbe1263b46e2","Type":"ContainerDied","Data":"9b97288d534cbc679ccbe2c863bdc9e3246fe56a62058efe4539f6bf25e7b142"}
Oct 30 07:30:03.991869 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:03.991761    6224 scope.go:122] "RemoveContainer" containerID="b43195bfeb2bb8057947f2035711430be1e6e4f89a0d75875c4d4af4e9ab3461"
Oct 30 07:30:03.992084 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:03.992068    6224 scope.go:122] "RemoveContainer" containerID="9b97288d534cbc679ccbe2c863bdc9e3246fe56a62058efe4539f6bf25e7b142"
Oct 30 07:30:03.992430 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:30:03.992395    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889090-fzvfn_odh-ai-gateway-infra(7b67837f-7acb-4ec6-86a5-dbe1263b46e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889090-fzvfn" podUID="7b67837f-7acb-4ec6-86a5-dbe1263b46e2"
Oct 30 07:30:03.993769 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:03.993749    6224 generic.go:356] "Generic (PLEG): container finished" podID="59474a91-6ad5-4a1b-9220-30356c14f255" containerID="8b2e71b7628db28fea8731bc1dfff57c8e29840298f9f9e32bcc76cf8b3ff470" exitCode=60
Oct 30 07:30:03.993903 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:03.993833    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889090-28b2b" event={"ID":"59474a91-6ad5-4a1b-9220-30356c14f255","Type":"ContainerDied","Data":"8b2e71b7628db28fea8731bc1dfff57c8e29840298f9f9e32bcc76cf8b3ff470"}
Oct 30 07:30:03.994105 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:03.994089    6224 scope.go:122] "RemoveContainer" containerID="8b2e71b7628db28fea8731bc1dfff57c8e29840298f9f9e32bcc76cf8b3ff470"
Oct 30 07:30:03.994368 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:30:03.994346    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889090-28b2b_odh-ai-gateway-infra(59474a91-6ad5-4a1b-9220-30356c14f255)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889090-28b2b" podUID="59474a91-6ad5-4a1b-9220-30356c14f255"
Oct 30 07:30:04.187682 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:04.187656    6224 scope.go:122] "RemoveContainer" containerID="4d4a4da6a1bc6863152cec647db415105c19a5f094dbe248df503052ce728669"
Oct 30 07:30:05.003450 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:05.003413    6224 generic.go:356] "Generic (PLEG): container finished" podID="d87fa68a-a7e0-41c1-92c3-65c1faec9821" containerID="30d6b47c1cb23702dfe61d2fa1cd1915341537c40940d067c37bef81edd903c2" exitCode=60
Oct 30 07:30:05.003945 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:05.003458    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889090-jn62g" event={"ID":"d87fa68a-a7e0-41c1-92c3-65c1faec9821","Type":"ContainerDied","Data":"30d6b47c1cb23702dfe61d2fa1cd1915341537c40940d067c37bef81edd903c2"}
Oct 30 07:30:05.003945 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:05.003517    6224 scope.go:122] "RemoveContainer" containerID="5c910463f311bcf9003119aba6716798f9acfc37056b8096213763e440f198cc"
Oct 30 07:30:05.003945 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:05.003895    6224 scope.go:122] "RemoveContainer" containerID="30d6b47c1cb23702dfe61d2fa1cd1915341537c40940d067c37bef81edd903c2"
Oct 30 07:30:05.004306 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:30:05.004281    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889090-jn62g_odh-ai-gateway-infra(d87fa68a-a7e0-41c1-92c3-65c1faec9821)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889090-jn62g" podUID="d87fa68a-a7e0-41c1-92c3-65c1faec9821"
Oct 30 07:30:05.005653 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:05.005633    6224 generic.go:356] "Generic (PLEG): container finished" podID="94c9d1b4-2f77-4e91-aa74-8bbb2bc04713" containerID="970e4def9622835f4b87dfdf87cb1ce9362179eb700aa69adc8482d4a103b9c1" exitCode=60
Oct 30 07:30:05.005748 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:05.005716    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889090-6fr4m" event={"ID":"94c9d1b4-2f77-4e91-aa74-8bbb2bc04713","Type":"ContainerDied","Data":"970e4def9622835f4b87dfdf87cb1ce9362179eb700aa69adc8482d4a103b9c1"}
Oct 30 07:30:05.005970 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:05.005955    6224 scope.go:122] "RemoveContainer" containerID="970e4def9622835f4b87dfdf87cb1ce9362179eb700aa69adc8482d4a103b9c1"
Oct 30 07:30:05.006301 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:30:05.006276    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889090-6fr4m_odh-ai-gateway-infra(94c9d1b4-2f77-4e91-aa74-8bbb2bc04713)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889090-6fr4m" podUID="94c9d1b4-2f77-4e91-aa74-8bbb2bc04713"
Oct 30 07:30:05.007527 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:05.007508    6224 generic.go:356] "Generic (PLEG): container finished" podID="cd765ec1-29ac-4173-9a95-17099c37379a" containerID="58eb2521f0b8cabb69dab628f0afd044a7e0085b0c3a48a591a2ffadbb2dde3c" exitCode=60
Oct 30 07:30:05.007620 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:05.007577    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889090-9ccls" event={"ID":"cd765ec1-29ac-4173-9a95-17099c37379a","Type":"ContainerDied","Data":"58eb2521f0b8cabb69dab628f0afd044a7e0085b0c3a48a591a2ffadbb2dde3c"}
Oct 30 07:30:05.007877 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:05.007860    6224 scope.go:122] "RemoveContainer" containerID="58eb2521f0b8cabb69dab628f0afd044a7e0085b0c3a48a591a2ffadbb2dde3c"
Oct 30 07:30:05.008180 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:30:05.008159    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-265384-29889090-9ccls_odh-ai-gateway-infra(cd765ec1-29ac-4173-9a95-17099c37379a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889090-9ccls" podUID="cd765ec1-29ac-4173-9a95-17099c37379a"
Oct 30 07:30:05.009427 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:05.009408    6224 scope.go:122] "RemoveContainer" containerID="9b97288d534cbc679ccbe2c863bdc9e3246fe56a62058efe4539f6bf25e7b142"
Oct 30 07:30:05.009667 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:30:05.009641    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889090-fzvfn_odh-ai-gateway-infra(7b67837f-7acb-4ec6-86a5-dbe1263b46e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889090-fzvfn" podUID="7b67837f-7acb-4ec6-86a5-dbe1263b46e2"
Oct 30 07:30:05.010899 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:05.010881    6224 scope.go:122] "RemoveContainer" containerID="8b2e71b7628db28fea8731bc1dfff57c8e29840298f9f9e32bcc76cf8b3ff470"
Oct 30 07:30:05.011124 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:30:05.011105    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889090-28b2b_odh-ai-gateway-infra(59474a91-6ad5-4a1b-9220-30356c14f255)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889090-28b2b" podUID="59474a91-6ad5-4a1b-9220-30356c14f255"
Oct 30 07:30:05.018356 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:05.018337    6224 scope.go:122] "RemoveContainer" containerID="484474c092c0dec64b786f026716045c2e6fa013abe009e15cf30fb8b15c32ec"
Oct 30 07:30:05.031838 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:05.031820    6224 scope.go:122] "RemoveContainer" containerID="d5157cf45160241602e0ea8ff20ff8538073fbc6c8443288ecb43ae9b4698e54"
Oct 30 07:30:06.020504 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:06.020475    6224 scope.go:122] "RemoveContainer" containerID="30d6b47c1cb23702dfe61d2fa1cd1915341537c40940d067c37bef81edd903c2"
Oct 30 07:30:06.020948 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:30:06.020854    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889090-jn62g_odh-ai-gateway-infra(d87fa68a-a7e0-41c1-92c3-65c1faec9821)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889090-jn62g" podUID="d87fa68a-a7e0-41c1-92c3-65c1faec9821"
Oct 30 07:30:06.022122 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:06.022098    6224 scope.go:122] "RemoveContainer" containerID="970e4def9622835f4b87dfdf87cb1ce9362179eb700aa69adc8482d4a103b9c1"
Oct 30 07:30:06.022441 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:30:06.022418    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889090-6fr4m_odh-ai-gateway-infra(94c9d1b4-2f77-4e91-aa74-8bbb2bc04713)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889090-6fr4m" podUID="94c9d1b4-2f77-4e91-aa74-8bbb2bc04713"
Oct 30 07:30:06.023716 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:06.023698    6224 scope.go:122] "RemoveContainer" containerID="58eb2521f0b8cabb69dab628f0afd044a7e0085b0c3a48a591a2ffadbb2dde3c"
Oct 30 07:30:06.024019 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:30:06.024000    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-265384-29889090-9ccls_odh-ai-gateway-infra(cd765ec1-29ac-4173-9a95-17099c37379a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889090-9ccls" podUID="cd765ec1-29ac-4173-9a95-17099c37379a"
Oct 30 07:30:15.497081 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:15.497049    6224 scope.go:122] "RemoveContainer" containerID="9b97288d534cbc679ccbe2c863bdc9e3246fe56a62058efe4539f6bf25e7b142"
Oct 30 07:30:15.497463 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:15.497171    6224 scope.go:122] "RemoveContainer" containerID="8b2e71b7628db28fea8731bc1dfff57c8e29840298f9f9e32bcc76cf8b3ff470"
Oct 30 07:30:16.097024 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:16.096990    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889090-fzvfn" event={"ID":"7b67837f-7acb-4ec6-86a5-dbe1263b46e2","Type":"ContainerStarted","Data":"be428dbb26031f11eefccfde3069eb8bcb961e82592e60dbc1ee1b07c1c6eeaa"}
Oct 30 07:30:16.098877 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:16.098852    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889090-28b2b" event={"ID":"59474a91-6ad5-4a1b-9220-30356c14f255","Type":"ContainerStarted","Data":"c596f0005dd2db88e121fcd6d6d5b0f5181b3da226a55cfc851b4f1460c381c8"}
Oct 30 07:30:16.496198 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:16.496169    6224 scope.go:122] "RemoveContainer" containerID="970e4def9622835f4b87dfdf87cb1ce9362179eb700aa69adc8482d4a103b9c1"
Oct 30 07:30:17.108147 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:17.108065    6224 generic.go:356] "Generic (PLEG): container finished" podID="7b67837f-7acb-4ec6-86a5-dbe1263b46e2" containerID="be428dbb26031f11eefccfde3069eb8bcb961e82592e60dbc1ee1b07c1c6eeaa" exitCode=60
Oct 30 07:30:17.108147 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:17.108133    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889090-fzvfn" event={"ID":"7b67837f-7acb-4ec6-86a5-dbe1263b46e2","Type":"ContainerDied","Data":"be428dbb26031f11eefccfde3069eb8bcb961e82592e60dbc1ee1b07c1c6eeaa"}
Oct 30 07:30:17.108628 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:17.108191    6224 scope.go:122] "RemoveContainer" containerID="9b97288d534cbc679ccbe2c863bdc9e3246fe56a62058efe4539f6bf25e7b142"
Oct 30 07:30:17.108628 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:17.108548    6224 scope.go:122] "RemoveContainer" containerID="be428dbb26031f11eefccfde3069eb8bcb961e82592e60dbc1ee1b07c1c6eeaa"
Oct 30 07:30:17.108935 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:30:17.108909    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889090-fzvfn_odh-ai-gateway-infra(7b67837f-7acb-4ec6-86a5-dbe1263b46e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889090-fzvfn" podUID="7b67837f-7acb-4ec6-86a5-dbe1263b46e2"
Oct 30 07:30:17.110897 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:17.110876    6224 generic.go:356] "Generic (PLEG): container finished" podID="59474a91-6ad5-4a1b-9220-30356c14f255" containerID="c596f0005dd2db88e121fcd6d6d5b0f5181b3da226a55cfc851b4f1460c381c8" exitCode=60
Oct 30 07:30:17.111000 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:17.110945    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889090-28b2b" event={"ID":"59474a91-6ad5-4a1b-9220-30356c14f255","Type":"ContainerDied","Data":"c596f0005dd2db88e121fcd6d6d5b0f5181b3da226a55cfc851b4f1460c381c8"}
Oct 30 07:30:17.111264 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:17.111247    6224 scope.go:122] "RemoveContainer" containerID="c596f0005dd2db88e121fcd6d6d5b0f5181b3da226a55cfc851b4f1460c381c8"
Oct 30 07:30:17.111542 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:30:17.111525    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889090-28b2b_odh-ai-gateway-infra(59474a91-6ad5-4a1b-9220-30356c14f255)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889090-28b2b" podUID="59474a91-6ad5-4a1b-9220-30356c14f255"
Oct 30 07:30:17.112892 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:17.112871    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889090-6fr4m" event={"ID":"94c9d1b4-2f77-4e91-aa74-8bbb2bc04713","Type":"ContainerStarted","Data":"ad2a011cb350b26ac551c2f28e5e5ac92cfcc88c340c0074e8d610ebdf85e6f3"}
Oct 30 07:30:17.120756 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:17.120739    6224 scope.go:122] "RemoveContainer" containerID="8b2e71b7628db28fea8731bc1dfff57c8e29840298f9f9e32bcc76cf8b3ff470"
Oct 30 07:30:17.153248 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:17.153219    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889090-fzvfn"]
Oct 30 07:30:17.197774 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:17.197745    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889090-28b2b"]
Oct 30 07:30:17.496703 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:17.496675    6224 scope.go:122] "RemoveContainer" containerID="58eb2521f0b8cabb69dab628f0afd044a7e0085b0c3a48a591a2ffadbb2dde3c"
Oct 30 07:30:17.496911 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:17.496821    6224 scope.go:122] "RemoveContainer" containerID="30d6b47c1cb23702dfe61d2fa1cd1915341537c40940d067c37bef81edd903c2"
Oct 30 07:30:18.126358 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:18.126326    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889090-jn62g" event={"ID":"d87fa68a-a7e0-41c1-92c3-65c1faec9821","Type":"ContainerStarted","Data":"509969990d6fe3d6e53ac978418b070f3e14e3071fc0bfaa35b4a8551d3f039a"}
Oct 30 07:30:18.128591 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:18.128568    6224 generic.go:356] "Generic (PLEG): container finished" podID="94c9d1b4-2f77-4e91-aa74-8bbb2bc04713" containerID="ad2a011cb350b26ac551c2f28e5e5ac92cfcc88c340c0074e8d610ebdf85e6f3" exitCode=60
Oct 30 07:30:18.128702 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:18.128640    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889090-6fr4m" event={"ID":"94c9d1b4-2f77-4e91-aa74-8bbb2bc04713","Type":"ContainerDied","Data":"ad2a011cb350b26ac551c2f28e5e5ac92cfcc88c340c0074e8d610ebdf85e6f3"}
Oct 30 07:30:18.128702 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:18.128695    6224 scope.go:122] "RemoveContainer" containerID="970e4def9622835f4b87dfdf87cb1ce9362179eb700aa69adc8482d4a103b9c1"
Oct 30 07:30:18.129355 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:18.129153    6224 scope.go:122] "RemoveContainer" containerID="ad2a011cb350b26ac551c2f28e5e5ac92cfcc88c340c0074e8d610ebdf85e6f3"
Oct 30 07:30:18.129525 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:30:18.129501    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889090-6fr4m_odh-ai-gateway-infra(94c9d1b4-2f77-4e91-aa74-8bbb2bc04713)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889090-6fr4m" podUID="94c9d1b4-2f77-4e91-aa74-8bbb2bc04713"
Oct 30 07:30:18.131052 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:18.131032    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889090-9ccls" event={"ID":"cd765ec1-29ac-4173-9a95-17099c37379a","Type":"ContainerStarted","Data":"f6e0c415d05d2b6b92af0caef78b39e475700780ba7fac0772e1b20e2fc808cb"}
Oct 30 07:30:18.237068 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:18.235522    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889090-6fr4m"]
Oct 30 07:30:18.358339 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:18.358315    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889090-fzvfn"
Oct 30 07:30:18.401222 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:18.401154    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889090-28b2b"
Oct 30 07:30:18.497367 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:18.497341    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b67837f-7acb-4ec6-86a5-dbe1263b46e2-kube-api-access-927tk\" (UniqueName: \"kubernetes.io/projected/7b67837f-7acb-4ec6-86a5-dbe1263b46e2-kube-api-access-927tk\") pod \"7b67837f-7acb-4ec6-86a5-dbe1263b46e2\" (UID: \"7b67837f-7acb-4ec6-86a5-dbe1263b46e2\") "
Oct 30 07:30:18.497542 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:18.497524    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59474a91-6ad5-4a1b-9220-30356c14f255-kube-api-access-plm8k\" (UniqueName: \"kubernetes.io/projected/59474a91-6ad5-4a1b-9220-30356c14f255-kube-api-access-plm8k\") pod \"59474a91-6ad5-4a1b-9220-30356c14f255\" (UID: \"59474a91-6ad5-4a1b-9220-30356c14f255\") "
Oct 30 07:30:18.497989 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:18.497628    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59474a91-6ad5-4a1b-9220-30356c14f255-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59474a91-6ad5-4a1b-9220-30356c14f255-maas-api-service-ca\") pod \"59474a91-6ad5-4a1b-9220-30356c14f255\" (UID: \"59474a91-6ad5-4a1b-9220-30356c14f255\") "
Oct 30 07:30:18.497989 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:18.497711    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b67837f-7acb-4ec6-86a5-dbe1263b46e2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b67837f-7acb-4ec6-86a5-dbe1263b46e2-maas-api-service-ca\") pod \"7b67837f-7acb-4ec6-86a5-dbe1263b46e2\" (UID: \"7b67837f-7acb-4ec6-86a5-dbe1263b46e2\") "
Oct 30 07:30:18.498160 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:18.498095    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59474a91-6ad5-4a1b-9220-30356c14f255-maas-api-service-ca" pod "59474a91-6ad5-4a1b-9220-30356c14f255" (UID: "59474a91-6ad5-4a1b-9220-30356c14f255"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:30:18.498411 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:18.498373    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b67837f-7acb-4ec6-86a5-dbe1263b46e2-maas-api-service-ca" pod "7b67837f-7acb-4ec6-86a5-dbe1263b46e2" (UID: "7b67837f-7acb-4ec6-86a5-dbe1263b46e2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:30:18.498946 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:18.498909    6224 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59474a91-6ad5-4a1b-9220-30356c14f255-maas-api-service-ca\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:30:18.498946 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:18.498932    6224 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b67837f-7acb-4ec6-86a5-dbe1263b46e2-maas-api-service-ca\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:30:18.501166 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:18.501142    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59474a91-6ad5-4a1b-9220-30356c14f255-kube-api-access-plm8k" pod "59474a91-6ad5-4a1b-9220-30356c14f255" (UID: "59474a91-6ad5-4a1b-9220-30356c14f255"). InnerVolumeSpecName "kube-api-access-plm8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:30:18.502786 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:18.502755    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b67837f-7acb-4ec6-86a5-dbe1263b46e2-kube-api-access-927tk" pod "7b67837f-7acb-4ec6-86a5-dbe1263b46e2" (UID: "7b67837f-7acb-4ec6-86a5-dbe1263b46e2"). InnerVolumeSpecName "kube-api-access-927tk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:30:18.600488 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:18.600450    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-927tk\" (UniqueName: \"kubernetes.io/projected/7b67837f-7acb-4ec6-86a5-dbe1263b46e2-kube-api-access-927tk\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:30:18.600488 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:18.600484    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plm8k\" (UniqueName: \"kubernetes.io/projected/59474a91-6ad5-4a1b-9220-30356c14f255-kube-api-access-plm8k\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:30:19.139693 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:19.139663    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889090-fzvfn"
Oct 30 07:30:19.140187 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:19.139662    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889090-fzvfn" event={"ID":"7b67837f-7acb-4ec6-86a5-dbe1263b46e2","Type":"ContainerDied","Data":"6e21d6a00488afbf58e0fc7b4c36b9147f29d3b27c07651f3216121ad2a41d10"}
Oct 30 07:30:19.140187 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:19.139787    6224 scope.go:122] "RemoveContainer" containerID="be428dbb26031f11eefccfde3069eb8bcb961e82592e60dbc1ee1b07c1c6eeaa"
Oct 30 07:30:19.141188 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:19.141170    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889090-28b2b"
Oct 30 07:30:19.141291 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:19.141194    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889090-28b2b" event={"ID":"59474a91-6ad5-4a1b-9220-30356c14f255","Type":"ContainerDied","Data":"654d6f4c1e2e8c4b13cdd69624af75c856b6e4d3e5baabf8838eee74b7376674"}
Oct 30 07:30:19.143511 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:19.143484    6224 generic.go:356] "Generic (PLEG): container finished" podID="d87fa68a-a7e0-41c1-92c3-65c1faec9821" containerID="509969990d6fe3d6e53ac978418b070f3e14e3071fc0bfaa35b4a8551d3f039a" exitCode=60
Oct 30 07:30:19.143609 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:19.143557    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889090-jn62g" event={"ID":"d87fa68a-a7e0-41c1-92c3-65c1faec9821","Type":"ContainerDied","Data":"509969990d6fe3d6e53ac978418b070f3e14e3071fc0bfaa35b4a8551d3f039a"}
Oct 30 07:30:19.144018 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:19.143996    6224 scope.go:122] "RemoveContainer" containerID="509969990d6fe3d6e53ac978418b070f3e14e3071fc0bfaa35b4a8551d3f039a"
Oct 30 07:30:19.144351 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:30:19.144321    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889090-jn62g_odh-ai-gateway-infra(d87fa68a-a7e0-41c1-92c3-65c1faec9821)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889090-jn62g" podUID="d87fa68a-a7e0-41c1-92c3-65c1faec9821"
Oct 30 07:30:19.147750 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:19.147717    6224 generic.go:356] "Generic (PLEG): container finished" podID="cd765ec1-29ac-4173-9a95-17099c37379a" containerID="f6e0c415d05d2b6b92af0caef78b39e475700780ba7fac0772e1b20e2fc808cb" exitCode=60
Oct 30 07:30:19.147858 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:19.147783    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889090-9ccls" event={"ID":"cd765ec1-29ac-4173-9a95-17099c37379a","Type":"ContainerDied","Data":"f6e0c415d05d2b6b92af0caef78b39e475700780ba7fac0772e1b20e2fc808cb"}
Oct 30 07:30:19.148230 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:19.148212    6224 scope.go:122] "RemoveContainer" containerID="f6e0c415d05d2b6b92af0caef78b39e475700780ba7fac0772e1b20e2fc808cb"
Oct 30 07:30:19.148550 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:30:19.148527    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-265384-29889090-9ccls_odh-ai-gateway-infra(cd765ec1-29ac-4173-9a95-17099c37379a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889090-9ccls" podUID="cd765ec1-29ac-4173-9a95-17099c37379a"
Oct 30 07:30:19.152080 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:19.152059    6224 scope.go:122] "RemoveContainer" containerID="c596f0005dd2db88e121fcd6d6d5b0f5181b3da226a55cfc851b4f1460c381c8"
Oct 30 07:30:19.161846 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:19.161801    6224 scope.go:122] "RemoveContainer" containerID="30d6b47c1cb23702dfe61d2fa1cd1915341537c40940d067c37bef81edd903c2"
Oct 30 07:30:19.175329 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:19.175301    6224 scope.go:122] "RemoveContainer" containerID="58eb2521f0b8cabb69dab628f0afd044a7e0085b0c3a48a591a2ffadbb2dde3c"
Oct 30 07:30:19.279271 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:19.279240    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889090-jn62g"]
Oct 30 07:30:19.316256 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:19.316231    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889090-6fr4m"
Oct 30 07:30:19.334040 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:19.334008    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889090-28b2b"]
Oct 30 07:30:19.337650 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:19.337620    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889090-28b2b"]
Oct 30 07:30:19.374187 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:19.374160    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889090-fzvfn"]
Oct 30 07:30:19.381599 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:19.381575    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-14271d-29889090-fzvfn"]
Oct 30 07:30:19.414577 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:19.414507    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889090-9ccls"]
Oct 30 07:30:19.420191 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:19.420162    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94c9d1b4-2f77-4e91-aa74-8bbb2bc04713-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94c9d1b4-2f77-4e91-aa74-8bbb2bc04713-maas-api-service-ca\") pod \"94c9d1b4-2f77-4e91-aa74-8bbb2bc04713\" (UID: \"94c9d1b4-2f77-4e91-aa74-8bbb2bc04713\") "
Oct 30 07:30:19.420399 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:19.420260    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94c9d1b4-2f77-4e91-aa74-8bbb2bc04713-kube-api-access-hfhtb\" (UniqueName: \"kubernetes.io/projected/94c9d1b4-2f77-4e91-aa74-8bbb2bc04713-kube-api-access-hfhtb\") pod \"94c9d1b4-2f77-4e91-aa74-8bbb2bc04713\" (UID: \"94c9d1b4-2f77-4e91-aa74-8bbb2bc04713\") "
Oct 30 07:30:19.420763 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:19.420596    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94c9d1b4-2f77-4e91-aa74-8bbb2bc04713-maas-api-service-ca" pod "94c9d1b4-2f77-4e91-aa74-8bbb2bc04713" (UID: "94c9d1b4-2f77-4e91-aa74-8bbb2bc04713"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:30:19.421180 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:19.421159    6224 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94c9d1b4-2f77-4e91-aa74-8bbb2bc04713-maas-api-service-ca\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:30:19.424352 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:19.424325    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94c9d1b4-2f77-4e91-aa74-8bbb2bc04713-kube-api-access-hfhtb" pod "94c9d1b4-2f77-4e91-aa74-8bbb2bc04713" (UID: "94c9d1b4-2f77-4e91-aa74-8bbb2bc04713"). InnerVolumeSpecName "kube-api-access-hfhtb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:30:19.504266 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:19.504230    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59474a91-6ad5-4a1b-9220-30356c14f255" path="/var/lib/kubelet/pods/59474a91-6ad5-4a1b-9220-30356c14f255/volumes"
Oct 30 07:30:19.504827 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:19.504789    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b67837f-7acb-4ec6-86a5-dbe1263b46e2" path="/var/lib/kubelet/pods/7b67837f-7acb-4ec6-86a5-dbe1263b46e2/volumes"
Oct 30 07:30:19.522211 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:19.522187    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfhtb\" (UniqueName: \"kubernetes.io/projected/94c9d1b4-2f77-4e91-aa74-8bbb2bc04713-kube-api-access-hfhtb\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:30:20.158029 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:20.158000    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889090-6fr4m"
Oct 30 07:30:20.158416 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:20.158026    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889090-6fr4m" event={"ID":"94c9d1b4-2f77-4e91-aa74-8bbb2bc04713","Type":"ContainerDied","Data":"c4486f98b5e4d5983c5273a4f6a12ca6d976f1a318cbf627de6adf04cf29e63b"}
Oct 30 07:30:20.158416 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:20.158088    6224 scope.go:122] "RemoveContainer" containerID="ad2a011cb350b26ac551c2f28e5e5ac92cfcc88c340c0074e8d610ebdf85e6f3"
Oct 30 07:30:20.256058 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:20.253766    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889090-6fr4m"]
Oct 30 07:30:20.269794 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:20.269659    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889090-6fr4m"]
Oct 30 07:30:20.431791 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:20.431713    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889090-9ccls"
Oct 30 07:30:20.436428 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:20.436065    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889090-jn62g"
Oct 30 07:30:20.536857 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:20.536819    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd765ec1-29ac-4173-9a95-17099c37379a-kube-api-access-6rq5f\" (UniqueName: \"kubernetes.io/projected/cd765ec1-29ac-4173-9a95-17099c37379a-kube-api-access-6rq5f\") pod \"cd765ec1-29ac-4173-9a95-17099c37379a\" (UID: \"cd765ec1-29ac-4173-9a95-17099c37379a\") "
Oct 30 07:30:20.537032 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:20.537001    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d87fa68a-a7e0-41c1-92c3-65c1faec9821-kube-api-access-dbwtz\" (UniqueName: \"kubernetes.io/projected/d87fa68a-a7e0-41c1-92c3-65c1faec9821-kube-api-access-dbwtz\") pod \"d87fa68a-a7e0-41c1-92c3-65c1faec9821\" (UID: \"d87fa68a-a7e0-41c1-92c3-65c1faec9821\") "
Oct 30 07:30:20.537099 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:20.537049    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd765ec1-29ac-4173-9a95-17099c37379a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd765ec1-29ac-4173-9a95-17099c37379a-maas-api-service-ca\") pod \"cd765ec1-29ac-4173-9a95-17099c37379a\" (UID: \"cd765ec1-29ac-4173-9a95-17099c37379a\") "
Oct 30 07:30:20.537292 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:20.537272    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d87fa68a-a7e0-41c1-92c3-65c1faec9821-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d87fa68a-a7e0-41c1-92c3-65c1faec9821-maas-api-service-ca\") pod \"d87fa68a-a7e0-41c1-92c3-65c1faec9821\" (UID: \"d87fa68a-a7e0-41c1-92c3-65c1faec9821\") "
Oct 30 07:30:20.541794 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:20.541755    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d87fa68a-a7e0-41c1-92c3-65c1faec9821-maas-api-service-ca" pod "d87fa68a-a7e0-41c1-92c3-65c1faec9821" (UID: "d87fa68a-a7e0-41c1-92c3-65c1faec9821"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:30:20.542250 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:20.542216    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd765ec1-29ac-4173-9a95-17099c37379a-maas-api-service-ca" pod "cd765ec1-29ac-4173-9a95-17099c37379a" (UID: "cd765ec1-29ac-4173-9a95-17099c37379a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:30:20.545845 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:20.544061    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d87fa68a-a7e0-41c1-92c3-65c1faec9821-kube-api-access-dbwtz" pod "d87fa68a-a7e0-41c1-92c3-65c1faec9821" (UID: "d87fa68a-a7e0-41c1-92c3-65c1faec9821"). InnerVolumeSpecName "kube-api-access-dbwtz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:30:20.551977 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:20.551948    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd765ec1-29ac-4173-9a95-17099c37379a-kube-api-access-6rq5f" pod "cd765ec1-29ac-4173-9a95-17099c37379a" (UID: "cd765ec1-29ac-4173-9a95-17099c37379a"). InnerVolumeSpecName "kube-api-access-6rq5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:30:20.640420 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:20.640385    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rq5f\" (UniqueName: \"kubernetes.io/projected/cd765ec1-29ac-4173-9a95-17099c37379a-kube-api-access-6rq5f\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:30:20.640420 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:20.640415    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbwtz\" (UniqueName: \"kubernetes.io/projected/d87fa68a-a7e0-41c1-92c3-65c1faec9821-kube-api-access-dbwtz\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:30:20.640420 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:20.640425    6224 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd765ec1-29ac-4173-9a95-17099c37379a-maas-api-service-ca\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:30:20.640705 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:20.640434    6224 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d87fa68a-a7e0-41c1-92c3-65c1faec9821-maas-api-service-ca\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:30:21.171002 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:21.170963    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889090-jn62g"
Oct 30 07:30:21.171002 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:21.170983    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889090-jn62g" event={"ID":"d87fa68a-a7e0-41c1-92c3-65c1faec9821","Type":"ContainerDied","Data":"f08a54aa589bca0f93e643c2d48c137981d7cd8818510c97105011d503fe8295"}
Oct 30 07:30:21.171440 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:21.171091    6224 scope.go:122] "RemoveContainer" containerID="509969990d6fe3d6e53ac978418b070f3e14e3071fc0bfaa35b4a8551d3f039a"
Oct 30 07:30:21.173691 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:21.173671    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889090-9ccls"
Oct 30 07:30:21.173691 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:21.173676    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889090-9ccls" event={"ID":"cd765ec1-29ac-4173-9a95-17099c37379a","Type":"ContainerDied","Data":"e4dbbae415c2f78095d2850a5f38e5c9b0613439803014cd7683a8e7140ba671"}
Oct 30 07:30:21.183041 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:21.183018    6224 scope.go:122] "RemoveContainer" containerID="f6e0c415d05d2b6b92af0caef78b39e475700780ba7fac0772e1b20e2fc808cb"
Oct 30 07:30:21.229338 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:21.229313    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889090-jn62g"]
Oct 30 07:30:21.242946 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:21.242917    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889090-jn62g"]
Oct 30 07:30:21.273947 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:21.273913    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889090-9ccls"]
Oct 30 07:30:21.277976 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:21.277948    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889090-9ccls"]
Oct 30 07:30:21.507506 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:21.507422    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94c9d1b4-2f77-4e91-aa74-8bbb2bc04713" path="/var/lib/kubelet/pods/94c9d1b4-2f77-4e91-aa74-8bbb2bc04713/volumes"
Oct 30 07:30:21.507935 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:21.507918    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd765ec1-29ac-4173-9a95-17099c37379a" path="/var/lib/kubelet/pods/cd765ec1-29ac-4173-9a95-17099c37379a/volumes"
Oct 30 07:30:21.508338 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:21.508324    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d87fa68a-a7e0-41c1-92c3-65c1faec9821" path="/var/lib/kubelet/pods/d87fa68a-a7e0-41c1-92c3-65c1faec9821/volumes"
Oct 30 07:30:23.172185 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:23.172153    6224 scope.go:122] "RemoveContainer" containerID="943bbdd422f55c8aa8f5e48542ceb9b64e1724e6345d1c8bee0593a3691d98d2"
Oct 30 07:30:23.182024 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:23.182005    6224 scope.go:122] "RemoveContainer" containerID="154a053b99a8d8d4b86b177e9ec1268b1c5cbb1a2295a7867dcebcf40ee92506"
Oct 30 07:30:23.192484 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:23.192459    6224 scope.go:122] "RemoveContainer" containerID="1b5a3b65466b8431bc64a51f957c30600081d3855126a26224d60e6d0d9319f1"
Oct 30 07:30:26.433095 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:26.433057    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:30:31.858901 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:30:31.858855    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:30:46.882488 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:46.882456    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7"]
Oct 30 07:30:46.882951 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:46.882669    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7" podUID="5611d3ae-2361-411e-903f-ebd0d5b5fba5" containerName="istio-proxy" containerID="cri-o://c502885b597dc38e28d29895403865104f6c4cb2cd717c8f66e0f73f10a6ba86" gracePeriod=30
Oct 30 07:30:46.896328 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:46.896298    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm"]
Oct 30 07:30:46.896524 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:46.896501    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm" podUID="5b5884ad-a370-40c8-89f5-6f260bbf234a" containerName="istio-proxy" containerID="cri-o://49b36688e9b6f9c2939fdb838afb3b74f4a069e6a0c550933a7d65189a962bf7" gracePeriod=30
Oct 30 07:30:46.900114 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:46.900089    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl"]
Oct 30 07:30:46.900250 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:46.900232    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl" podUID="1a470b54-ad63-40f7-b306-f8ee1dc8cdcd" containerName="istio-proxy" containerID="cri-o://6d13c4ff0518e5f5b38459897146a543bb2115422d7bfb74102c81906fbbc42f" gracePeriod=30
Oct 30 07:30:52.190688 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.190320    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7"
Oct 30 07:30:52.228118 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.226542    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5611d3ae-2361-411e-903f-ebd0d5b5fba5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5611d3ae-2361-411e-903f-ebd0d5b5fba5-credential-socket\") pod \"5611d3ae-2361-411e-903f-ebd0d5b5fba5\" (UID: \"5611d3ae-2361-411e-903f-ebd0d5b5fba5\") "
Oct 30 07:30:52.228118 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.226617    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5611d3ae-2361-411e-903f-ebd0d5b5fba5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5611d3ae-2361-411e-903f-ebd0d5b5fba5-istio-data\") pod \"5611d3ae-2361-411e-903f-ebd0d5b5fba5\" (UID: \"5611d3ae-2361-411e-903f-ebd0d5b5fba5\") "
Oct 30 07:30:52.228118 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.226741    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5611d3ae-2361-411e-903f-ebd0d5b5fba5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5611d3ae-2361-411e-903f-ebd0d5b5fba5-workload-socket\") pod \"5611d3ae-2361-411e-903f-ebd0d5b5fba5\" (UID: \"5611d3ae-2361-411e-903f-ebd0d5b5fba5\") "
Oct 30 07:30:52.228118 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.226850    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5611d3ae-2361-411e-903f-ebd0d5b5fba5-credential-socket" pod "5611d3ae-2361-411e-903f-ebd0d5b5fba5" (UID: "5611d3ae-2361-411e-903f-ebd0d5b5fba5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:30:52.228118 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.226971    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5611d3ae-2361-411e-903f-ebd0d5b5fba5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5611d3ae-2361-411e-903f-ebd0d5b5fba5-workload-certs\") pod \"5611d3ae-2361-411e-903f-ebd0d5b5fba5\" (UID: \"5611d3ae-2361-411e-903f-ebd0d5b5fba5\") "
Oct 30 07:30:52.228118 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.227050    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5611d3ae-2361-411e-903f-ebd0d5b5fba5-istio-data" pod "5611d3ae-2361-411e-903f-ebd0d5b5fba5" (UID: "5611d3ae-2361-411e-903f-ebd0d5b5fba5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:30:52.228118 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.227115    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5611d3ae-2361-411e-903f-ebd0d5b5fba5-istio-token\" (UniqueName: \"kubernetes.io/projected/5611d3ae-2361-411e-903f-ebd0d5b5fba5-istio-token\") pod \"5611d3ae-2361-411e-903f-ebd0d5b5fba5\" (UID: \"5611d3ae-2361-411e-903f-ebd0d5b5fba5\") "
Oct 30 07:30:52.228118 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.227181    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5611d3ae-2361-411e-903f-ebd0d5b5fba5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5611d3ae-2361-411e-903f-ebd0d5b5fba5-istio-envoy\") pod \"5611d3ae-2361-411e-903f-ebd0d5b5fba5\" (UID: \"5611d3ae-2361-411e-903f-ebd0d5b5fba5\") "
Oct 30 07:30:52.228118 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.227217    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5611d3ae-2361-411e-903f-ebd0d5b5fba5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5611d3ae-2361-411e-903f-ebd0d5b5fba5-istio-podinfo\") pod \"5611d3ae-2361-411e-903f-ebd0d5b5fba5\" (UID: \"5611d3ae-2361-411e-903f-ebd0d5b5fba5\") "
Oct 30 07:30:52.228118 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.227252    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5611d3ae-2361-411e-903f-ebd0d5b5fba5-workload-socket" pod "5611d3ae-2361-411e-903f-ebd0d5b5fba5" (UID: "5611d3ae-2361-411e-903f-ebd0d5b5fba5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:30:52.228118 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.227287    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5611d3ae-2361-411e-903f-ebd0d5b5fba5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5611d3ae-2361-411e-903f-ebd0d5b5fba5-istiod-ca-cert\") pod \"5611d3ae-2361-411e-903f-ebd0d5b5fba5\" (UID: \"5611d3ae-2361-411e-903f-ebd0d5b5fba5\") "
Oct 30 07:30:52.228118 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.227428    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5611d3ae-2361-411e-903f-ebd0d5b5fba5-kube-api-access-2h58d\" (UniqueName: \"kubernetes.io/projected/5611d3ae-2361-411e-903f-ebd0d5b5fba5-kube-api-access-2h58d\") pod \"5611d3ae-2361-411e-903f-ebd0d5b5fba5\" (UID: \"5611d3ae-2361-411e-903f-ebd0d5b5fba5\") "
Oct 30 07:30:52.228118 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.227903    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5611d3ae-2361-411e-903f-ebd0d5b5fba5-istiod-ca-cert" pod "5611d3ae-2361-411e-903f-ebd0d5b5fba5" (UID: "5611d3ae-2361-411e-903f-ebd0d5b5fba5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:30:52.228933 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.228234    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5611d3ae-2361-411e-903f-ebd0d5b5fba5-workload-certs" pod "5611d3ae-2361-411e-903f-ebd0d5b5fba5" (UID: "5611d3ae-2361-411e-903f-ebd0d5b5fba5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:30:52.228933 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.228306    6224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5611d3ae-2361-411e-903f-ebd0d5b5fba5-credential-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:30:52.228933 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.228326    6224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5611d3ae-2361-411e-903f-ebd0d5b5fba5-istio-data\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:30:52.228933 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.228341    6224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5611d3ae-2361-411e-903f-ebd0d5b5fba5-workload-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:30:52.228933 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.228357    6224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5611d3ae-2361-411e-903f-ebd0d5b5fba5-istiod-ca-cert\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:30:52.233129 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.233096    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5611d3ae-2361-411e-903f-ebd0d5b5fba5-istio-envoy" pod "5611d3ae-2361-411e-903f-ebd0d5b5fba5" (UID: "5611d3ae-2361-411e-903f-ebd0d5b5fba5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:30:52.233129 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.233122    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5611d3ae-2361-411e-903f-ebd0d5b5fba5-istio-podinfo" pod "5611d3ae-2361-411e-903f-ebd0d5b5fba5" (UID: "5611d3ae-2361-411e-903f-ebd0d5b5fba5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:30:52.233283 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.233174    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5611d3ae-2361-411e-903f-ebd0d5b5fba5-istio-token" pod "5611d3ae-2361-411e-903f-ebd0d5b5fba5" (UID: "5611d3ae-2361-411e-903f-ebd0d5b5fba5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:30:52.234593 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.234565    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5611d3ae-2361-411e-903f-ebd0d5b5fba5-kube-api-access-2h58d" pod "5611d3ae-2361-411e-903f-ebd0d5b5fba5" (UID: "5611d3ae-2361-411e-903f-ebd0d5b5fba5"). InnerVolumeSpecName "kube-api-access-2h58d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:30:52.235577 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.235557    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm"
Oct 30 07:30:52.251325 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.251309    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl"
Oct 30 07:30:52.329461 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.329427    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a470b54-ad63-40f7-b306-f8ee1dc8cdcd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a470b54-ad63-40f7-b306-f8ee1dc8cdcd-workload-certs\") pod \"1a470b54-ad63-40f7-b306-f8ee1dc8cdcd\" (UID: \"1a470b54-ad63-40f7-b306-f8ee1dc8cdcd\") "
Oct 30 07:30:52.329624 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.329494    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a470b54-ad63-40f7-b306-f8ee1dc8cdcd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a470b54-ad63-40f7-b306-f8ee1dc8cdcd-workload-socket\") pod \"1a470b54-ad63-40f7-b306-f8ee1dc8cdcd\" (UID: \"1a470b54-ad63-40f7-b306-f8ee1dc8cdcd\") "
Oct 30 07:30:52.329624 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.329532    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a470b54-ad63-40f7-b306-f8ee1dc8cdcd-istio-token\" (UniqueName: \"kubernetes.io/projected/1a470b54-ad63-40f7-b306-f8ee1dc8cdcd-istio-token\") pod \"1a470b54-ad63-40f7-b306-f8ee1dc8cdcd\" (UID: \"1a470b54-ad63-40f7-b306-f8ee1dc8cdcd\") "
Oct 30 07:30:52.329624 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.329563    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a470b54-ad63-40f7-b306-f8ee1dc8cdcd-kube-api-access-j2t4h\" (UniqueName: \"kubernetes.io/projected/1a470b54-ad63-40f7-b306-f8ee1dc8cdcd-kube-api-access-j2t4h\") pod \"1a470b54-ad63-40f7-b306-f8ee1dc8cdcd\" (UID: \"1a470b54-ad63-40f7-b306-f8ee1dc8cdcd\") "
Oct 30 07:30:52.329624 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.329594    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a470b54-ad63-40f7-b306-f8ee1dc8cdcd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a470b54-ad63-40f7-b306-f8ee1dc8cdcd-istiod-ca-cert\") pod \"1a470b54-ad63-40f7-b306-f8ee1dc8cdcd\" (UID: \"1a470b54-ad63-40f7-b306-f8ee1dc8cdcd\") "
Oct 30 07:30:52.329898 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.329768    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a470b54-ad63-40f7-b306-f8ee1dc8cdcd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a470b54-ad63-40f7-b306-f8ee1dc8cdcd-istio-podinfo\") pod \"1a470b54-ad63-40f7-b306-f8ee1dc8cdcd\" (UID: \"1a470b54-ad63-40f7-b306-f8ee1dc8cdcd\") "
Oct 30 07:30:52.329898 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.329772    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a470b54-ad63-40f7-b306-f8ee1dc8cdcd-workload-certs" pod "1a470b54-ad63-40f7-b306-f8ee1dc8cdcd" (UID: "1a470b54-ad63-40f7-b306-f8ee1dc8cdcd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:30:52.330013 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.329897    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b5884ad-a370-40c8-89f5-6f260bbf234a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b5884ad-a370-40c8-89f5-6f260bbf234a-istio-data\") pod \"5b5884ad-a370-40c8-89f5-6f260bbf234a\" (UID: \"5b5884ad-a370-40c8-89f5-6f260bbf234a\") "
Oct 30 07:30:52.330013 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.329902    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a470b54-ad63-40f7-b306-f8ee1dc8cdcd-workload-socket" pod "1a470b54-ad63-40f7-b306-f8ee1dc8cdcd" (UID: "1a470b54-ad63-40f7-b306-f8ee1dc8cdcd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:30:52.330013 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.329970    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b5884ad-a370-40c8-89f5-6f260bbf234a-kube-api-access-xjt2h\" (UniqueName: \"kubernetes.io/projected/5b5884ad-a370-40c8-89f5-6f260bbf234a-kube-api-access-xjt2h\") pod \"5b5884ad-a370-40c8-89f5-6f260bbf234a\" (UID: \"5b5884ad-a370-40c8-89f5-6f260bbf234a\") "
Oct 30 07:30:52.330179 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.330014    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a470b54-ad63-40f7-b306-f8ee1dc8cdcd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a470b54-ad63-40f7-b306-f8ee1dc8cdcd-istio-data\") pod \"1a470b54-ad63-40f7-b306-f8ee1dc8cdcd\" (UID: \"1a470b54-ad63-40f7-b306-f8ee1dc8cdcd\") "
Oct 30 07:30:52.330179 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.330043    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b5884ad-a370-40c8-89f5-6f260bbf234a-istio-token\" (UniqueName: \"kubernetes.io/projected/5b5884ad-a370-40c8-89f5-6f260bbf234a-istio-token\") pod \"5b5884ad-a370-40c8-89f5-6f260bbf234a\" (UID: \"5b5884ad-a370-40c8-89f5-6f260bbf234a\") "
Oct 30 07:30:52.330179 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.330118    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b5884ad-a370-40c8-89f5-6f260bbf234a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b5884ad-a370-40c8-89f5-6f260bbf234a-credential-socket\") pod \"5b5884ad-a370-40c8-89f5-6f260bbf234a\" (UID: \"5b5884ad-a370-40c8-89f5-6f260bbf234a\") "
Oct 30 07:30:52.330179 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.330133    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b5884ad-a370-40c8-89f5-6f260bbf234a-istio-data" pod "5b5884ad-a370-40c8-89f5-6f260bbf234a" (UID: "5b5884ad-a370-40c8-89f5-6f260bbf234a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:30:52.330405 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.330201    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b5884ad-a370-40c8-89f5-6f260bbf234a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b5884ad-a370-40c8-89f5-6f260bbf234a-istiod-ca-cert\") pod \"5b5884ad-a370-40c8-89f5-6f260bbf234a\" (UID: \"5b5884ad-a370-40c8-89f5-6f260bbf234a\") "
Oct 30 07:30:52.330405 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.330258    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b5884ad-a370-40c8-89f5-6f260bbf234a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b5884ad-a370-40c8-89f5-6f260bbf234a-istio-envoy\") pod \"5b5884ad-a370-40c8-89f5-6f260bbf234a\" (UID: \"5b5884ad-a370-40c8-89f5-6f260bbf234a\") "
Oct 30 07:30:52.330405 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.330325    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a470b54-ad63-40f7-b306-f8ee1dc8cdcd-istio-data" pod "1a470b54-ad63-40f7-b306-f8ee1dc8cdcd" (UID: "1a470b54-ad63-40f7-b306-f8ee1dc8cdcd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:30:52.330405 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.330360    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b5884ad-a370-40c8-89f5-6f260bbf234a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b5884ad-a370-40c8-89f5-6f260bbf234a-workload-socket\") pod \"5b5884ad-a370-40c8-89f5-6f260bbf234a\" (UID: \"5b5884ad-a370-40c8-89f5-6f260bbf234a\") "
Oct 30 07:30:52.330624 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.330406    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a470b54-ad63-40f7-b306-f8ee1dc8cdcd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a470b54-ad63-40f7-b306-f8ee1dc8cdcd-istio-envoy\") pod \"1a470b54-ad63-40f7-b306-f8ee1dc8cdcd\" (UID: \"1a470b54-ad63-40f7-b306-f8ee1dc8cdcd\") "
Oct 30 07:30:52.330624 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.330461    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a470b54-ad63-40f7-b306-f8ee1dc8cdcd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a470b54-ad63-40f7-b306-f8ee1dc8cdcd-credential-socket\") pod \"1a470b54-ad63-40f7-b306-f8ee1dc8cdcd\" (UID: \"1a470b54-ad63-40f7-b306-f8ee1dc8cdcd\") "
Oct 30 07:30:52.330624 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.330500    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b5884ad-a370-40c8-89f5-6f260bbf234a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b5884ad-a370-40c8-89f5-6f260bbf234a-workload-certs\") pod \"5b5884ad-a370-40c8-89f5-6f260bbf234a\" (UID: \"5b5884ad-a370-40c8-89f5-6f260bbf234a\") "
Oct 30 07:30:52.330624 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.330523    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b5884ad-a370-40c8-89f5-6f260bbf234a-credential-socket" pod "5b5884ad-a370-40c8-89f5-6f260bbf234a" (UID: "5b5884ad-a370-40c8-89f5-6f260bbf234a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:30:52.330624 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.330551    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5b5884ad-a370-40c8-89f5-6f260bbf234a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b5884ad-a370-40c8-89f5-6f260bbf234a-istio-podinfo\") pod \"5b5884ad-a370-40c8-89f5-6f260bbf234a\" (UID: \"5b5884ad-a370-40c8-89f5-6f260bbf234a\") "
Oct 30 07:30:52.331451 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.331423    6224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a470b54-ad63-40f7-b306-f8ee1dc8cdcd-workload-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:30:52.331559 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.331457    6224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b5884ad-a370-40c8-89f5-6f260bbf234a-istio-data\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:30:52.331559 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.331473    6224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a470b54-ad63-40f7-b306-f8ee1dc8cdcd-istio-data\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:30:52.331559 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.331489    6224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5611d3ae-2361-411e-903f-ebd0d5b5fba5-workload-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:30:52.331559 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.331502    6224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5611d3ae-2361-411e-903f-ebd0d5b5fba5-istio-token\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:30:52.331559 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.331516    6224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5611d3ae-2361-411e-903f-ebd0d5b5fba5-istio-envoy\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:30:52.331559 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.331530    6224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5611d3ae-2361-411e-903f-ebd0d5b5fba5-istio-podinfo\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:30:52.331559 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.331544    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h58d\" (UniqueName: \"kubernetes.io/projected/5611d3ae-2361-411e-903f-ebd0d5b5fba5-kube-api-access-2h58d\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:30:52.331559 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.331559    6224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a470b54-ad63-40f7-b306-f8ee1dc8cdcd-workload-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:30:52.332007 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.330551    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a470b54-ad63-40f7-b306-f8ee1dc8cdcd-istiod-ca-cert" pod "1a470b54-ad63-40f7-b306-f8ee1dc8cdcd" (UID: "1a470b54-ad63-40f7-b306-f8ee1dc8cdcd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:30:52.332007 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.331774    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a470b54-ad63-40f7-b306-f8ee1dc8cdcd-credential-socket" pod "1a470b54-ad63-40f7-b306-f8ee1dc8cdcd" (UID: "1a470b54-ad63-40f7-b306-f8ee1dc8cdcd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:30:52.332122 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.332023    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b5884ad-a370-40c8-89f5-6f260bbf234a-workload-certs" pod "5b5884ad-a370-40c8-89f5-6f260bbf234a" (UID: "5b5884ad-a370-40c8-89f5-6f260bbf234a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:30:52.332962 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.332934    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a470b54-ad63-40f7-b306-f8ee1dc8cdcd-istio-podinfo" pod "1a470b54-ad63-40f7-b306-f8ee1dc8cdcd" (UID: "1a470b54-ad63-40f7-b306-f8ee1dc8cdcd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:30:52.333774 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.333749    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b5884ad-a370-40c8-89f5-6f260bbf234a-istio-token" pod "5b5884ad-a370-40c8-89f5-6f260bbf234a" (UID: "5b5884ad-a370-40c8-89f5-6f260bbf234a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:30:52.334005 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.333978    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a470b54-ad63-40f7-b306-f8ee1dc8cdcd-istio-token" pod "1a470b54-ad63-40f7-b306-f8ee1dc8cdcd" (UID: "1a470b54-ad63-40f7-b306-f8ee1dc8cdcd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:30:52.334248 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.334219    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b5884ad-a370-40c8-89f5-6f260bbf234a-workload-socket" pod "5b5884ad-a370-40c8-89f5-6f260bbf234a" (UID: "5b5884ad-a370-40c8-89f5-6f260bbf234a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:30:52.334248 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.334226    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b5884ad-a370-40c8-89f5-6f260bbf234a-istiod-ca-cert" pod "5b5884ad-a370-40c8-89f5-6f260bbf234a" (UID: "5b5884ad-a370-40c8-89f5-6f260bbf234a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:30:52.334478 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.334268    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a470b54-ad63-40f7-b306-f8ee1dc8cdcd-kube-api-access-j2t4h" pod "1a470b54-ad63-40f7-b306-f8ee1dc8cdcd" (UID: "1a470b54-ad63-40f7-b306-f8ee1dc8cdcd"). InnerVolumeSpecName "kube-api-access-j2t4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:30:52.334676 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.334651    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a470b54-ad63-40f7-b306-f8ee1dc8cdcd-istio-envoy" pod "1a470b54-ad63-40f7-b306-f8ee1dc8cdcd" (UID: "1a470b54-ad63-40f7-b306-f8ee1dc8cdcd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:30:52.334754 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.334717    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b5884ad-a370-40c8-89f5-6f260bbf234a-istio-podinfo" pod "5b5884ad-a370-40c8-89f5-6f260bbf234a" (UID: "5b5884ad-a370-40c8-89f5-6f260bbf234a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:30:52.334754 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.334719    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b5884ad-a370-40c8-89f5-6f260bbf234a-kube-api-access-xjt2h" pod "5b5884ad-a370-40c8-89f5-6f260bbf234a" (UID: "5b5884ad-a370-40c8-89f5-6f260bbf234a"). InnerVolumeSpecName "kube-api-access-xjt2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:30:52.335686 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.335666    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b5884ad-a370-40c8-89f5-6f260bbf234a-istio-envoy" pod "5b5884ad-a370-40c8-89f5-6f260bbf234a" (UID: "5b5884ad-a370-40c8-89f5-6f260bbf234a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:30:52.433794 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.433714    6224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a470b54-ad63-40f7-b306-f8ee1dc8cdcd-istio-envoy\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:30:52.433794 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.433749    6224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a470b54-ad63-40f7-b306-f8ee1dc8cdcd-credential-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:30:52.433794 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.433762    6224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b5884ad-a370-40c8-89f5-6f260bbf234a-workload-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:30:52.433794 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.433775    6224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b5884ad-a370-40c8-89f5-6f260bbf234a-istio-podinfo\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:30:52.433794 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.433788    6224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a470b54-ad63-40f7-b306-f8ee1dc8cdcd-istio-token\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:30:52.433794 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.433798    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2t4h\" (UniqueName: \"kubernetes.io/projected/1a470b54-ad63-40f7-b306-f8ee1dc8cdcd-kube-api-access-j2t4h\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:30:52.434112 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.433822    6224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a470b54-ad63-40f7-b306-f8ee1dc8cdcd-istiod-ca-cert\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:30:52.434112 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.433835    6224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a470b54-ad63-40f7-b306-f8ee1dc8cdcd-istio-podinfo\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:30:52.434112 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.433848    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjt2h\" (UniqueName: \"kubernetes.io/projected/5b5884ad-a370-40c8-89f5-6f260bbf234a-kube-api-access-xjt2h\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:30:52.434112 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.433856    6224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b5884ad-a370-40c8-89f5-6f260bbf234a-istio-token\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:30:52.434112 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.433888    6224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b5884ad-a370-40c8-89f5-6f260bbf234a-credential-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:30:52.434112 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.433896    6224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b5884ad-a370-40c8-89f5-6f260bbf234a-istiod-ca-cert\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:30:52.434112 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.433905    6224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b5884ad-a370-40c8-89f5-6f260bbf234a-istio-envoy\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:30:52.434112 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.433914    6224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b5884ad-a370-40c8-89f5-6f260bbf234a-workload-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:30:52.489240 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.489203    6224 generic.go:356] "Generic (PLEG): container finished" podID="5b5884ad-a370-40c8-89f5-6f260bbf234a" containerID="49b36688e9b6f9c2939fdb838afb3b74f4a069e6a0c550933a7d65189a962bf7" exitCode=0
Oct 30 07:30:52.489410 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.489266    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm"
Oct 30 07:30:52.489410 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.489284    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm" event={"ID":"5b5884ad-a370-40c8-89f5-6f260bbf234a","Type":"ContainerDied","Data":"49b36688e9b6f9c2939fdb838afb3b74f4a069e6a0c550933a7d65189a962bf7"}
Oct 30 07:30:52.489410 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.489384    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm" event={"ID":"5b5884ad-a370-40c8-89f5-6f260bbf234a","Type":"ContainerDied","Data":"a4954a9ae47c36e411c3220113a4537816e055aea72f4eee562c5a8944d43255"}
Oct 30 07:30:52.489575 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.489416    6224 scope.go:122] "RemoveContainer" containerID="49b36688e9b6f9c2939fdb838afb3b74f4a069e6a0c550933a7d65189a962bf7"
Oct 30 07:30:52.493768 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.493740    6224 generic.go:356] "Generic (PLEG): container finished" podID="5611d3ae-2361-411e-903f-ebd0d5b5fba5" containerID="c502885b597dc38e28d29895403865104f6c4cb2cd717c8f66e0f73f10a6ba86" exitCode=0
Oct 30 07:30:52.493947 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.493817    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7"
Oct 30 07:30:52.493947 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.493871    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7" event={"ID":"5611d3ae-2361-411e-903f-ebd0d5b5fba5","Type":"ContainerDied","Data":"c502885b597dc38e28d29895403865104f6c4cb2cd717c8f66e0f73f10a6ba86"}
Oct 30 07:30:52.493947 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.493918    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7" event={"ID":"5611d3ae-2361-411e-903f-ebd0d5b5fba5","Type":"ContainerDied","Data":"d8c375dd046a792866a2cdc53663e5cee97c7c68dedca4c9d99fc389441020cc"}
Oct 30 07:30:52.497290 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.497266    6224 generic.go:356] "Generic (PLEG): container finished" podID="1a470b54-ad63-40f7-b306-f8ee1dc8cdcd" containerID="6d13c4ff0518e5f5b38459897146a543bb2115422d7bfb74102c81906fbbc42f" exitCode=0
Oct 30 07:30:52.497397 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.497346    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl" event={"ID":"1a470b54-ad63-40f7-b306-f8ee1dc8cdcd","Type":"ContainerDied","Data":"6d13c4ff0518e5f5b38459897146a543bb2115422d7bfb74102c81906fbbc42f"}
Oct 30 07:30:52.497397 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.497380    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl" event={"ID":"1a470b54-ad63-40f7-b306-f8ee1dc8cdcd","Type":"ContainerDied","Data":"a68dfdb117487926c4abc34778881f396884fdaff79ea43e5484490d4840a897"}
Oct 30 07:30:52.497530 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.497446    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl"
Oct 30 07:30:52.506937 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.506919    6224 scope.go:122] "RemoveContainer" containerID="49b36688e9b6f9c2939fdb838afb3b74f4a069e6a0c550933a7d65189a962bf7"
Oct 30 07:30:52.507299 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:30:52.507271    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49b36688e9b6f9c2939fdb838afb3b74f4a069e6a0c550933a7d65189a962bf7\": container with ID starting with 49b36688e9b6f9c2939fdb838afb3b74f4a069e6a0c550933a7d65189a962bf7 not found: ID does not exist" containerID="49b36688e9b6f9c2939fdb838afb3b74f4a069e6a0c550933a7d65189a962bf7"
Oct 30 07:30:52.507414 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.507307    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49b36688e9b6f9c2939fdb838afb3b74f4a069e6a0c550933a7d65189a962bf7"} err="failed to get container status \"49b36688e9b6f9c2939fdb838afb3b74f4a069e6a0c550933a7d65189a962bf7\": rpc error: code = NotFound desc = could not find container \"49b36688e9b6f9c2939fdb838afb3b74f4a069e6a0c550933a7d65189a962bf7\": container with ID starting with 49b36688e9b6f9c2939fdb838afb3b74f4a069e6a0c550933a7d65189a962bf7 not found: ID does not exist"
Oct 30 07:30:52.507414 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.507330    6224 scope.go:122] "RemoveContainer" containerID="c502885b597dc38e28d29895403865104f6c4cb2cd717c8f66e0f73f10a6ba86"
Oct 30 07:30:52.519113 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.519096    6224 scope.go:122] "RemoveContainer" containerID="c502885b597dc38e28d29895403865104f6c4cb2cd717c8f66e0f73f10a6ba86"
Oct 30 07:30:52.519340 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:30:52.519323    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c502885b597dc38e28d29895403865104f6c4cb2cd717c8f66e0f73f10a6ba86\": container with ID starting with c502885b597dc38e28d29895403865104f6c4cb2cd717c8f66e0f73f10a6ba86 not found: ID does not exist" containerID="c502885b597dc38e28d29895403865104f6c4cb2cd717c8f66e0f73f10a6ba86"
Oct 30 07:30:52.519425 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.519347    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c502885b597dc38e28d29895403865104f6c4cb2cd717c8f66e0f73f10a6ba86"} err="failed to get container status \"c502885b597dc38e28d29895403865104f6c4cb2cd717c8f66e0f73f10a6ba86\": rpc error: code = NotFound desc = could not find container \"c502885b597dc38e28d29895403865104f6c4cb2cd717c8f66e0f73f10a6ba86\": container with ID starting with c502885b597dc38e28d29895403865104f6c4cb2cd717c8f66e0f73f10a6ba86 not found: ID does not exist"
Oct 30 07:30:52.519425 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.519369    6224 scope.go:122] "RemoveContainer" containerID="6d13c4ff0518e5f5b38459897146a543bb2115422d7bfb74102c81906fbbc42f"
Oct 30 07:30:52.531613 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.531595    6224 scope.go:122] "RemoveContainer" containerID="6d13c4ff0518e5f5b38459897146a543bb2115422d7bfb74102c81906fbbc42f"
Oct 30 07:30:52.531846 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:30:52.531826    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d13c4ff0518e5f5b38459897146a543bb2115422d7bfb74102c81906fbbc42f\": container with ID starting with 6d13c4ff0518e5f5b38459897146a543bb2115422d7bfb74102c81906fbbc42f not found: ID does not exist" containerID="6d13c4ff0518e5f5b38459897146a543bb2115422d7bfb74102c81906fbbc42f"
Oct 30 07:30:52.531900 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.531852    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d13c4ff0518e5f5b38459897146a543bb2115422d7bfb74102c81906fbbc42f"} err="failed to get container status \"6d13c4ff0518e5f5b38459897146a543bb2115422d7bfb74102c81906fbbc42f\": rpc error: code = NotFound desc = could not find container \"6d13c4ff0518e5f5b38459897146a543bb2115422d7bfb74102c81906fbbc42f\": container with ID starting with 6d13c4ff0518e5f5b38459897146a543bb2115422d7bfb74102c81906fbbc42f not found: ID does not exist"
Oct 30 07:30:52.578924 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.578900    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm"]
Oct 30 07:30:52.594384 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.594355    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm"]
Oct 30 07:30:52.682603 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.682575    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl"]
Oct 30 07:30:52.711702 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.711624    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-7jkgl"]
Oct 30 07:30:52.765769 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.765735    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7"]
Oct 30 07:30:52.769389 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:52.769364    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-77jg7"]
Oct 30 07:30:53.146116 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:53.146082    6224 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 07:30:53.146301 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:53.146146    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-497rm" podUID="5b5884ad-a370-40c8-89f5-6f260bbf234a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:30:53.504573 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:53.504488    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a470b54-ad63-40f7-b306-f8ee1dc8cdcd" path="/var/lib/kubelet/pods/1a470b54-ad63-40f7-b306-f8ee1dc8cdcd/volumes"
Oct 30 07:30:53.505256 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:53.505234    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5611d3ae-2361-411e-903f-ebd0d5b5fba5" path="/var/lib/kubelet/pods/5611d3ae-2361-411e-903f-ebd0d5b5fba5/volumes"
Oct 30 07:30:53.505773 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:30:53.505750    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b5884ad-a370-40c8-89f5-6f260bbf234a" path="/var/lib/kubelet/pods/5b5884ad-a370-40c8-89f5-6f260bbf234a/volumes"
Oct 30 07:31:01.861408 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:31:01.861370    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:31:04.602393 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:04.602358    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:31:13.114081 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:13.114045    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45"]
Oct 30 07:31:13.114503 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:13.114257    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45" podUID="04947652-67c7-45de-a2dc-5c2856bfdad8" containerName="istio-proxy" containerID="cri-o://343928d02c00e86c5d9e05408d492bca5b3d5a6897e4ce066a6f675d3c14e862" gracePeriod=30
Oct 30 07:31:17.780403 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:17.780366    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5"]
Oct 30 07:31:17.780932 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:17.780606    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5" podUID="3290bdb2-873c-4502-bb84-e8205fabbedf" containerName="istio-proxy" containerID="cri-o://adb93b29a9d3570a89b11a5c032c2808a0a011118ac49905dc715d1d7d2c11e1" gracePeriod=30
Oct 30 07:31:17.793285 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:17.793255    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g"]
Oct 30 07:31:17.793472 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:17.793437    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g" podUID="6169ad23-c3a7-40de-bfd2-8f232a428238" containerName="istio-proxy" containerID="cri-o://95a0c499d0623ec7ee4f606403bcb87f45ed644f99923fe6e7cd98e0cd6934fd" gracePeriod=30
Oct 30 07:31:18.401583 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:18.401556    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45"
Oct 30 07:31:18.508488 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:18.508456    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04947652-67c7-45de-a2dc-5c2856bfdad8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04947652-67c7-45de-a2dc-5c2856bfdad8-credential-socket\") pod \"04947652-67c7-45de-a2dc-5c2856bfdad8\" (UID: \"04947652-67c7-45de-a2dc-5c2856bfdad8\") "
Oct 30 07:31:18.508673 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:18.508508    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04947652-67c7-45de-a2dc-5c2856bfdad8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04947652-67c7-45de-a2dc-5c2856bfdad8-istio-data\") pod \"04947652-67c7-45de-a2dc-5c2856bfdad8\" (UID: \"04947652-67c7-45de-a2dc-5c2856bfdad8\") "
Oct 30 07:31:18.508673 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:18.508566    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04947652-67c7-45de-a2dc-5c2856bfdad8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04947652-67c7-45de-a2dc-5c2856bfdad8-workload-certs\") pod \"04947652-67c7-45de-a2dc-5c2856bfdad8\" (UID: \"04947652-67c7-45de-a2dc-5c2856bfdad8\") "
Oct 30 07:31:18.508673 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:18.508665    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04947652-67c7-45de-a2dc-5c2856bfdad8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04947652-67c7-45de-a2dc-5c2856bfdad8-workload-socket\") pod \"04947652-67c7-45de-a2dc-5c2856bfdad8\" (UID: \"04947652-67c7-45de-a2dc-5c2856bfdad8\") "
Oct 30 07:31:18.508899 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:18.508705    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/04947652-67c7-45de-a2dc-5c2856bfdad8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04947652-67c7-45de-a2dc-5c2856bfdad8-istio-podinfo\") pod \"04947652-67c7-45de-a2dc-5c2856bfdad8\" (UID: \"04947652-67c7-45de-a2dc-5c2856bfdad8\") "
Oct 30 07:31:18.508899 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:18.508741    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04947652-67c7-45de-a2dc-5c2856bfdad8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04947652-67c7-45de-a2dc-5c2856bfdad8-istio-envoy\") pod \"04947652-67c7-45de-a2dc-5c2856bfdad8\" (UID: \"04947652-67c7-45de-a2dc-5c2856bfdad8\") "
Oct 30 07:31:18.508899 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:18.508776    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04947652-67c7-45de-a2dc-5c2856bfdad8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04947652-67c7-45de-a2dc-5c2856bfdad8-istiod-ca-cert\") pod \"04947652-67c7-45de-a2dc-5c2856bfdad8\" (UID: \"04947652-67c7-45de-a2dc-5c2856bfdad8\") "
Oct 30 07:31:18.508899 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:18.508857    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04947652-67c7-45de-a2dc-5c2856bfdad8-kube-api-access-44c6m\" (UniqueName: \"kubernetes.io/projected/04947652-67c7-45de-a2dc-5c2856bfdad8-kube-api-access-44c6m\") pod \"04947652-67c7-45de-a2dc-5c2856bfdad8\" (UID: \"04947652-67c7-45de-a2dc-5c2856bfdad8\") "
Oct 30 07:31:18.509121 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:18.508875    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04947652-67c7-45de-a2dc-5c2856bfdad8-istio-data" pod "04947652-67c7-45de-a2dc-5c2856bfdad8" (UID: "04947652-67c7-45de-a2dc-5c2856bfdad8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:31:18.509121 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:18.508918    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04947652-67c7-45de-a2dc-5c2856bfdad8-istio-token\" (UniqueName: \"kubernetes.io/projected/04947652-67c7-45de-a2dc-5c2856bfdad8-istio-token\") pod \"04947652-67c7-45de-a2dc-5c2856bfdad8\" (UID: \"04947652-67c7-45de-a2dc-5c2856bfdad8\") "
Oct 30 07:31:18.509121 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:18.508950    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04947652-67c7-45de-a2dc-5c2856bfdad8-workload-certs" pod "04947652-67c7-45de-a2dc-5c2856bfdad8" (UID: "04947652-67c7-45de-a2dc-5c2856bfdad8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:31:18.509121 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:18.509037    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04947652-67c7-45de-a2dc-5c2856bfdad8-workload-socket" pod "04947652-67c7-45de-a2dc-5c2856bfdad8" (UID: "04947652-67c7-45de-a2dc-5c2856bfdad8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:31:18.510052 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:18.509559    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04947652-67c7-45de-a2dc-5c2856bfdad8-credential-socket" pod "04947652-67c7-45de-a2dc-5c2856bfdad8" (UID: "04947652-67c7-45de-a2dc-5c2856bfdad8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:31:18.510052 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:18.510020    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04947652-67c7-45de-a2dc-5c2856bfdad8-istiod-ca-cert" pod "04947652-67c7-45de-a2dc-5c2856bfdad8" (UID: "04947652-67c7-45de-a2dc-5c2856bfdad8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:31:18.511089 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:18.511064    6224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04947652-67c7-45de-a2dc-5c2856bfdad8-credential-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:31:18.511229 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:18.511100    6224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04947652-67c7-45de-a2dc-5c2856bfdad8-istio-data\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:31:18.511229 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:18.511117    6224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04947652-67c7-45de-a2dc-5c2856bfdad8-workload-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:31:18.511229 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:18.511129    6224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04947652-67c7-45de-a2dc-5c2856bfdad8-workload-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:31:18.518961 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:18.511997    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04947652-67c7-45de-a2dc-5c2856bfdad8-istio-envoy" pod "04947652-67c7-45de-a2dc-5c2856bfdad8" (UID: "04947652-67c7-45de-a2dc-5c2856bfdad8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:31:18.518961 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:18.512063    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04947652-67c7-45de-a2dc-5c2856bfdad8-istio-token" pod "04947652-67c7-45de-a2dc-5c2856bfdad8" (UID: "04947652-67c7-45de-a2dc-5c2856bfdad8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:31:18.518961 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:18.512597    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04947652-67c7-45de-a2dc-5c2856bfdad8-istio-podinfo" pod "04947652-67c7-45de-a2dc-5c2856bfdad8" (UID: "04947652-67c7-45de-a2dc-5c2856bfdad8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:31:18.523249 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:18.523219    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04947652-67c7-45de-a2dc-5c2856bfdad8-kube-api-access-44c6m" pod "04947652-67c7-45de-a2dc-5c2856bfdad8" (UID: "04947652-67c7-45de-a2dc-5c2856bfdad8"). InnerVolumeSpecName "kube-api-access-44c6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:31:18.613052 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:18.613012    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44c6m\" (UniqueName: \"kubernetes.io/projected/04947652-67c7-45de-a2dc-5c2856bfdad8-kube-api-access-44c6m\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:31:18.613052 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:18.613045    6224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04947652-67c7-45de-a2dc-5c2856bfdad8-istio-token\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:31:18.613052 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:18.613054    6224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04947652-67c7-45de-a2dc-5c2856bfdad8-istio-podinfo\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:31:18.613052 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:18.613062    6224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04947652-67c7-45de-a2dc-5c2856bfdad8-istio-envoy\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:31:18.613404 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:18.613071    6224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04947652-67c7-45de-a2dc-5c2856bfdad8-istiod-ca-cert\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:31:18.793324 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:18.793294    6224 generic.go:356] "Generic (PLEG): container finished" podID="04947652-67c7-45de-a2dc-5c2856bfdad8" containerID="343928d02c00e86c5d9e05408d492bca5b3d5a6897e4ce066a6f675d3c14e862" exitCode=0
Oct 30 07:31:18.793704 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:18.793351    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45"
Oct 30 07:31:18.793704 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:18.793377    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45" event={"ID":"04947652-67c7-45de-a2dc-5c2856bfdad8","Type":"ContainerDied","Data":"343928d02c00e86c5d9e05408d492bca5b3d5a6897e4ce066a6f675d3c14e862"}
Oct 30 07:31:18.793704 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:18.793465    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45" event={"ID":"04947652-67c7-45de-a2dc-5c2856bfdad8","Type":"ContainerDied","Data":"82ba5abeec7ebcc9d1df96f20909ea3a73edb40e85ecd63cc4857b6a4601453a"}
Oct 30 07:31:18.793704 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:18.793492    6224 scope.go:122] "RemoveContainer" containerID="343928d02c00e86c5d9e05408d492bca5b3d5a6897e4ce066a6f675d3c14e862"
Oct 30 07:31:18.808992 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:18.808973    6224 scope.go:122] "RemoveContainer" containerID="343928d02c00e86c5d9e05408d492bca5b3d5a6897e4ce066a6f675d3c14e862"
Oct 30 07:31:18.809353 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:31:18.809289    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"343928d02c00e86c5d9e05408d492bca5b3d5a6897e4ce066a6f675d3c14e862\": container with ID starting with 343928d02c00e86c5d9e05408d492bca5b3d5a6897e4ce066a6f675d3c14e862 not found: ID does not exist" containerID="343928d02c00e86c5d9e05408d492bca5b3d5a6897e4ce066a6f675d3c14e862"
Oct 30 07:31:18.809491 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:18.809369    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"343928d02c00e86c5d9e05408d492bca5b3d5a6897e4ce066a6f675d3c14e862"} err="failed to get container status \"343928d02c00e86c5d9e05408d492bca5b3d5a6897e4ce066a6f675d3c14e862\": rpc error: code = NotFound desc = could not find container \"343928d02c00e86c5d9e05408d492bca5b3d5a6897e4ce066a6f675d3c14e862\": container with ID starting with 343928d02c00e86c5d9e05408d492bca5b3d5a6897e4ce066a6f675d3c14e862 not found: ID does not exist"
Oct 30 07:31:18.855650 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:18.855609    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45"]
Oct 30 07:31:18.869394 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:18.869366    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-vfd45"]
Oct 30 07:31:19.503607 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:19.503574    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04947652-67c7-45de-a2dc-5c2856bfdad8" path="/var/lib/kubelet/pods/04947652-67c7-45de-a2dc-5c2856bfdad8/volumes"
Oct 30 07:31:23.071588 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.071559    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5"
Oct 30 07:31:23.116625 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.116596    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g"
Oct 30 07:31:23.180222 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.180193    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6169ad23-c3a7-40de-bfd2-8f232a428238-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6169ad23-c3a7-40de-bfd2-8f232a428238-istio-envoy\") pod \"6169ad23-c3a7-40de-bfd2-8f232a428238\" (UID: \"6169ad23-c3a7-40de-bfd2-8f232a428238\") "
Oct 30 07:31:23.180406 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.180247    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3290bdb2-873c-4502-bb84-e8205fabbedf-kube-api-access-hkcd4\" (UniqueName: \"kubernetes.io/projected/3290bdb2-873c-4502-bb84-e8205fabbedf-kube-api-access-hkcd4\") pod \"3290bdb2-873c-4502-bb84-e8205fabbedf\" (UID: \"3290bdb2-873c-4502-bb84-e8205fabbedf\") "
Oct 30 07:31:23.180406 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.180286    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6169ad23-c3a7-40de-bfd2-8f232a428238-kube-api-access-pwpk5\" (UniqueName: \"kubernetes.io/projected/6169ad23-c3a7-40de-bfd2-8f232a428238-kube-api-access-pwpk5\") pod \"6169ad23-c3a7-40de-bfd2-8f232a428238\" (UID: \"6169ad23-c3a7-40de-bfd2-8f232a428238\") "
Oct 30 07:31:23.180406 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.180313    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6169ad23-c3a7-40de-bfd2-8f232a428238-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6169ad23-c3a7-40de-bfd2-8f232a428238-workload-socket\") pod \"6169ad23-c3a7-40de-bfd2-8f232a428238\" (UID: \"6169ad23-c3a7-40de-bfd2-8f232a428238\") "
Oct 30 07:31:23.180406 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.180384    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3290bdb2-873c-4502-bb84-e8205fabbedf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3290bdb2-873c-4502-bb84-e8205fabbedf-istiod-ca-cert\") pod \"3290bdb2-873c-4502-bb84-e8205fabbedf\" (UID: \"3290bdb2-873c-4502-bb84-e8205fabbedf\") "
Oct 30 07:31:23.180653 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.180434    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3290bdb2-873c-4502-bb84-e8205fabbedf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3290bdb2-873c-4502-bb84-e8205fabbedf-istio-envoy\") pod \"3290bdb2-873c-4502-bb84-e8205fabbedf\" (UID: \"3290bdb2-873c-4502-bb84-e8205fabbedf\") "
Oct 30 07:31:23.180653 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.180472    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6169ad23-c3a7-40de-bfd2-8f232a428238-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6169ad23-c3a7-40de-bfd2-8f232a428238-istiod-ca-cert\") pod \"6169ad23-c3a7-40de-bfd2-8f232a428238\" (UID: \"6169ad23-c3a7-40de-bfd2-8f232a428238\") "
Oct 30 07:31:23.180653 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.180507    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3290bdb2-873c-4502-bb84-e8205fabbedf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3290bdb2-873c-4502-bb84-e8205fabbedf-credential-socket\") pod \"3290bdb2-873c-4502-bb84-e8205fabbedf\" (UID: \"3290bdb2-873c-4502-bb84-e8205fabbedf\") "
Oct 30 07:31:23.180653 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.180557    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3290bdb2-873c-4502-bb84-e8205fabbedf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3290bdb2-873c-4502-bb84-e8205fabbedf-istio-podinfo\") pod \"3290bdb2-873c-4502-bb84-e8205fabbedf\" (UID: \"3290bdb2-873c-4502-bb84-e8205fabbedf\") "
Oct 30 07:31:23.180653 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.180600    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6169ad23-c3a7-40de-bfd2-8f232a428238-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6169ad23-c3a7-40de-bfd2-8f232a428238-istio-data\") pod \"6169ad23-c3a7-40de-bfd2-8f232a428238\" (UID: \"6169ad23-c3a7-40de-bfd2-8f232a428238\") "
Oct 30 07:31:23.180934 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.180719    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6169ad23-c3a7-40de-bfd2-8f232a428238-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6169ad23-c3a7-40de-bfd2-8f232a428238-workload-certs\") pod \"6169ad23-c3a7-40de-bfd2-8f232a428238\" (UID: \"6169ad23-c3a7-40de-bfd2-8f232a428238\") "
Oct 30 07:31:23.180934 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.180779    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6169ad23-c3a7-40de-bfd2-8f232a428238-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6169ad23-c3a7-40de-bfd2-8f232a428238-istio-podinfo\") pod \"6169ad23-c3a7-40de-bfd2-8f232a428238\" (UID: \"6169ad23-c3a7-40de-bfd2-8f232a428238\") "
Oct 30 07:31:23.180934 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.180850    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6169ad23-c3a7-40de-bfd2-8f232a428238-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6169ad23-c3a7-40de-bfd2-8f232a428238-credential-socket\") pod \"6169ad23-c3a7-40de-bfd2-8f232a428238\" (UID: \"6169ad23-c3a7-40de-bfd2-8f232a428238\") "
Oct 30 07:31:23.181091 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.180953    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3290bdb2-873c-4502-bb84-e8205fabbedf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3290bdb2-873c-4502-bb84-e8205fabbedf-workload-certs\") pod \"3290bdb2-873c-4502-bb84-e8205fabbedf\" (UID: \"3290bdb2-873c-4502-bb84-e8205fabbedf\") "
Oct 30 07:31:23.181091 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.180981    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3290bdb2-873c-4502-bb84-e8205fabbedf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3290bdb2-873c-4502-bb84-e8205fabbedf-workload-socket\") pod \"3290bdb2-873c-4502-bb84-e8205fabbedf\" (UID: \"3290bdb2-873c-4502-bb84-e8205fabbedf\") "
Oct 30 07:31:23.181091 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.181014    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6169ad23-c3a7-40de-bfd2-8f232a428238-istio-token\" (UniqueName: \"kubernetes.io/projected/6169ad23-c3a7-40de-bfd2-8f232a428238-istio-token\") pod \"6169ad23-c3a7-40de-bfd2-8f232a428238\" (UID: \"6169ad23-c3a7-40de-bfd2-8f232a428238\") "
Oct 30 07:31:23.181091 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.181080    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3290bdb2-873c-4502-bb84-e8205fabbedf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3290bdb2-873c-4502-bb84-e8205fabbedf-istio-data\") pod \"3290bdb2-873c-4502-bb84-e8205fabbedf\" (UID: \"3290bdb2-873c-4502-bb84-e8205fabbedf\") "
Oct 30 07:31:23.181312 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.181129    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3290bdb2-873c-4502-bb84-e8205fabbedf-istio-token\" (UniqueName: \"kubernetes.io/projected/3290bdb2-873c-4502-bb84-e8205fabbedf-istio-token\") pod \"3290bdb2-873c-4502-bb84-e8205fabbedf\" (UID: \"3290bdb2-873c-4502-bb84-e8205fabbedf\") "
Oct 30 07:31:23.181443 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.181414    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6169ad23-c3a7-40de-bfd2-8f232a428238-workload-socket" pod "6169ad23-c3a7-40de-bfd2-8f232a428238" (UID: "6169ad23-c3a7-40de-bfd2-8f232a428238"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:31:23.181947 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.181918    6224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6169ad23-c3a7-40de-bfd2-8f232a428238-workload-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:31:23.182298 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.182273    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3290bdb2-873c-4502-bb84-e8205fabbedf-istiod-ca-cert" pod "3290bdb2-873c-4502-bb84-e8205fabbedf" (UID: "3290bdb2-873c-4502-bb84-e8205fabbedf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:31:23.182618 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.182594    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6169ad23-c3a7-40de-bfd2-8f232a428238-credential-socket" pod "6169ad23-c3a7-40de-bfd2-8f232a428238" (UID: "6169ad23-c3a7-40de-bfd2-8f232a428238"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:31:23.182937 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.182912    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3290bdb2-873c-4502-bb84-e8205fabbedf-workload-socket" pod "3290bdb2-873c-4502-bb84-e8205fabbedf" (UID: "3290bdb2-873c-4502-bb84-e8205fabbedf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:31:23.183161 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.183138    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3290bdb2-873c-4502-bb84-e8205fabbedf-workload-certs" pod "3290bdb2-873c-4502-bb84-e8205fabbedf" (UID: "3290bdb2-873c-4502-bb84-e8205fabbedf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:31:23.183360 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.183339    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3290bdb2-873c-4502-bb84-e8205fabbedf-istio-data" pod "3290bdb2-873c-4502-bb84-e8205fabbedf" (UID: "3290bdb2-873c-4502-bb84-e8205fabbedf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:31:23.183605 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.183577    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3290bdb2-873c-4502-bb84-e8205fabbedf-istio-token" pod "3290bdb2-873c-4502-bb84-e8205fabbedf" (UID: "3290bdb2-873c-4502-bb84-e8205fabbedf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:31:23.183794 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.183777    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6169ad23-c3a7-40de-bfd2-8f232a428238-istio-data" pod "6169ad23-c3a7-40de-bfd2-8f232a428238" (UID: "6169ad23-c3a7-40de-bfd2-8f232a428238"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:31:23.183915 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.183855    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6169ad23-c3a7-40de-bfd2-8f232a428238-istio-podinfo" pod "6169ad23-c3a7-40de-bfd2-8f232a428238" (UID: "6169ad23-c3a7-40de-bfd2-8f232a428238"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:31:23.184044 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.183973    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6169ad23-c3a7-40de-bfd2-8f232a428238-workload-certs" pod "6169ad23-c3a7-40de-bfd2-8f232a428238" (UID: "6169ad23-c3a7-40de-bfd2-8f232a428238"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:31:23.184159 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.184050    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3290bdb2-873c-4502-bb84-e8205fabbedf-credential-socket" pod "3290bdb2-873c-4502-bb84-e8205fabbedf" (UID: "3290bdb2-873c-4502-bb84-e8205fabbedf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:31:23.184159 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.184094    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6169ad23-c3a7-40de-bfd2-8f232a428238-istiod-ca-cert" pod "6169ad23-c3a7-40de-bfd2-8f232a428238" (UID: "6169ad23-c3a7-40de-bfd2-8f232a428238"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:31:23.184471 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.184444    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6169ad23-c3a7-40de-bfd2-8f232a428238-istio-envoy" pod "6169ad23-c3a7-40de-bfd2-8f232a428238" (UID: "6169ad23-c3a7-40de-bfd2-8f232a428238"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:31:23.184587 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.184540    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3290bdb2-873c-4502-bb84-e8205fabbedf-kube-api-access-hkcd4" pod "3290bdb2-873c-4502-bb84-e8205fabbedf" (UID: "3290bdb2-873c-4502-bb84-e8205fabbedf"). InnerVolumeSpecName "kube-api-access-hkcd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:31:23.185668 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.185634    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3290bdb2-873c-4502-bb84-e8205fabbedf-istio-envoy" pod "3290bdb2-873c-4502-bb84-e8205fabbedf" (UID: "3290bdb2-873c-4502-bb84-e8205fabbedf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:31:23.186070 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.186039    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6169ad23-c3a7-40de-bfd2-8f232a428238-kube-api-access-pwpk5" pod "6169ad23-c3a7-40de-bfd2-8f232a428238" (UID: "6169ad23-c3a7-40de-bfd2-8f232a428238"). InnerVolumeSpecName "kube-api-access-pwpk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:31:23.186294 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.186265    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6169ad23-c3a7-40de-bfd2-8f232a428238-istio-token" pod "6169ad23-c3a7-40de-bfd2-8f232a428238" (UID: "6169ad23-c3a7-40de-bfd2-8f232a428238"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:31:23.187090 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.187068    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3290bdb2-873c-4502-bb84-e8205fabbedf-istio-podinfo" pod "3290bdb2-873c-4502-bb84-e8205fabbedf" (UID: "3290bdb2-873c-4502-bb84-e8205fabbedf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:31:23.269378 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.269347    6224 scope.go:122] "RemoveContainer" containerID="1dc32a4e513facb27140c6505e4e2bb8b313e6ece9283b7f8902dbb9f77472b5"
Oct 30 07:31:23.283381 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.283356    6224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3290bdb2-873c-4502-bb84-e8205fabbedf-workload-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:31:23.283381 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.283382    6224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3290bdb2-873c-4502-bb84-e8205fabbedf-workload-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:31:23.283518 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.283391    6224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6169ad23-c3a7-40de-bfd2-8f232a428238-istio-token\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:31:23.283518 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.283400    6224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3290bdb2-873c-4502-bb84-e8205fabbedf-istio-data\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:31:23.283518 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.283408    6224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3290bdb2-873c-4502-bb84-e8205fabbedf-istio-token\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:31:23.283518 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.283416    6224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6169ad23-c3a7-40de-bfd2-8f232a428238-istio-envoy\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:31:23.283518 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.283424    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkcd4\" (UniqueName: \"kubernetes.io/projected/3290bdb2-873c-4502-bb84-e8205fabbedf-kube-api-access-hkcd4\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:31:23.283518 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.283433    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwpk5\" (UniqueName: \"kubernetes.io/projected/6169ad23-c3a7-40de-bfd2-8f232a428238-kube-api-access-pwpk5\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:31:23.283518 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.283443    6224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3290bdb2-873c-4502-bb84-e8205fabbedf-istiod-ca-cert\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:31:23.283518 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.283450    6224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3290bdb2-873c-4502-bb84-e8205fabbedf-istio-envoy\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:31:23.283518 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.283458    6224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6169ad23-c3a7-40de-bfd2-8f232a428238-istiod-ca-cert\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:31:23.283518 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.283466    6224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3290bdb2-873c-4502-bb84-e8205fabbedf-credential-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:31:23.283518 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.283474    6224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3290bdb2-873c-4502-bb84-e8205fabbedf-istio-podinfo\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:31:23.283518 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.283481    6224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6169ad23-c3a7-40de-bfd2-8f232a428238-istio-data\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:31:23.283518 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.283490    6224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6169ad23-c3a7-40de-bfd2-8f232a428238-workload-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:31:23.283518 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.283498    6224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6169ad23-c3a7-40de-bfd2-8f232a428238-istio-podinfo\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:31:23.283518 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.283506    6224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6169ad23-c3a7-40de-bfd2-8f232a428238-credential-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:31:23.831568 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.831529    6224 generic.go:356] "Generic (PLEG): container finished" podID="6169ad23-c3a7-40de-bfd2-8f232a428238" containerID="95a0c499d0623ec7ee4f606403bcb87f45ed644f99923fe6e7cd98e0cd6934fd" exitCode=0
Oct 30 07:31:23.831723 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.831582    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g"
Oct 30 07:31:23.831723 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.831675    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g" event={"ID":"6169ad23-c3a7-40de-bfd2-8f232a428238","Type":"ContainerDied","Data":"95a0c499d0623ec7ee4f606403bcb87f45ed644f99923fe6e7cd98e0cd6934fd"}
Oct 30 07:31:23.831902 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.831735    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g" event={"ID":"6169ad23-c3a7-40de-bfd2-8f232a428238","Type":"ContainerDied","Data":"36f2843cf3cfd54545508a06328a139c17fe6ab3e7323fcfbba36e1152ffa1ce"}
Oct 30 07:31:23.831902 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.831771    6224 scope.go:122] "RemoveContainer" containerID="95a0c499d0623ec7ee4f606403bcb87f45ed644f99923fe6e7cd98e0cd6934fd"
Oct 30 07:31:23.833488 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.833465    6224 generic.go:356] "Generic (PLEG): container finished" podID="3290bdb2-873c-4502-bb84-e8205fabbedf" containerID="adb93b29a9d3570a89b11a5c032c2808a0a011118ac49905dc715d1d7d2c11e1" exitCode=0
Oct 30 07:31:23.833776 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.833607    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5"
Oct 30 07:31:23.833776 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.833641    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5" event={"ID":"3290bdb2-873c-4502-bb84-e8205fabbedf","Type":"ContainerDied","Data":"adb93b29a9d3570a89b11a5c032c2808a0a011118ac49905dc715d1d7d2c11e1"}
Oct 30 07:31:23.833776 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.833687    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5" event={"ID":"3290bdb2-873c-4502-bb84-e8205fabbedf","Type":"ContainerDied","Data":"f048f97f854335f9aa1be896d5a41e05432486a34061af69b98e77b95190533b"}
Oct 30 07:31:23.842437 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.842419    6224 scope.go:122] "RemoveContainer" containerID="95a0c499d0623ec7ee4f606403bcb87f45ed644f99923fe6e7cd98e0cd6934fd"
Oct 30 07:31:23.842695 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:31:23.842675    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95a0c499d0623ec7ee4f606403bcb87f45ed644f99923fe6e7cd98e0cd6934fd\": container with ID starting with 95a0c499d0623ec7ee4f606403bcb87f45ed644f99923fe6e7cd98e0cd6934fd not found: ID does not exist" containerID="95a0c499d0623ec7ee4f606403bcb87f45ed644f99923fe6e7cd98e0cd6934fd"
Oct 30 07:31:23.842747 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.842702    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95a0c499d0623ec7ee4f606403bcb87f45ed644f99923fe6e7cd98e0cd6934fd"} err="failed to get container status \"95a0c499d0623ec7ee4f606403bcb87f45ed644f99923fe6e7cd98e0cd6934fd\": rpc error: code = NotFound desc = could not find container \"95a0c499d0623ec7ee4f606403bcb87f45ed644f99923fe6e7cd98e0cd6934fd\": container with ID starting with 95a0c499d0623ec7ee4f606403bcb87f45ed644f99923fe6e7cd98e0cd6934fd not found: ID does not exist"
Oct 30 07:31:23.842747 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.842719    6224 scope.go:122] "RemoveContainer" containerID="adb93b29a9d3570a89b11a5c032c2808a0a011118ac49905dc715d1d7d2c11e1"
Oct 30 07:31:23.855243 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.855220    6224 scope.go:122] "RemoveContainer" containerID="adb93b29a9d3570a89b11a5c032c2808a0a011118ac49905dc715d1d7d2c11e1"
Oct 30 07:31:23.855697 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:31:23.855669    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adb93b29a9d3570a89b11a5c032c2808a0a011118ac49905dc715d1d7d2c11e1\": container with ID starting with adb93b29a9d3570a89b11a5c032c2808a0a011118ac49905dc715d1d7d2c11e1 not found: ID does not exist" containerID="adb93b29a9d3570a89b11a5c032c2808a0a011118ac49905dc715d1d7d2c11e1"
Oct 30 07:31:23.855828 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.855706    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adb93b29a9d3570a89b11a5c032c2808a0a011118ac49905dc715d1d7d2c11e1"} err="failed to get container status \"adb93b29a9d3570a89b11a5c032c2808a0a011118ac49905dc715d1d7d2c11e1\": rpc error: code = NotFound desc = could not find container \"adb93b29a9d3570a89b11a5c032c2808a0a011118ac49905dc715d1d7d2c11e1\": container with ID starting with adb93b29a9d3570a89b11a5c032c2808a0a011118ac49905dc715d1d7d2c11e1 not found: ID does not exist"
Oct 30 07:31:23.896682 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.896653    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g"]
Oct 30 07:31:23.913028 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.913002    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-g2x5g"]
Oct 30 07:31:23.952717 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.952683    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5"]
Oct 30 07:31:23.958068 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:23.958042    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-hfvs5"]
Oct 30 07:31:25.513688 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:25.513652    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3290bdb2-873c-4502-bb84-e8205fabbedf" path="/var/lib/kubelet/pods/3290bdb2-873c-4502-bb84-e8205fabbedf/volumes"
Oct 30 07:31:25.514147 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:25.514132    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6169ad23-c3a7-40de-bfd2-8f232a428238" path="/var/lib/kubelet/pods/6169ad23-c3a7-40de-bfd2-8f232a428238/volumes"
Oct 30 07:31:31.196685 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:31.196644    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-14271d/auth-test-model-14271d-kserve-6c67747f77-k8psp"]
Oct 30 07:31:31.869138 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:31:31.869092    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:31:32.517675 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:32.517643    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp"]
Oct 30 07:31:32.518126 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:32.517830    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp" podUID="c5fe37b6-8794-4108-aee2-5da540d6bc0c" containerName="istio-proxy" containerID="cri-o://5ed20299d171217c79fd9550036ae9d2b0bd263d98e7fc9741170c0b5a3a55ce" gracePeriod=30
Oct 30 07:31:37.805385 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:37.805355    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp"
Oct 30 07:31:37.947682 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:37.947647    6224 generic.go:356] "Generic (PLEG): container finished" podID="c5fe37b6-8794-4108-aee2-5da540d6bc0c" containerID="5ed20299d171217c79fd9550036ae9d2b0bd263d98e7fc9741170c0b5a3a55ce" exitCode=0
Oct 30 07:31:37.947887 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:37.947695    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp"
Oct 30 07:31:37.947887 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:37.947738    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp" event={"ID":"c5fe37b6-8794-4108-aee2-5da540d6bc0c","Type":"ContainerDied","Data":"5ed20299d171217c79fd9550036ae9d2b0bd263d98e7fc9741170c0b5a3a55ce"}
Oct 30 07:31:37.947887 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:37.947796    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp" event={"ID":"c5fe37b6-8794-4108-aee2-5da540d6bc0c","Type":"ContainerDied","Data":"76dd169ae8fa2a2a4327fb9bfa4c2cae1bf22bb1f24f3d2163a23b4e8d4a2df5"}
Oct 30 07:31:37.947887 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:37.947843    6224 scope.go:122] "RemoveContainer" containerID="5ed20299d171217c79fd9550036ae9d2b0bd263d98e7fc9741170c0b5a3a55ce"
Oct 30 07:31:37.958292 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:37.958261    6224 scope.go:122] "RemoveContainer" containerID="5ed20299d171217c79fd9550036ae9d2b0bd263d98e7fc9741170c0b5a3a55ce"
Oct 30 07:31:37.958640 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:31:37.958618    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ed20299d171217c79fd9550036ae9d2b0bd263d98e7fc9741170c0b5a3a55ce\": container with ID starting with 5ed20299d171217c79fd9550036ae9d2b0bd263d98e7fc9741170c0b5a3a55ce not found: ID does not exist" containerID="5ed20299d171217c79fd9550036ae9d2b0bd263d98e7fc9741170c0b5a3a55ce"
Oct 30 07:31:37.958722 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:37.958652    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ed20299d171217c79fd9550036ae9d2b0bd263d98e7fc9741170c0b5a3a55ce"} err="failed to get container status \"5ed20299d171217c79fd9550036ae9d2b0bd263d98e7fc9741170c0b5a3a55ce\": rpc error: code = NotFound desc = could not find container \"5ed20299d171217c79fd9550036ae9d2b0bd263d98e7fc9741170c0b5a3a55ce\": container with ID starting with 5ed20299d171217c79fd9550036ae9d2b0bd263d98e7fc9741170c0b5a3a55ce not found: ID does not exist"
Oct 30 07:31:37.987250 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:37.987217    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5fe37b6-8794-4108-aee2-5da540d6bc0c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5fe37b6-8794-4108-aee2-5da540d6bc0c-credential-socket\") pod \"c5fe37b6-8794-4108-aee2-5da540d6bc0c\" (UID: \"c5fe37b6-8794-4108-aee2-5da540d6bc0c\") "
Oct 30 07:31:37.987441 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:37.987267    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5fe37b6-8794-4108-aee2-5da540d6bc0c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5fe37b6-8794-4108-aee2-5da540d6bc0c-workload-socket\") pod \"c5fe37b6-8794-4108-aee2-5da540d6bc0c\" (UID: \"c5fe37b6-8794-4108-aee2-5da540d6bc0c\") "
Oct 30 07:31:37.987441 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:37.987402    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5fe37b6-8794-4108-aee2-5da540d6bc0c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5fe37b6-8794-4108-aee2-5da540d6bc0c-istio-podinfo\") pod \"c5fe37b6-8794-4108-aee2-5da540d6bc0c\" (UID: \"c5fe37b6-8794-4108-aee2-5da540d6bc0c\") "
Oct 30 07:31:37.987576 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:37.987452    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5fe37b6-8794-4108-aee2-5da540d6bc0c-kube-api-access-r2f44\" (UniqueName: \"kubernetes.io/projected/c5fe37b6-8794-4108-aee2-5da540d6bc0c-kube-api-access-r2f44\") pod \"c5fe37b6-8794-4108-aee2-5da540d6bc0c\" (UID: \"c5fe37b6-8794-4108-aee2-5da540d6bc0c\") "
Oct 30 07:31:37.987576 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:37.987485    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5fe37b6-8794-4108-aee2-5da540d6bc0c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5fe37b6-8794-4108-aee2-5da540d6bc0c-istio-envoy\") pod \"c5fe37b6-8794-4108-aee2-5da540d6bc0c\" (UID: \"c5fe37b6-8794-4108-aee2-5da540d6bc0c\") "
Oct 30 07:31:37.987576 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:37.987518    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5fe37b6-8794-4108-aee2-5da540d6bc0c-istio-token\" (UniqueName: \"kubernetes.io/projected/c5fe37b6-8794-4108-aee2-5da540d6bc0c-istio-token\") pod \"c5fe37b6-8794-4108-aee2-5da540d6bc0c\" (UID: \"c5fe37b6-8794-4108-aee2-5da540d6bc0c\") "
Oct 30 07:31:37.987576 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:37.987531    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5fe37b6-8794-4108-aee2-5da540d6bc0c-credential-socket" pod "c5fe37b6-8794-4108-aee2-5da540d6bc0c" (UID: "c5fe37b6-8794-4108-aee2-5da540d6bc0c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:31:37.987576 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:37.987545    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5fe37b6-8794-4108-aee2-5da540d6bc0c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5fe37b6-8794-4108-aee2-5da540d6bc0c-workload-certs\") pod \"c5fe37b6-8794-4108-aee2-5da540d6bc0c\" (UID: \"c5fe37b6-8794-4108-aee2-5da540d6bc0c\") "
Oct 30 07:31:37.987856 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:37.987606    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5fe37b6-8794-4108-aee2-5da540d6bc0c-workload-socket" pod "c5fe37b6-8794-4108-aee2-5da540d6bc0c" (UID: "c5fe37b6-8794-4108-aee2-5da540d6bc0c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:31:37.987856 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:37.987639    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5fe37b6-8794-4108-aee2-5da540d6bc0c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5fe37b6-8794-4108-aee2-5da540d6bc0c-istiod-ca-cert\") pod \"c5fe37b6-8794-4108-aee2-5da540d6bc0c\" (UID: \"c5fe37b6-8794-4108-aee2-5da540d6bc0c\") "
Oct 30 07:31:37.987856 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:37.987736    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5fe37b6-8794-4108-aee2-5da540d6bc0c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5fe37b6-8794-4108-aee2-5da540d6bc0c-istio-data\") pod \"c5fe37b6-8794-4108-aee2-5da540d6bc0c\" (UID: \"c5fe37b6-8794-4108-aee2-5da540d6bc0c\") "
Oct 30 07:31:37.988013 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:37.987907    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5fe37b6-8794-4108-aee2-5da540d6bc0c-workload-certs" pod "c5fe37b6-8794-4108-aee2-5da540d6bc0c" (UID: "c5fe37b6-8794-4108-aee2-5da540d6bc0c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:31:37.988142 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:37.988117    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5fe37b6-8794-4108-aee2-5da540d6bc0c-istio-data" pod "c5fe37b6-8794-4108-aee2-5da540d6bc0c" (UID: "c5fe37b6-8794-4108-aee2-5da540d6bc0c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:31:37.988184 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:37.988164    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5fe37b6-8794-4108-aee2-5da540d6bc0c-istiod-ca-cert" pod "c5fe37b6-8794-4108-aee2-5da540d6bc0c" (UID: "c5fe37b6-8794-4108-aee2-5da540d6bc0c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:31:37.988421 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:37.988397    6224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5fe37b6-8794-4108-aee2-5da540d6bc0c-istiod-ca-cert\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:31:37.988918 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:37.988427    6224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5fe37b6-8794-4108-aee2-5da540d6bc0c-istio-data\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:31:37.988918 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:37.988441    6224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5fe37b6-8794-4108-aee2-5da540d6bc0c-credential-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:31:37.988918 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:37.988457    6224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5fe37b6-8794-4108-aee2-5da540d6bc0c-workload-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:31:37.988918 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:37.988474    6224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5fe37b6-8794-4108-aee2-5da540d6bc0c-workload-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:31:37.990112 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:37.990084    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5fe37b6-8794-4108-aee2-5da540d6bc0c-istio-token" pod "c5fe37b6-8794-4108-aee2-5da540d6bc0c" (UID: "c5fe37b6-8794-4108-aee2-5da540d6bc0c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:31:37.990290 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:37.990125    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5fe37b6-8794-4108-aee2-5da540d6bc0c-kube-api-access-r2f44" pod "c5fe37b6-8794-4108-aee2-5da540d6bc0c" (UID: "c5fe37b6-8794-4108-aee2-5da540d6bc0c"). InnerVolumeSpecName "kube-api-access-r2f44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:31:37.990504 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:37.990468    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5fe37b6-8794-4108-aee2-5da540d6bc0c-istio-podinfo" pod "c5fe37b6-8794-4108-aee2-5da540d6bc0c" (UID: "c5fe37b6-8794-4108-aee2-5da540d6bc0c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:31:37.990630 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:37.990519    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5fe37b6-8794-4108-aee2-5da540d6bc0c-istio-envoy" pod "c5fe37b6-8794-4108-aee2-5da540d6bc0c" (UID: "c5fe37b6-8794-4108-aee2-5da540d6bc0c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:31:38.090351 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:38.090276    6224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5fe37b6-8794-4108-aee2-5da540d6bc0c-istio-podinfo\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:31:38.090351 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:38.090304    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2f44\" (UniqueName: \"kubernetes.io/projected/c5fe37b6-8794-4108-aee2-5da540d6bc0c-kube-api-access-r2f44\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:31:38.090351 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:38.090314    6224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5fe37b6-8794-4108-aee2-5da540d6bc0c-istio-envoy\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:31:38.090351 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:38.090322    6224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5fe37b6-8794-4108-aee2-5da540d6bc0c-istio-token\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:31:38.298092 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:38.298064    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp"]
Oct 30 07:31:38.311789 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:38.311760    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-52gdp"]
Oct 30 07:31:39.504713 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:39.504680    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5fe37b6-8794-4108-aee2-5da540d6bc0c" path="/var/lib/kubelet/pods/c5fe37b6-8794-4108-aee2-5da540d6bc0c/volumes"
Oct 30 07:31:46.223772 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:46.223732    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-14271d/auth-test-model-14271d-kserve-6c67747f77-k8psp" podUID="f5bf8b84-73bd-4729-b5a7-1776c09f23f4" containerName="main" containerID="cri-o://9a844d9df856143fa2fbc735216cacf392e12846a578fef95d92684876aa2c7e" gracePeriod=45
Oct 30 07:31:46.530385 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:46.530357    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-14271d/auth-test-model-14271d-kserve-6c67747f77-k8psp"
Oct 30 07:31:46.706498 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:46.706469    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5bf8b84-73bd-4729-b5a7-1776c09f23f4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5bf8b84-73bd-4729-b5a7-1776c09f23f4-dshm\") pod \"f5bf8b84-73bd-4729-b5a7-1776c09f23f4\" (UID: \"f5bf8b84-73bd-4729-b5a7-1776c09f23f4\") "
Oct 30 07:31:46.706669 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:46.706636    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5bf8b84-73bd-4729-b5a7-1776c09f23f4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5bf8b84-73bd-4729-b5a7-1776c09f23f4-tmp-dir\") pod \"f5bf8b84-73bd-4729-b5a7-1776c09f23f4\" (UID: \"f5bf8b84-73bd-4729-b5a7-1776c09f23f4\") "
Oct 30 07:31:46.706669 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:46.706660    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5bf8b84-73bd-4729-b5a7-1776c09f23f4-kube-api-access-fnnr8\" (UniqueName: \"kubernetes.io/projected/f5bf8b84-73bd-4729-b5a7-1776c09f23f4-kube-api-access-fnnr8\") pod \"f5bf8b84-73bd-4729-b5a7-1776c09f23f4\" (UID: \"f5bf8b84-73bd-4729-b5a7-1776c09f23f4\") "
Oct 30 07:31:46.706793 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:46.706677    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5bf8b84-73bd-4729-b5a7-1776c09f23f4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5bf8b84-73bd-4729-b5a7-1776c09f23f4-model-cache\") pod \"f5bf8b84-73bd-4729-b5a7-1776c09f23f4\" (UID: \"f5bf8b84-73bd-4729-b5a7-1776c09f23f4\") "
Oct 30 07:31:46.706793 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:46.706721    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5bf8b84-73bd-4729-b5a7-1776c09f23f4-home\" (UniqueName: \"kubernetes.io/empty-dir/f5bf8b84-73bd-4729-b5a7-1776c09f23f4-home\") pod \"f5bf8b84-73bd-4729-b5a7-1776c09f23f4\" (UID: \"f5bf8b84-73bd-4729-b5a7-1776c09f23f4\") "
Oct 30 07:31:46.706793 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:46.706767    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5bf8b84-73bd-4729-b5a7-1776c09f23f4-tls-certs\" (UniqueName: \"kubernetes.io/secret/f5bf8b84-73bd-4729-b5a7-1776c09f23f4-tls-certs\") pod \"f5bf8b84-73bd-4729-b5a7-1776c09f23f4\" (UID: \"f5bf8b84-73bd-4729-b5a7-1776c09f23f4\") "
Oct 30 07:31:46.706990 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:46.706934    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5bf8b84-73bd-4729-b5a7-1776c09f23f4-tmp-dir" pod "f5bf8b84-73bd-4729-b5a7-1776c09f23f4" (UID: "f5bf8b84-73bd-4729-b5a7-1776c09f23f4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:31:46.707082 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:46.707060    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5bf8b84-73bd-4729-b5a7-1776c09f23f4-model-cache" pod "f5bf8b84-73bd-4729-b5a7-1776c09f23f4" (UID: "f5bf8b84-73bd-4729-b5a7-1776c09f23f4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:31:46.707147 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:46.707075    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5bf8b84-73bd-4729-b5a7-1776c09f23f4-home" pod "f5bf8b84-73bd-4729-b5a7-1776c09f23f4" (UID: "f5bf8b84-73bd-4729-b5a7-1776c09f23f4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:31:46.707556 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:46.707531    6224 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5bf8b84-73bd-4729-b5a7-1776c09f23f4-tmp-dir\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:31:46.707653 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:46.707561    6224 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5bf8b84-73bd-4729-b5a7-1776c09f23f4-model-cache\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:31:46.707653 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:46.707576    6224 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5bf8b84-73bd-4729-b5a7-1776c09f23f4-home\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:31:46.708910 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:46.708883    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5bf8b84-73bd-4729-b5a7-1776c09f23f4-dshm" pod "f5bf8b84-73bd-4729-b5a7-1776c09f23f4" (UID: "f5bf8b84-73bd-4729-b5a7-1776c09f23f4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:31:46.708910 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:46.708896    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5bf8b84-73bd-4729-b5a7-1776c09f23f4-kube-api-access-fnnr8" pod "f5bf8b84-73bd-4729-b5a7-1776c09f23f4" (UID: "f5bf8b84-73bd-4729-b5a7-1776c09f23f4"). InnerVolumeSpecName "kube-api-access-fnnr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:31:46.709059 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:46.708985    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5bf8b84-73bd-4729-b5a7-1776c09f23f4-tls-certs" pod "f5bf8b84-73bd-4729-b5a7-1776c09f23f4" (UID: "f5bf8b84-73bd-4729-b5a7-1776c09f23f4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:31:46.808492 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:46.808460    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnnr8\" (UniqueName: \"kubernetes.io/projected/f5bf8b84-73bd-4729-b5a7-1776c09f23f4-kube-api-access-fnnr8\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:31:46.808492 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:46.808488    6224 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5bf8b84-73bd-4729-b5a7-1776c09f23f4-tls-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:31:46.808674 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:46.808499    6224 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5bf8b84-73bd-4729-b5a7-1776c09f23f4-dshm\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:31:47.011636 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:47.011600    6224 generic.go:356] "Generic (PLEG): container finished" podID="f5bf8b84-73bd-4729-b5a7-1776c09f23f4" containerID="9a844d9df856143fa2fbc735216cacf392e12846a578fef95d92684876aa2c7e" exitCode=0
Oct 30 07:31:47.011834 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:47.011652    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-14271d/auth-test-model-14271d-kserve-6c67747f77-k8psp"
Oct 30 07:31:47.011834 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:47.011690    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-14271d/auth-test-model-14271d-kserve-6c67747f77-k8psp" event={"ID":"f5bf8b84-73bd-4729-b5a7-1776c09f23f4","Type":"ContainerDied","Data":"9a844d9df856143fa2fbc735216cacf392e12846a578fef95d92684876aa2c7e"}
Oct 30 07:31:47.011834 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:47.011739    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-14271d/auth-test-model-14271d-kserve-6c67747f77-k8psp" event={"ID":"f5bf8b84-73bd-4729-b5a7-1776c09f23f4","Type":"ContainerDied","Data":"47415a471c0bfd4b8eaf8655be240736de41b58ecc83ab7a1b9bcc1b10daf939"}
Oct 30 07:31:47.011834 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:47.011757    6224 scope.go:122] "RemoveContainer" containerID="9a844d9df856143fa2fbc735216cacf392e12846a578fef95d92684876aa2c7e"
Oct 30 07:31:47.021882 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:47.021859    6224 scope.go:122] "RemoveContainer" containerID="9a844d9df856143fa2fbc735216cacf392e12846a578fef95d92684876aa2c7e"
Oct 30 07:31:47.022180 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:31:47.022153    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a844d9df856143fa2fbc735216cacf392e12846a578fef95d92684876aa2c7e\": container with ID starting with 9a844d9df856143fa2fbc735216cacf392e12846a578fef95d92684876aa2c7e not found: ID does not exist" containerID="9a844d9df856143fa2fbc735216cacf392e12846a578fef95d92684876aa2c7e"
Oct 30 07:31:47.022331 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:47.022191    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a844d9df856143fa2fbc735216cacf392e12846a578fef95d92684876aa2c7e"} err="failed to get container status \"9a844d9df856143fa2fbc735216cacf392e12846a578fef95d92684876aa2c7e\": rpc error: code = NotFound desc = could not find container \"9a844d9df856143fa2fbc735216cacf392e12846a578fef95d92684876aa2c7e\": container with ID starting with 9a844d9df856143fa2fbc735216cacf392e12846a578fef95d92684876aa2c7e not found: ID does not exist"
Oct 30 07:31:47.069516 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:47.069457    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-14271d/auth-test-model-14271d-kserve-6c67747f77-k8psp"]
Oct 30 07:31:47.083383 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:47.083359    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-14271d/auth-test-model-14271d-kserve-6c67747f77-k8psp"]
Oct 30 07:31:47.438799 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:47.438695    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-14271d/auth-test-model-14271d-kserve-6c67747f77-k8psp" podUID="f5bf8b84-73bd-4729-b5a7-1776c09f23f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.121:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:31:47.505130 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:31:47.505087    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5bf8b84-73bd-4729-b5a7-1776c09f23f4" path="/var/lib/kubelet/pods/f5bf8b84-73bd-4729-b5a7-1776c09f23f4/volumes"
Oct 30 07:32:01.861877 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:32:01.861838    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:32:06.318653 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:06.318575    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-265384/auth-test-model-265384-kserve-556c8f946f-7l2tb"]
Oct 30 07:32:14.011526 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:14.011493    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:32:14.340037 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:14.339989    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-06e19a/auth-test-model-06e19a-kserve-6947fb644f-szn9m"]
Oct 30 07:32:21.339597 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:21.339552    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-265384/auth-test-model-265384-kserve-556c8f946f-7l2tb" podUID="012fc6e1-7b44-448e-89d4-848cede712f0" containerName="main" containerID="cri-o://5db83520c42f8087319353fbcd8981372df1cae5c97e1a7fad01f4d19a268af5" gracePeriod=45
Oct 30 07:32:21.631236 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:21.631203    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-265384/auth-test-model-265384-kserve-556c8f946f-7l2tb"
Oct 30 07:32:21.767776 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:21.767739    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/012fc6e1-7b44-448e-89d4-848cede712f0-kube-api-access-s2sdc\" (UniqueName: \"kubernetes.io/projected/012fc6e1-7b44-448e-89d4-848cede712f0-kube-api-access-s2sdc\") pod \"012fc6e1-7b44-448e-89d4-848cede712f0\" (UID: \"012fc6e1-7b44-448e-89d4-848cede712f0\") "
Oct 30 07:32:21.767969 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:21.767818    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/012fc6e1-7b44-448e-89d4-848cede712f0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/012fc6e1-7b44-448e-89d4-848cede712f0-model-cache\") pod \"012fc6e1-7b44-448e-89d4-848cede712f0\" (UID: \"012fc6e1-7b44-448e-89d4-848cede712f0\") "
Oct 30 07:32:21.767969 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:21.767875    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/012fc6e1-7b44-448e-89d4-848cede712f0-tls-certs\" (UniqueName: \"kubernetes.io/secret/012fc6e1-7b44-448e-89d4-848cede712f0-tls-certs\") pod \"012fc6e1-7b44-448e-89d4-848cede712f0\" (UID: \"012fc6e1-7b44-448e-89d4-848cede712f0\") "
Oct 30 07:32:21.767969 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:21.767917    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/012fc6e1-7b44-448e-89d4-848cede712f0-home\" (UniqueName: \"kubernetes.io/empty-dir/012fc6e1-7b44-448e-89d4-848cede712f0-home\") pod \"012fc6e1-7b44-448e-89d4-848cede712f0\" (UID: \"012fc6e1-7b44-448e-89d4-848cede712f0\") "
Oct 30 07:32:21.768155 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:21.768012    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/012fc6e1-7b44-448e-89d4-848cede712f0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/012fc6e1-7b44-448e-89d4-848cede712f0-dshm\") pod \"012fc6e1-7b44-448e-89d4-848cede712f0\" (UID: \"012fc6e1-7b44-448e-89d4-848cede712f0\") "
Oct 30 07:32:21.768155 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:21.768077    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/012fc6e1-7b44-448e-89d4-848cede712f0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/012fc6e1-7b44-448e-89d4-848cede712f0-tmp-dir\") pod \"012fc6e1-7b44-448e-89d4-848cede712f0\" (UID: \"012fc6e1-7b44-448e-89d4-848cede712f0\") "
Oct 30 07:32:21.768278 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:21.768145    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/012fc6e1-7b44-448e-89d4-848cede712f0-model-cache" pod "012fc6e1-7b44-448e-89d4-848cede712f0" (UID: "012fc6e1-7b44-448e-89d4-848cede712f0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:32:21.768278 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:21.768249    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/012fc6e1-7b44-448e-89d4-848cede712f0-home" pod "012fc6e1-7b44-448e-89d4-848cede712f0" (UID: "012fc6e1-7b44-448e-89d4-848cede712f0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:32:21.768549 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:21.768525    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/012fc6e1-7b44-448e-89d4-848cede712f0-tmp-dir" pod "012fc6e1-7b44-448e-89d4-848cede712f0" (UID: "012fc6e1-7b44-448e-89d4-848cede712f0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:32:21.768764 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:21.768745    6224 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/012fc6e1-7b44-448e-89d4-848cede712f0-model-cache\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:32:21.768871 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:21.768767    6224 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/012fc6e1-7b44-448e-89d4-848cede712f0-home\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:32:21.768871 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:21.768782    6224 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/012fc6e1-7b44-448e-89d4-848cede712f0-tmp-dir\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:32:21.770243 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:21.770217    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/012fc6e1-7b44-448e-89d4-848cede712f0-dshm" pod "012fc6e1-7b44-448e-89d4-848cede712f0" (UID: "012fc6e1-7b44-448e-89d4-848cede712f0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:32:21.770346 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:21.770296    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/012fc6e1-7b44-448e-89d4-848cede712f0-kube-api-access-s2sdc" pod "012fc6e1-7b44-448e-89d4-848cede712f0" (UID: "012fc6e1-7b44-448e-89d4-848cede712f0"). InnerVolumeSpecName "kube-api-access-s2sdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:32:21.770754 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:21.770732    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/012fc6e1-7b44-448e-89d4-848cede712f0-tls-certs" pod "012fc6e1-7b44-448e-89d4-848cede712f0" (UID: "012fc6e1-7b44-448e-89d4-848cede712f0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:32:21.869787 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:21.869719    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2sdc\" (UniqueName: \"kubernetes.io/projected/012fc6e1-7b44-448e-89d4-848cede712f0-kube-api-access-s2sdc\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:32:21.869787 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:21.869753    6224 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/012fc6e1-7b44-448e-89d4-848cede712f0-tls-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:32:21.869787 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:21.869767    6224 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/012fc6e1-7b44-448e-89d4-848cede712f0-dshm\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:32:22.293652 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.293620    6224 generic.go:356] "Generic (PLEG): container finished" podID="012fc6e1-7b44-448e-89d4-848cede712f0" containerID="5db83520c42f8087319353fbcd8981372df1cae5c97e1a7fad01f4d19a268af5" exitCode=0
Oct 30 07:32:22.293831 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.293665    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-265384/auth-test-model-265384-kserve-556c8f946f-7l2tb"
Oct 30 07:32:22.293831 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.293697    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-265384/auth-test-model-265384-kserve-556c8f946f-7l2tb" event={"ID":"012fc6e1-7b44-448e-89d4-848cede712f0","Type":"ContainerDied","Data":"5db83520c42f8087319353fbcd8981372df1cae5c97e1a7fad01f4d19a268af5"}
Oct 30 07:32:22.293831 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.293736    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-265384/auth-test-model-265384-kserve-556c8f946f-7l2tb" event={"ID":"012fc6e1-7b44-448e-89d4-848cede712f0","Type":"ContainerDied","Data":"4c19044692a32a6fd6ed57b64570e90ddf0105df782f464ffc631a18428a6279"}
Oct 30 07:32:22.293831 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.293753    6224 scope.go:122] "RemoveContainer" containerID="5db83520c42f8087319353fbcd8981372df1cae5c97e1a7fad01f4d19a268af5"
Oct 30 07:32:22.304688 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.304670    6224 scope.go:122] "RemoveContainer" containerID="5db83520c42f8087319353fbcd8981372df1cae5c97e1a7fad01f4d19a268af5"
Oct 30 07:32:22.304965 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:32:22.304944    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5db83520c42f8087319353fbcd8981372df1cae5c97e1a7fad01f4d19a268af5\": container with ID starting with 5db83520c42f8087319353fbcd8981372df1cae5c97e1a7fad01f4d19a268af5 not found: ID does not exist" containerID="5db83520c42f8087319353fbcd8981372df1cae5c97e1a7fad01f4d19a268af5"
Oct 30 07:32:22.305041 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.304976    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5db83520c42f8087319353fbcd8981372df1cae5c97e1a7fad01f4d19a268af5"} err="failed to get container status \"5db83520c42f8087319353fbcd8981372df1cae5c97e1a7fad01f4d19a268af5\": rpc error: code = NotFound desc = could not find container \"5db83520c42f8087319353fbcd8981372df1cae5c97e1a7fad01f4d19a268af5\": container with ID starting with 5db83520c42f8087319353fbcd8981372df1cae5c97e1a7fad01f4d19a268af5 not found: ID does not exist"
Oct 30 07:32:22.361308 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.361281    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-265384/auth-test-model-265384-kserve-556c8f946f-7l2tb"]
Oct 30 07:32:22.374246 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.374219    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-265384/auth-test-model-265384-kserve-556c8f946f-7l2tb"]
Oct 30 07:32:22.400857 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.400822    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-8e7948-78667cb64b-2lp7r"]
Oct 30 07:32:22.401977 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.401950    6224 cpu_manager.go:414] "containerMap: removing container" podUID="d87fa68a-a7e0-41c1-92c3-65c1faec9821" containerName="cleanup"
Oct 30 07:32:22.402150 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.402133    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d87fa68a-a7e0-41c1-92c3-65c1faec9821" containerName="cleanup"
Oct 30 07:32:22.402244 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.402234    6224 cpu_manager.go:414] "containerMap: removing container" podUID="94c9d1b4-2f77-4e91-aa74-8bbb2bc04713" containerName="cleanup"
Oct 30 07:32:22.402329 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.402318    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94c9d1b4-2f77-4e91-aa74-8bbb2bc04713" containerName="cleanup"
Oct 30 07:32:22.402412 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.402402    6224 cpu_manager.go:414] "containerMap: removing container" podUID="7b67837f-7acb-4ec6-86a5-dbe1263b46e2" containerName="cleanup"
Oct 30 07:32:22.402489 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.402479    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b67837f-7acb-4ec6-86a5-dbe1263b46e2" containerName="cleanup"
Oct 30 07:32:22.402578 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.402568    6224 cpu_manager.go:414] "containerMap: removing container" podUID="cd765ec1-29ac-4173-9a95-17099c37379a" containerName="cleanup"
Oct 30 07:32:22.402658 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.402648    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd765ec1-29ac-4173-9a95-17099c37379a" containerName="cleanup"
Oct 30 07:32:22.402735 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.402726    6224 cpu_manager.go:414] "containerMap: removing container" podUID="cd765ec1-29ac-4173-9a95-17099c37379a" containerName="cleanup"
Oct 30 07:32:22.402833 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.402821    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd765ec1-29ac-4173-9a95-17099c37379a" containerName="cleanup"
Oct 30 07:32:22.402926 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.402915    6224 cpu_manager.go:414] "containerMap: removing container" podUID="7b67837f-7acb-4ec6-86a5-dbe1263b46e2" containerName="cleanup"
Oct 30 07:32:22.403061 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.403047    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b67837f-7acb-4ec6-86a5-dbe1263b46e2" containerName="cleanup"
Oct 30 07:32:22.403149 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.403139    6224 cpu_manager.go:414] "containerMap: removing container" podUID="59474a91-6ad5-4a1b-9220-30356c14f255" containerName="cleanup"
Oct 30 07:32:22.403229 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.403219    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59474a91-6ad5-4a1b-9220-30356c14f255" containerName="cleanup"
Oct 30 07:32:22.435027 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.434995    6224 cpu_manager.go:414] "containerMap: removing container" podUID="5b5884ad-a370-40c8-89f5-6f260bbf234a" containerName="istio-proxy"
Oct 30 07:32:22.435230 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.435213    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b5884ad-a370-40c8-89f5-6f260bbf234a" containerName="istio-proxy"
Oct 30 07:32:22.435348 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.435336    6224 cpu_manager.go:414] "containerMap: removing container" podUID="59474a91-6ad5-4a1b-9220-30356c14f255" containerName="cleanup"
Oct 30 07:32:22.435433 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.435423    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59474a91-6ad5-4a1b-9220-30356c14f255" containerName="cleanup"
Oct 30 07:32:22.435510 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.435501    6224 cpu_manager.go:414] "containerMap: removing container" podUID="012fc6e1-7b44-448e-89d4-848cede712f0" containerName="main"
Oct 30 07:32:22.435592 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.435581    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="012fc6e1-7b44-448e-89d4-848cede712f0" containerName="main"
Oct 30 07:32:22.435669 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.435660    6224 cpu_manager.go:414] "containerMap: removing container" podUID="04947652-67c7-45de-a2dc-5c2856bfdad8" containerName="istio-proxy"
Oct 30 07:32:22.435743 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.435733    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04947652-67c7-45de-a2dc-5c2856bfdad8" containerName="istio-proxy"
Oct 30 07:32:22.435855 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.435843    6224 cpu_manager.go:414] "containerMap: removing container" podUID="c5fe37b6-8794-4108-aee2-5da540d6bc0c" containerName="istio-proxy"
Oct 30 07:32:22.435945 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.435935    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5fe37b6-8794-4108-aee2-5da540d6bc0c" containerName="istio-proxy"
Oct 30 07:32:22.436026 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.436016    6224 cpu_manager.go:414] "containerMap: removing container" podUID="3290bdb2-873c-4502-bb84-e8205fabbedf" containerName="istio-proxy"
Oct 30 07:32:22.436100 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.436090    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3290bdb2-873c-4502-bb84-e8205fabbedf" containerName="istio-proxy"
Oct 30 07:32:22.436180 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.436171    6224 cpu_manager.go:414] "containerMap: removing container" podUID="d87fa68a-a7e0-41c1-92c3-65c1faec9821" containerName="cleanup"
Oct 30 07:32:22.436257 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.436246    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d87fa68a-a7e0-41c1-92c3-65c1faec9821" containerName="cleanup"
Oct 30 07:32:22.448891 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.448861    6224 cpu_manager.go:414] "containerMap: removing container" podUID="6169ad23-c3a7-40de-bfd2-8f232a428238" containerName="istio-proxy"
Oct 30 07:32:22.449082 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.449067    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6169ad23-c3a7-40de-bfd2-8f232a428238" containerName="istio-proxy"
Oct 30 07:32:22.449193 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.449180    6224 cpu_manager.go:414] "containerMap: removing container" podUID="94c9d1b4-2f77-4e91-aa74-8bbb2bc04713" containerName="cleanup"
Oct 30 07:32:22.449271 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.449260    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94c9d1b4-2f77-4e91-aa74-8bbb2bc04713" containerName="cleanup"
Oct 30 07:32:22.449356 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.449346    6224 cpu_manager.go:414] "containerMap: removing container" podUID="5611d3ae-2361-411e-903f-ebd0d5b5fba5" containerName="istio-proxy"
Oct 30 07:32:22.449433 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.449423    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5611d3ae-2361-411e-903f-ebd0d5b5fba5" containerName="istio-proxy"
Oct 30 07:32:22.449519 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.449510    6224 cpu_manager.go:414] "containerMap: removing container" podUID="94c9d1b4-2f77-4e91-aa74-8bbb2bc04713" containerName="cleanup"
Oct 30 07:32:22.483833 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.449590    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94c9d1b4-2f77-4e91-aa74-8bbb2bc04713" containerName="cleanup"
Oct 30 07:32:22.484078 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.484054    6224 cpu_manager.go:414] "containerMap: removing container" podUID="cd765ec1-29ac-4173-9a95-17099c37379a" containerName="cleanup"
Oct 30 07:32:22.484203 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.484190    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd765ec1-29ac-4173-9a95-17099c37379a" containerName="cleanup"
Oct 30 07:32:22.484317 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.484305    6224 cpu_manager.go:414] "containerMap: removing container" podUID="f5bf8b84-73bd-4729-b5a7-1776c09f23f4" containerName="main"
Oct 30 07:32:22.484400 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.484387    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5bf8b84-73bd-4729-b5a7-1776c09f23f4" containerName="main"
Oct 30 07:32:22.484486 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.484476    6224 cpu_manager.go:414] "containerMap: removing container" podUID="1a470b54-ad63-40f7-b306-f8ee1dc8cdcd" containerName="istio-proxy"
Oct 30 07:32:22.484562 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.484552    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a470b54-ad63-40f7-b306-f8ee1dc8cdcd" containerName="istio-proxy"
Oct 30 07:32:22.484852 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.484833    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="3290bdb2-873c-4502-bb84-e8205fabbedf" containerName="istio-proxy"
Oct 30 07:32:22.484976 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.484964    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="d87fa68a-a7e0-41c1-92c3-65c1faec9821" containerName="cleanup"
Oct 30 07:32:22.485071 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.485061    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="6169ad23-c3a7-40de-bfd2-8f232a428238" containerName="istio-proxy"
Oct 30 07:32:22.485165 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.485154    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="94c9d1b4-2f77-4e91-aa74-8bbb2bc04713" containerName="cleanup"
Oct 30 07:32:22.485246 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.485236    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="5611d3ae-2361-411e-903f-ebd0d5b5fba5" containerName="istio-proxy"
Oct 30 07:32:22.485328 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.485318    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="94c9d1b4-2f77-4e91-aa74-8bbb2bc04713" containerName="cleanup"
Oct 30 07:32:22.485405 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.485396    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd765ec1-29ac-4173-9a95-17099c37379a" containerName="cleanup"
Oct 30 07:32:22.485483 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.485474    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5bf8b84-73bd-4729-b5a7-1776c09f23f4" containerName="main"
Oct 30 07:32:22.485562 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.485553    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="d87fa68a-a7e0-41c1-92c3-65c1faec9821" containerName="cleanup"
Oct 30 07:32:22.485633 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.485624    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b67837f-7acb-4ec6-86a5-dbe1263b46e2" containerName="cleanup"
Oct 30 07:32:22.485708 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.485698    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a470b54-ad63-40f7-b306-f8ee1dc8cdcd" containerName="istio-proxy"
Oct 30 07:32:22.485783 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.485773    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="59474a91-6ad5-4a1b-9220-30356c14f255" containerName="cleanup"
Oct 30 07:32:22.485891 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.485879    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="d87fa68a-a7e0-41c1-92c3-65c1faec9821" containerName="cleanup"
Oct 30 07:32:22.485972 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.485963    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd765ec1-29ac-4173-9a95-17099c37379a" containerName="cleanup"
Oct 30 07:32:22.486062 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.486039    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b67837f-7acb-4ec6-86a5-dbe1263b46e2" containerName="cleanup"
Oct 30 07:32:22.486154 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.486143    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b5884ad-a370-40c8-89f5-6f260bbf234a" containerName="istio-proxy"
Oct 30 07:32:22.486233 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.486223    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="59474a91-6ad5-4a1b-9220-30356c14f255" containerName="cleanup"
Oct 30 07:32:22.486308 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.486298    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="012fc6e1-7b44-448e-89d4-848cede712f0" containerName="main"
Oct 30 07:32:22.486384 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.486373    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="04947652-67c7-45de-a2dc-5c2856bfdad8" containerName="istio-proxy"
Oct 30 07:32:22.486461 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.486451    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5fe37b6-8794-4108-aee2-5da540d6bc0c" containerName="istio-proxy"
Oct 30 07:32:22.535725 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.535413    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-8e7948-78667cb64b-2lp7r"]
Oct 30 07:32:22.535999 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.535681    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-8e7948-78667cb64b-2lp7r"
Oct 30 07:32:22.554613 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.554588    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-8e7948\""
Oct 30 07:32:22.554786 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.554770    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-8e7948-dockercfg-65k9x\""
Oct 30 07:32:22.620243 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.620213    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64034fab-2b81-4f54-9d0c-5f2360ab6274-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-8e7948-78667cb64b-2lp7r\" (UID: \"64034fab-2b81-4f54-9d0c-5f2360ab6274\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-8e7948-78667cb64b-2lp7r"
Oct 30 07:32:22.620407 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.620349    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zckh\" (UniqueName: \"kubernetes.io/projected/64034fab-2b81-4f54-9d0c-5f2360ab6274-kube-api-access-4zckh\") pod \"payload-processing-e2e-extmod-a-8e7948-78667cb64b-2lp7r\" (UID: \"64034fab-2b81-4f54-9d0c-5f2360ab6274\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-8e7948-78667cb64b-2lp7r"
Oct 30 07:32:22.721896 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.721858    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64034fab-2b81-4f54-9d0c-5f2360ab6274-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-8e7948-78667cb64b-2lp7r\" (UID: \"64034fab-2b81-4f54-9d0c-5f2360ab6274\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-8e7948-78667cb64b-2lp7r"
Oct 30 07:32:22.722071 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.721977    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zckh\" (UniqueName: \"kubernetes.io/projected/64034fab-2b81-4f54-9d0c-5f2360ab6274-kube-api-access-4zckh\") pod \"payload-processing-e2e-extmod-a-8e7948-78667cb64b-2lp7r\" (UID: \"64034fab-2b81-4f54-9d0c-5f2360ab6274\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-8e7948-78667cb64b-2lp7r"
Oct 30 07:32:22.722667 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.722645    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64034fab-2b81-4f54-9d0c-5f2360ab6274-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-8e7948-78667cb64b-2lp7r\" (UID: \"64034fab-2b81-4f54-9d0c-5f2360ab6274\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-8e7948-78667cb64b-2lp7r"
Oct 30 07:32:22.747508 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.747475    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zckh\" (UniqueName: \"kubernetes.io/projected/64034fab-2b81-4f54-9d0c-5f2360ab6274-kube-api-access-4zckh\") pod \"payload-processing-e2e-extmod-a-8e7948-78667cb64b-2lp7r\" (UID: \"64034fab-2b81-4f54-9d0c-5f2360ab6274\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-8e7948-78667cb64b-2lp7r"
Oct 30 07:32:22.856435 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:22.856360    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-8e7948-78667cb64b-2lp7r"
Oct 30 07:32:23.188256 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:32:23.188225    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64034fab_2b81_4f54_9d0c_5f2360ab6274.slice/crio-e59d84c1e4e47a834f84d55a06c1a44c9bda13dfc9386ad54a3a6aa3548b3933 WatchSource:0}: Error finding container e59d84c1e4e47a834f84d55a06c1a44c9bda13dfc9386ad54a3a6aa3548b3933: Status 404 returned error can't find the container with id e59d84c1e4e47a834f84d55a06c1a44c9bda13dfc9386ad54a3a6aa3548b3933
Oct 30 07:32:23.190976 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:23.190950    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-8e7948-78667cb64b-2lp7r"]
Oct 30 07:32:23.303471 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:23.303440    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-8e7948-78667cb64b-2lp7r" event={"ID":"64034fab-2b81-4f54-9d0c-5f2360ab6274","Type":"ContainerStarted","Data":"64a199f4d13078738b4faed451e11cb7b2296b8b5b2a3d8aaa2a63134ccefc38"}
Oct 30 07:32:23.303592 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:23.303494    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-8e7948-78667cb64b-2lp7r" event={"ID":"64034fab-2b81-4f54-9d0c-5f2360ab6274","Type":"ContainerStarted","Data":"e59d84c1e4e47a834f84d55a06c1a44c9bda13dfc9386ad54a3a6aa3548b3933"}
Oct 30 07:32:23.303592 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:23.303519    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-8e7948-78667cb64b-2lp7r"
Oct 30 07:32:23.352004 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:23.351963    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-8e7948-78667cb64b-2lp7r" podStartSLOduration=1.351951595 podStartE2EDuration="1.351951595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:32:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:32:23.349501722 +0000 UTC m=+2101.958908693" watchObservedRunningTime="2026-10-30 07:32:23.351951595 +0000 UTC m=+2101.961358594"
Oct 30 07:32:23.507049 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:23.506972    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="012fc6e1-7b44-448e-89d4-848cede712f0" path="/var/lib/kubelet/pods/012fc6e1-7b44-448e-89d4-848cede712f0/volumes"
Oct 30 07:32:26.491534 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:26.491491    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-932c00/auth-test-model-932c00-kserve-5496946777-b8b4x"]
Oct 30 07:32:29.365304 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:29.365249    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-06e19a/auth-test-model-06e19a-kserve-6947fb644f-szn9m" podUID="b8925605-f5fb-43cd-ac97-157c5a5973c9" containerName="main" containerID="cri-o://c768138fecaf83d7177db5e91958dabd201311696c6f20e382ac0af01e99444b" gracePeriod=45
Oct 30 07:32:29.626019 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:29.625962    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-06e19a/auth-test-model-06e19a-kserve-6947fb644f-szn9m"
Oct 30 07:32:29.735896 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:29.735864    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8925605-f5fb-43cd-ac97-157c5a5973c9-home\" (UniqueName: \"kubernetes.io/empty-dir/b8925605-f5fb-43cd-ac97-157c5a5973c9-home\") pod \"b8925605-f5fb-43cd-ac97-157c5a5973c9\" (UID: \"b8925605-f5fb-43cd-ac97-157c5a5973c9\") "
Oct 30 07:32:29.736080 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:29.735926    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8925605-f5fb-43cd-ac97-157c5a5973c9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8925605-f5fb-43cd-ac97-157c5a5973c9-model-cache\") pod \"b8925605-f5fb-43cd-ac97-157c5a5973c9\" (UID: \"b8925605-f5fb-43cd-ac97-157c5a5973c9\") "
Oct 30 07:32:29.736080 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:29.736058    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8925605-f5fb-43cd-ac97-157c5a5973c9-kube-api-access-djjh4\" (UniqueName: \"kubernetes.io/projected/b8925605-f5fb-43cd-ac97-157c5a5973c9-kube-api-access-djjh4\") pod \"b8925605-f5fb-43cd-ac97-157c5a5973c9\" (UID: \"b8925605-f5fb-43cd-ac97-157c5a5973c9\") "
Oct 30 07:32:29.736212 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:29.736140    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8925605-f5fb-43cd-ac97-157c5a5973c9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8925605-f5fb-43cd-ac97-157c5a5973c9-tmp-dir\") pod \"b8925605-f5fb-43cd-ac97-157c5a5973c9\" (UID: \"b8925605-f5fb-43cd-ac97-157c5a5973c9\") "
Oct 30 07:32:29.736212 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:29.736181    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8925605-f5fb-43cd-ac97-157c5a5973c9-home" pod "b8925605-f5fb-43cd-ac97-157c5a5973c9" (UID: "b8925605-f5fb-43cd-ac97-157c5a5973c9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:32:29.736212 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:29.736199    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8925605-f5fb-43cd-ac97-157c5a5973c9-tls-certs\" (UniqueName: \"kubernetes.io/secret/b8925605-f5fb-43cd-ac97-157c5a5973c9-tls-certs\") pod \"b8925605-f5fb-43cd-ac97-157c5a5973c9\" (UID: \"b8925605-f5fb-43cd-ac97-157c5a5973c9\") "
Oct 30 07:32:29.736336 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:29.736230    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8925605-f5fb-43cd-ac97-157c5a5973c9-model-cache" pod "b8925605-f5fb-43cd-ac97-157c5a5973c9" (UID: "b8925605-f5fb-43cd-ac97-157c5a5973c9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:32:29.736336 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:29.736282    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8925605-f5fb-43cd-ac97-157c5a5973c9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8925605-f5fb-43cd-ac97-157c5a5973c9-dshm\") pod \"b8925605-f5fb-43cd-ac97-157c5a5973c9\" (UID: \"b8925605-f5fb-43cd-ac97-157c5a5973c9\") "
Oct 30 07:32:29.736420 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:29.736388    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8925605-f5fb-43cd-ac97-157c5a5973c9-tmp-dir" pod "b8925605-f5fb-43cd-ac97-157c5a5973c9" (UID: "b8925605-f5fb-43cd-ac97-157c5a5973c9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:32:29.736732 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:29.736710    6224 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8925605-f5fb-43cd-ac97-157c5a5973c9-tmp-dir\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:32:29.736732 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:29.736732    6224 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8925605-f5fb-43cd-ac97-157c5a5973c9-home\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:32:29.736924 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:29.736744    6224 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8925605-f5fb-43cd-ac97-157c5a5973c9-model-cache\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:32:29.738388 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:29.738363    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8925605-f5fb-43cd-ac97-157c5a5973c9-dshm" pod "b8925605-f5fb-43cd-ac97-157c5a5973c9" (UID: "b8925605-f5fb-43cd-ac97-157c5a5973c9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:32:29.738471 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:29.738385    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8925605-f5fb-43cd-ac97-157c5a5973c9-tls-certs" pod "b8925605-f5fb-43cd-ac97-157c5a5973c9" (UID: "b8925605-f5fb-43cd-ac97-157c5a5973c9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:32:29.738471 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:29.738434    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8925605-f5fb-43cd-ac97-157c5a5973c9-kube-api-access-djjh4" pod "b8925605-f5fb-43cd-ac97-157c5a5973c9" (UID: "b8925605-f5fb-43cd-ac97-157c5a5973c9"). InnerVolumeSpecName "kube-api-access-djjh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:32:29.838187 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:29.838162    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djjh4\" (UniqueName: \"kubernetes.io/projected/b8925605-f5fb-43cd-ac97-157c5a5973c9-kube-api-access-djjh4\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:32:29.838187 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:29.838186    6224 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8925605-f5fb-43cd-ac97-157c5a5973c9-tls-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:32:29.838351 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:29.838196    6224 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8925605-f5fb-43cd-ac97-157c5a5973c9-dshm\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:32:30.353851 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:30.353801    6224 generic.go:356] "Generic (PLEG): container finished" podID="b8925605-f5fb-43cd-ac97-157c5a5973c9" containerID="c768138fecaf83d7177db5e91958dabd201311696c6f20e382ac0af01e99444b" exitCode=0
Oct 30 07:32:30.354029 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:30.353873    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-06e19a/auth-test-model-06e19a-kserve-6947fb644f-szn9m"
Oct 30 07:32:30.354029 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:30.353871    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-06e19a/auth-test-model-06e19a-kserve-6947fb644f-szn9m" event={"ID":"b8925605-f5fb-43cd-ac97-157c5a5973c9","Type":"ContainerDied","Data":"c768138fecaf83d7177db5e91958dabd201311696c6f20e382ac0af01e99444b"}
Oct 30 07:32:30.354029 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:30.353917    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-06e19a/auth-test-model-06e19a-kserve-6947fb644f-szn9m" event={"ID":"b8925605-f5fb-43cd-ac97-157c5a5973c9","Type":"ContainerDied","Data":"48c6ab1b00576964317b2a59f38ea730ae739dda42fb3eac2f160ebcef6bd060"}
Oct 30 07:32:30.354029 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:30.353940    6224 scope.go:122] "RemoveContainer" containerID="c768138fecaf83d7177db5e91958dabd201311696c6f20e382ac0af01e99444b"
Oct 30 07:32:30.365283 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:30.365255    6224 scope.go:122] "RemoveContainer" containerID="c768138fecaf83d7177db5e91958dabd201311696c6f20e382ac0af01e99444b"
Oct 30 07:32:30.365571 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:32:30.365547    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c768138fecaf83d7177db5e91958dabd201311696c6f20e382ac0af01e99444b\": container with ID starting with c768138fecaf83d7177db5e91958dabd201311696c6f20e382ac0af01e99444b not found: ID does not exist" containerID="c768138fecaf83d7177db5e91958dabd201311696c6f20e382ac0af01e99444b"
Oct 30 07:32:30.365905 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:30.365575    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c768138fecaf83d7177db5e91958dabd201311696c6f20e382ac0af01e99444b"} err="failed to get container status \"c768138fecaf83d7177db5e91958dabd201311696c6f20e382ac0af01e99444b\": rpc error: code = NotFound desc = could not find container \"c768138fecaf83d7177db5e91958dabd201311696c6f20e382ac0af01e99444b\": container with ID starting with c768138fecaf83d7177db5e91958dabd201311696c6f20e382ac0af01e99444b not found: ID does not exist"
Oct 30 07:32:30.415078 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:30.415053    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-06e19a/auth-test-model-06e19a-kserve-6947fb644f-szn9m"]
Oct 30 07:32:30.417981 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:30.417956    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-06e19a/auth-test-model-06e19a-kserve-6947fb644f-szn9m"]
Oct 30 07:32:31.504265 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:31.504231    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8925605-f5fb-43cd-ac97-157c5a5973c9" path="/var/lib/kubelet/pods/b8925605-f5fb-43cd-ac97-157c5a5973c9/volumes"
Oct 30 07:32:31.862160 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:32:31.862123    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:32:34.316496 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:34.316461    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-8e7948-78667cb64b-2lp7r"
Oct 30 07:32:34.319267 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:34.319243    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8f1df-f98c8b658-g2h4t"]
Oct 30 07:32:34.319796 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:34.319783    6224 cpu_manager.go:414] "containerMap: removing container" podUID="b8925605-f5fb-43cd-ac97-157c5a5973c9" containerName="main"
Oct 30 07:32:34.319915 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:34.319798    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8925605-f5fb-43cd-ac97-157c5a5973c9" containerName="main"
Oct 30 07:32:34.319915 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:34.319824    6224 cpu_manager.go:414] "containerMap: removing container" podUID="7b67837f-7acb-4ec6-86a5-dbe1263b46e2" containerName="cleanup"
Oct 30 07:32:34.319915 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:34.319830    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b67837f-7acb-4ec6-86a5-dbe1263b46e2" containerName="cleanup"
Oct 30 07:32:34.319915 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:34.319844    6224 cpu_manager.go:414] "containerMap: removing container" podUID="59474a91-6ad5-4a1b-9220-30356c14f255" containerName="cleanup"
Oct 30 07:32:34.319915 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:34.319849    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59474a91-6ad5-4a1b-9220-30356c14f255" containerName="cleanup"
Oct 30 07:32:34.319915 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:34.319862    6224 cpu_manager.go:414] "containerMap: removing container" podUID="d87fa68a-a7e0-41c1-92c3-65c1faec9821" containerName="cleanup"
Oct 30 07:32:34.319915 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:34.319867    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d87fa68a-a7e0-41c1-92c3-65c1faec9821" containerName="cleanup"
Oct 30 07:32:34.320125 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:34.319973    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8925605-f5fb-43cd-ac97-157c5a5973c9" containerName="main"
Oct 30 07:32:34.320125 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:34.319984    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b67837f-7acb-4ec6-86a5-dbe1263b46e2" containerName="cleanup"
Oct 30 07:32:34.320125 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:34.319992    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="59474a91-6ad5-4a1b-9220-30356c14f255" containerName="cleanup"
Oct 30 07:32:34.320125 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:34.320001    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="94c9d1b4-2f77-4e91-aa74-8bbb2bc04713" containerName="cleanup"
Oct 30 07:32:34.320125 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:34.320008    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd765ec1-29ac-4173-9a95-17099c37379a" containerName="cleanup"
Oct 30 07:32:34.324357 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:34.324336    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8f1df-f98c8b658-g2h4t"
Oct 30 07:32:34.353589 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:34.353559    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8f1df-f98c8b658-g2h4t"]
Oct 30 07:32:34.353918 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:34.353897    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-a8f1df-serving-cert\""
Oct 30 07:32:34.397937 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:34.397905    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/319ad66a-6d8b-40bf-8a32-3576e26f34ed-maas-api-tls\") pod \"maas-api-e2e-extmod-a-a8f1df-f98c8b658-g2h4t\" (UID: \"319ad66a-6d8b-40bf-8a32-3576e26f34ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8f1df-f98c8b658-g2h4t"
Oct 30 07:32:34.398112 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:34.397963    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/319ad66a-6d8b-40bf-8a32-3576e26f34ed-metrics-certs\") pod \"maas-api-e2e-extmod-a-a8f1df-f98c8b658-g2h4t\" (UID: \"319ad66a-6d8b-40bf-8a32-3576e26f34ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8f1df-f98c8b658-g2h4t"
Oct 30 07:32:34.398174 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:34.398114    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/319ad66a-6d8b-40bf-8a32-3576e26f34ed-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-a8f1df-f98c8b658-g2h4t\" (UID: \"319ad66a-6d8b-40bf-8a32-3576e26f34ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8f1df-f98c8b658-g2h4t"
Oct 30 07:32:34.398229 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:34.398174    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p5xd\" (UniqueName: \"kubernetes.io/projected/319ad66a-6d8b-40bf-8a32-3576e26f34ed-kube-api-access-7p5xd\") pod \"maas-api-e2e-extmod-a-a8f1df-f98c8b658-g2h4t\" (UID: \"319ad66a-6d8b-40bf-8a32-3576e26f34ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8f1df-f98c8b658-g2h4t"
Oct 30 07:32:34.500048 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:34.500017    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/319ad66a-6d8b-40bf-8a32-3576e26f34ed-maas-api-tls\") pod \"maas-api-e2e-extmod-a-a8f1df-f98c8b658-g2h4t\" (UID: \"319ad66a-6d8b-40bf-8a32-3576e26f34ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8f1df-f98c8b658-g2h4t"
Oct 30 07:32:34.500048 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:34.500056    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/319ad66a-6d8b-40bf-8a32-3576e26f34ed-metrics-certs\") pod \"maas-api-e2e-extmod-a-a8f1df-f98c8b658-g2h4t\" (UID: \"319ad66a-6d8b-40bf-8a32-3576e26f34ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8f1df-f98c8b658-g2h4t"
Oct 30 07:32:34.500288 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:34.500110    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/319ad66a-6d8b-40bf-8a32-3576e26f34ed-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-a8f1df-f98c8b658-g2h4t\" (UID: \"319ad66a-6d8b-40bf-8a32-3576e26f34ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8f1df-f98c8b658-g2h4t"
Oct 30 07:32:34.500288 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:34.500143    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p5xd\" (UniqueName: \"kubernetes.io/projected/319ad66a-6d8b-40bf-8a32-3576e26f34ed-kube-api-access-7p5xd\") pod \"maas-api-e2e-extmod-a-a8f1df-f98c8b658-g2h4t\" (UID: \"319ad66a-6d8b-40bf-8a32-3576e26f34ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8f1df-f98c8b658-g2h4t"
Oct 30 07:32:34.500288 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:32:34.500150    6224 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8f1df-serving-cert: secret "maas-api-e2e-extmod-a-a8f1df-serving-cert" not found
Oct 30 07:32:34.500288 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:32:34.500217    6224 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/319ad66a-6d8b-40bf-8a32-3576e26f34ed-maas-api-tls podName:319ad66a-6d8b-40bf-8a32-3576e26f34ed nodeName:}" failed. No retries permitted until 2026-10-30 07:32:35.000202583 +0000 UTC m=+2113.609609545 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/319ad66a-6d8b-40bf-8a32-3576e26f34ed-maas-api-tls") pod "maas-api-e2e-extmod-a-a8f1df-f98c8b658-g2h4t" (UID: "319ad66a-6d8b-40bf-8a32-3576e26f34ed") : secret "maas-api-e2e-extmod-a-a8f1df-serving-cert" not found
Oct 30 07:32:34.500760 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:34.500741    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/319ad66a-6d8b-40bf-8a32-3576e26f34ed-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-a8f1df-f98c8b658-g2h4t\" (UID: \"319ad66a-6d8b-40bf-8a32-3576e26f34ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8f1df-f98c8b658-g2h4t"
Oct 30 07:32:34.502592 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:34.502569    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/319ad66a-6d8b-40bf-8a32-3576e26f34ed-metrics-certs\") pod \"maas-api-e2e-extmod-a-a8f1df-f98c8b658-g2h4t\" (UID: \"319ad66a-6d8b-40bf-8a32-3576e26f34ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8f1df-f98c8b658-g2h4t"
Oct 30 07:32:34.548606 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:34.548570    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p5xd\" (UniqueName: \"kubernetes.io/projected/319ad66a-6d8b-40bf-8a32-3576e26f34ed-kube-api-access-7p5xd\") pod \"maas-api-e2e-extmod-a-a8f1df-f98c8b658-g2h4t\" (UID: \"319ad66a-6d8b-40bf-8a32-3576e26f34ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8f1df-f98c8b658-g2h4t"
Oct 30 07:32:35.006327 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:35.006293    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/319ad66a-6d8b-40bf-8a32-3576e26f34ed-maas-api-tls\") pod \"maas-api-e2e-extmod-a-a8f1df-f98c8b658-g2h4t\" (UID: \"319ad66a-6d8b-40bf-8a32-3576e26f34ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8f1df-f98c8b658-g2h4t"
Oct 30 07:32:35.008873 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:35.008847    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/319ad66a-6d8b-40bf-8a32-3576e26f34ed-maas-api-tls\") pod \"maas-api-e2e-extmod-a-a8f1df-f98c8b658-g2h4t\" (UID: \"319ad66a-6d8b-40bf-8a32-3576e26f34ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8f1df-f98c8b658-g2h4t"
Oct 30 07:32:35.237885 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:35.237850    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8f1df-f98c8b658-g2h4t"
Oct 30 07:32:35.604980 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:35.604943    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8f1df-f98c8b658-g2h4t"]
Oct 30 07:32:35.661173 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:35.661148    6224 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 07:32:36.406478 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:36.406378    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8f1df-f98c8b658-g2h4t" event={"ID":"319ad66a-6d8b-40bf-8a32-3576e26f34ed","Type":"ContainerStarted","Data":"ddf2d0fa99cd2f1814e7e202d2ce7d8a861c4fb7895918564a58fbc1e98382c8"}
Oct 30 07:32:36.406677 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:36.406482    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8f1df-f98c8b658-g2h4t" event={"ID":"319ad66a-6d8b-40bf-8a32-3576e26f34ed","Type":"ContainerStarted","Data":"b4baec71214b99314a2e3f1adc94dfbc982e941932ef6bbfdea8a5cffe119d53"}
Oct 30 07:32:36.406677 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:36.406554    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8f1df-f98c8b658-g2h4t"
Oct 30 07:32:36.459104 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:36.459041    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8f1df-f98c8b658-g2h4t" podStartSLOduration=2.054095763 podStartE2EDuration="2.459021522s" totalImagesPullingTime="404.925759ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:32:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:32:36.457246911 +0000 UTC m=+2115.066653907" watchObservedRunningTime="2026-10-30 07:32:36.459021522 +0000 UTC m=+2115.068428493"
Oct 30 07:32:38.325901 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:38.325868    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf"]
Oct 30 07:32:38.330712 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:38.330684    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf"
Oct 30 07:32:38.358825 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:38.358782    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf"]
Oct 30 07:32:38.365779 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:38.365749    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-9bec5c-openshift-default-dockercfg-8w5kc\""
Oct 30 07:32:38.457829 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:38.456975    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89326bce-8281-4747-aa3f-c1d37c615234-istio-token\") pod \"e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf\" (UID: \"89326bce-8281-4747-aa3f-c1d37c615234\") " pod="openshift-ingress/e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf"
Oct 30 07:32:38.457829 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:38.457050    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89326bce-8281-4747-aa3f-c1d37c615234-istiod-ca-cert\") pod \"e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf\" (UID: \"89326bce-8281-4747-aa3f-c1d37c615234\") " pod="openshift-ingress/e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf"
Oct 30 07:32:38.457829 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:38.457112    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89326bce-8281-4747-aa3f-c1d37c615234-workload-socket\") pod \"e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf\" (UID: \"89326bce-8281-4747-aa3f-c1d37c615234\") " pod="openshift-ingress/e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf"
Oct 30 07:32:38.457829 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:38.457221    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rqh2\" (UniqueName: \"kubernetes.io/projected/89326bce-8281-4747-aa3f-c1d37c615234-kube-api-access-9rqh2\") pod \"e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf\" (UID: \"89326bce-8281-4747-aa3f-c1d37c615234\") " pod="openshift-ingress/e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf"
Oct 30 07:32:38.457829 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:38.457262    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89326bce-8281-4747-aa3f-c1d37c615234-istio-podinfo\") pod \"e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf\" (UID: \"89326bce-8281-4747-aa3f-c1d37c615234\") " pod="openshift-ingress/e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf"
Oct 30 07:32:38.457829 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:38.457400    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89326bce-8281-4747-aa3f-c1d37c615234-credential-socket\") pod \"e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf\" (UID: \"89326bce-8281-4747-aa3f-c1d37c615234\") " pod="openshift-ingress/e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf"
Oct 30 07:32:38.457829 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:38.457437    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89326bce-8281-4747-aa3f-c1d37c615234-istio-data\") pod \"e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf\" (UID: \"89326bce-8281-4747-aa3f-c1d37c615234\") " pod="openshift-ingress/e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf"
Oct 30 07:32:38.457829 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:38.457467    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89326bce-8281-4747-aa3f-c1d37c615234-istio-envoy\") pod \"e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf\" (UID: \"89326bce-8281-4747-aa3f-c1d37c615234\") " pod="openshift-ingress/e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf"
Oct 30 07:32:38.457829 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:38.457556    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89326bce-8281-4747-aa3f-c1d37c615234-workload-certs\") pod \"e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf\" (UID: \"89326bce-8281-4747-aa3f-c1d37c615234\") " pod="openshift-ingress/e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf"
Oct 30 07:32:38.558828 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:38.558679    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89326bce-8281-4747-aa3f-c1d37c615234-credential-socket\") pod \"e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf\" (UID: \"89326bce-8281-4747-aa3f-c1d37c615234\") " pod="openshift-ingress/e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf"
Oct 30 07:32:38.558828 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:38.558746    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89326bce-8281-4747-aa3f-c1d37c615234-istio-data\") pod \"e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf\" (UID: \"89326bce-8281-4747-aa3f-c1d37c615234\") " pod="openshift-ingress/e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf"
Oct 30 07:32:38.558828 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:38.558777    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89326bce-8281-4747-aa3f-c1d37c615234-istio-envoy\") pod \"e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf\" (UID: \"89326bce-8281-4747-aa3f-c1d37c615234\") " pod="openshift-ingress/e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf"
Oct 30 07:32:38.559194 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:38.558886    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89326bce-8281-4747-aa3f-c1d37c615234-workload-certs\") pod \"e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf\" (UID: \"89326bce-8281-4747-aa3f-c1d37c615234\") " pod="openshift-ingress/e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf"
Oct 30 07:32:38.559194 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:38.558916    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89326bce-8281-4747-aa3f-c1d37c615234-istio-token\") pod \"e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf\" (UID: \"89326bce-8281-4747-aa3f-c1d37c615234\") " pod="openshift-ingress/e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf"
Oct 30 07:32:38.559194 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:38.558997    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89326bce-8281-4747-aa3f-c1d37c615234-istiod-ca-cert\") pod \"e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf\" (UID: \"89326bce-8281-4747-aa3f-c1d37c615234\") " pod="openshift-ingress/e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf"
Oct 30 07:32:38.559194 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:38.559070    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89326bce-8281-4747-aa3f-c1d37c615234-workload-socket\") pod \"e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf\" (UID: \"89326bce-8281-4747-aa3f-c1d37c615234\") " pod="openshift-ingress/e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf"
Oct 30 07:32:38.559194 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:38.559185    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rqh2\" (UniqueName: \"kubernetes.io/projected/89326bce-8281-4747-aa3f-c1d37c615234-kube-api-access-9rqh2\") pod \"e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf\" (UID: \"89326bce-8281-4747-aa3f-c1d37c615234\") " pod="openshift-ingress/e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf"
Oct 30 07:32:38.559461 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:38.559225    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89326bce-8281-4747-aa3f-c1d37c615234-istio-podinfo\") pod \"e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf\" (UID: \"89326bce-8281-4747-aa3f-c1d37c615234\") " pod="openshift-ingress/e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf"
Oct 30 07:32:38.562835 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:38.560135    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89326bce-8281-4747-aa3f-c1d37c615234-credential-socket\") pod \"e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf\" (UID: \"89326bce-8281-4747-aa3f-c1d37c615234\") " pod="openshift-ingress/e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf"
Oct 30 07:32:38.562835 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:38.560377    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89326bce-8281-4747-aa3f-c1d37c615234-istio-data\") pod \"e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf\" (UID: \"89326bce-8281-4747-aa3f-c1d37c615234\") " pod="openshift-ingress/e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf"
Oct 30 07:32:38.562835 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:38.561467    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89326bce-8281-4747-aa3f-c1d37c615234-workload-certs\") pod \"e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf\" (UID: \"89326bce-8281-4747-aa3f-c1d37c615234\") " pod="openshift-ingress/e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf"
Oct 30 07:32:38.562835 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:38.562380    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89326bce-8281-4747-aa3f-c1d37c615234-istiod-ca-cert\") pod \"e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf\" (UID: \"89326bce-8281-4747-aa3f-c1d37c615234\") " pod="openshift-ingress/e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf"
Oct 30 07:32:38.585873 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:38.581516    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89326bce-8281-4747-aa3f-c1d37c615234-workload-socket\") pod \"e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf\" (UID: \"89326bce-8281-4747-aa3f-c1d37c615234\") " pod="openshift-ingress/e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf"
Oct 30 07:32:38.585873 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:38.581888    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89326bce-8281-4747-aa3f-c1d37c615234-istio-envoy\") pod \"e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf\" (UID: \"89326bce-8281-4747-aa3f-c1d37c615234\") " pod="openshift-ingress/e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf"
Oct 30 07:32:38.594827 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:38.587784    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89326bce-8281-4747-aa3f-c1d37c615234-istio-podinfo\") pod \"e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf\" (UID: \"89326bce-8281-4747-aa3f-c1d37c615234\") " pod="openshift-ingress/e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf"
Oct 30 07:32:38.594827 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:38.588646    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89326bce-8281-4747-aa3f-c1d37c615234-istio-token\") pod \"e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf\" (UID: \"89326bce-8281-4747-aa3f-c1d37c615234\") " pod="openshift-ingress/e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf"
Oct 30 07:32:38.609830 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:38.607444    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rqh2\" (UniqueName: \"kubernetes.io/projected/89326bce-8281-4747-aa3f-c1d37c615234-kube-api-access-9rqh2\") pod \"e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf\" (UID: \"89326bce-8281-4747-aa3f-c1d37c615234\") " pod="openshift-ingress/e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf"
Oct 30 07:32:38.650358 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:38.650324    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf"
Oct 30 07:32:38.976956 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:32:38.976926    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89326bce_8281_4747_aa3f_c1d37c615234.slice/crio-89c6d5643fc0e465fee073874ef2e0236df3fa2465dcba3d516bd03d8e0940f8 WatchSource:0}: Error finding container 89c6d5643fc0e465fee073874ef2e0236df3fa2465dcba3d516bd03d8e0940f8: Status 404 returned error can't find the container with id 89c6d5643fc0e465fee073874ef2e0236df3fa2465dcba3d516bd03d8e0940f8
Oct 30 07:32:38.977079 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:38.977031    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf"]
Oct 30 07:32:38.979797 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:38.979761    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:32:38.979931 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:38.979856    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:32:38.979931 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:38.979903    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:32:39.429478 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:39.429440    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf" event={"ID":"89326bce-8281-4747-aa3f-c1d37c615234","Type":"ContainerStarted","Data":"f83e71bb373bc44313b4a523c72c94150cc2ef2427a9d57a48c1de416ed47e29"}
Oct 30 07:32:39.430027 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:39.429502    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf" event={"ID":"89326bce-8281-4747-aa3f-c1d37c615234","Type":"ContainerStarted","Data":"89c6d5643fc0e465fee073874ef2e0236df3fa2465dcba3d516bd03d8e0940f8"}
Oct 30 07:32:39.507215 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:39.507170    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf" podStartSLOduration=1.50715662 podStartE2EDuration="1.50715662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:32:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:32:39.493142223 +0000 UTC m=+2118.102549194" watchObservedRunningTime="2026-10-30 07:32:39.50715662 +0000 UTC m=+2118.116563589"
Oct 30 07:32:39.650479 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:39.650449    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf"
Oct 30 07:32:40.658167 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:40.658131    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf"
Oct 30 07:32:41.443520 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:41.443484    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf"
Oct 30 07:32:41.444437 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:41.444415    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf"
Oct 30 07:32:41.515638 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:41.515596    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-932c00/auth-test-model-932c00-kserve-5496946777-b8b4x" podUID="72f1915d-8b19-4459-8450-436b4c30ba38" containerName="main" containerID="cri-o://f2accd87f99618c7f295c4bc73161d4fb2083de40a5fc22369e8daa6173f3f1a" gracePeriod=45
Oct 30 07:32:41.673960 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:41.673922    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-932c00/auth-test-model-932c00-kserve-5496946777-b8b4x" podUID="72f1915d-8b19-4459-8450-436b4c30ba38" containerName="main" probeResult="failure" output="Get \"https://10.134.0.124:8000/ready\": dial tcp 10.134.0.124:8000: connect: connection refused"
Oct 30 07:32:41.793904 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:41.793879    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-932c00/auth-test-model-932c00-kserve-5496946777-b8b4x"
Oct 30 07:32:41.910029 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:41.909997    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72f1915d-8b19-4459-8450-436b4c30ba38-dshm\" (UniqueName: \"kubernetes.io/empty-dir/72f1915d-8b19-4459-8450-436b4c30ba38-dshm\") pod \"72f1915d-8b19-4459-8450-436b4c30ba38\" (UID: \"72f1915d-8b19-4459-8450-436b4c30ba38\") "
Oct 30 07:32:41.910215 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:41.910102    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72f1915d-8b19-4459-8450-436b4c30ba38-kube-api-access-qxxqq\" (UniqueName: \"kubernetes.io/projected/72f1915d-8b19-4459-8450-436b4c30ba38-kube-api-access-qxxqq\") pod \"72f1915d-8b19-4459-8450-436b4c30ba38\" (UID: \"72f1915d-8b19-4459-8450-436b4c30ba38\") "
Oct 30 07:32:41.910215 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:41.910166    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72f1915d-8b19-4459-8450-436b4c30ba38-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72f1915d-8b19-4459-8450-436b4c30ba38-model-cache\") pod \"72f1915d-8b19-4459-8450-436b4c30ba38\" (UID: \"72f1915d-8b19-4459-8450-436b4c30ba38\") "
Oct 30 07:32:41.910338 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:41.910228    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72f1915d-8b19-4459-8450-436b4c30ba38-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72f1915d-8b19-4459-8450-436b4c30ba38-tmp-dir\") pod \"72f1915d-8b19-4459-8450-436b4c30ba38\" (UID: \"72f1915d-8b19-4459-8450-436b4c30ba38\") "
Oct 30 07:32:41.910338 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:41.910278    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72f1915d-8b19-4459-8450-436b4c30ba38-tls-certs\" (UniqueName: \"kubernetes.io/secret/72f1915d-8b19-4459-8450-436b4c30ba38-tls-certs\") pod \"72f1915d-8b19-4459-8450-436b4c30ba38\" (UID: \"72f1915d-8b19-4459-8450-436b4c30ba38\") "
Oct 30 07:32:41.910338 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:41.910305    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72f1915d-8b19-4459-8450-436b4c30ba38-home\" (UniqueName: \"kubernetes.io/empty-dir/72f1915d-8b19-4459-8450-436b4c30ba38-home\") pod \"72f1915d-8b19-4459-8450-436b4c30ba38\" (UID: \"72f1915d-8b19-4459-8450-436b4c30ba38\") "
Oct 30 07:32:41.910500 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:41.910458    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f1915d-8b19-4459-8450-436b4c30ba38-model-cache" pod "72f1915d-8b19-4459-8450-436b4c30ba38" (UID: "72f1915d-8b19-4459-8450-436b4c30ba38"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:32:41.910553 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:41.910536    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f1915d-8b19-4459-8450-436b4c30ba38-tmp-dir" pod "72f1915d-8b19-4459-8450-436b4c30ba38" (UID: "72f1915d-8b19-4459-8450-436b4c30ba38"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:32:41.910683 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:41.910646    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f1915d-8b19-4459-8450-436b4c30ba38-home" pod "72f1915d-8b19-4459-8450-436b4c30ba38" (UID: "72f1915d-8b19-4459-8450-436b4c30ba38"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:32:41.910975 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:41.910955    6224 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72f1915d-8b19-4459-8450-436b4c30ba38-tmp-dir\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:32:41.910975 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:41.910975    6224 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72f1915d-8b19-4459-8450-436b4c30ba38-home\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:32:41.911114 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:41.910985    6224 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72f1915d-8b19-4459-8450-436b4c30ba38-model-cache\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:32:41.912319 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:41.912295    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f1915d-8b19-4459-8450-436b4c30ba38-dshm" pod "72f1915d-8b19-4459-8450-436b4c30ba38" (UID: "72f1915d-8b19-4459-8450-436b4c30ba38"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:32:41.912417 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:41.912386    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72f1915d-8b19-4459-8450-436b4c30ba38-tls-certs" pod "72f1915d-8b19-4459-8450-436b4c30ba38" (UID: "72f1915d-8b19-4459-8450-436b4c30ba38"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:32:41.912469 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:41.912422    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72f1915d-8b19-4459-8450-436b4c30ba38-kube-api-access-qxxqq" pod "72f1915d-8b19-4459-8450-436b4c30ba38" (UID: "72f1915d-8b19-4459-8450-436b4c30ba38"). InnerVolumeSpecName "kube-api-access-qxxqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:32:42.011797 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:42.011712    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxxqq\" (UniqueName: \"kubernetes.io/projected/72f1915d-8b19-4459-8450-436b4c30ba38-kube-api-access-qxxqq\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:32:42.011797 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:42.011745    6224 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72f1915d-8b19-4459-8450-436b4c30ba38-tls-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:32:42.011797 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:42.011754    6224 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72f1915d-8b19-4459-8450-436b4c30ba38-dshm\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:32:42.418177 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:42.418149    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8f1df-f98c8b658-g2h4t"
Oct 30 07:32:42.451454 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:42.451422    6224 generic.go:356] "Generic (PLEG): container finished" podID="72f1915d-8b19-4459-8450-436b4c30ba38" containerID="f2accd87f99618c7f295c4bc73161d4fb2083de40a5fc22369e8daa6173f3f1a" exitCode=0
Oct 30 07:32:42.451616 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:42.451481    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-932c00/auth-test-model-932c00-kserve-5496946777-b8b4x"
Oct 30 07:32:42.451616 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:42.451506    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-932c00/auth-test-model-932c00-kserve-5496946777-b8b4x" event={"ID":"72f1915d-8b19-4459-8450-436b4c30ba38","Type":"ContainerDied","Data":"f2accd87f99618c7f295c4bc73161d4fb2083de40a5fc22369e8daa6173f3f1a"}
Oct 30 07:32:42.451616 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:42.451545    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-932c00/auth-test-model-932c00-kserve-5496946777-b8b4x" event={"ID":"72f1915d-8b19-4459-8450-436b4c30ba38","Type":"ContainerDied","Data":"058dae9ad3a94d88212f2dfa0c2b6030f4398ba8054dcc65940f2a3165a7bb86"}
Oct 30 07:32:42.451616 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:42.451565    6224 scope.go:122] "RemoveContainer" containerID="f2accd87f99618c7f295c4bc73161d4fb2083de40a5fc22369e8daa6173f3f1a"
Oct 30 07:32:42.465424 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:42.464765    6224 scope.go:122] "RemoveContainer" containerID="f2accd87f99618c7f295c4bc73161d4fb2083de40a5fc22369e8daa6173f3f1a"
Oct 30 07:32:42.465424 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:32:42.465354    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2accd87f99618c7f295c4bc73161d4fb2083de40a5fc22369e8daa6173f3f1a\": container with ID starting with f2accd87f99618c7f295c4bc73161d4fb2083de40a5fc22369e8daa6173f3f1a not found: ID does not exist" containerID="f2accd87f99618c7f295c4bc73161d4fb2083de40a5fc22369e8daa6173f3f1a"
Oct 30 07:32:42.465424 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:42.465386    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2accd87f99618c7f295c4bc73161d4fb2083de40a5fc22369e8daa6173f3f1a"} err="failed to get container status \"f2accd87f99618c7f295c4bc73161d4fb2083de40a5fc22369e8daa6173f3f1a\": rpc error: code = NotFound desc = could not find container \"f2accd87f99618c7f295c4bc73161d4fb2083de40a5fc22369e8daa6173f3f1a\": container with ID starting with f2accd87f99618c7f295c4bc73161d4fb2083de40a5fc22369e8daa6173f3f1a not found: ID does not exist"
Oct 30 07:32:42.517375 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:42.517325    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-932c00/auth-test-model-932c00-kserve-5496946777-b8b4x"]
Oct 30 07:32:42.522531 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:42.522498    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-932c00/auth-test-model-932c00-kserve-5496946777-b8b4x"]
Oct 30 07:32:43.504110 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:43.504078    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72f1915d-8b19-4459-8450-436b4c30ba38" path="/var/lib/kubelet/pods/72f1915d-8b19-4459-8450-436b4c30ba38/volumes"
Oct 30 07:32:54.478965 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:32:54.478878    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:33:01.859282 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:33:01.859241    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:33:19.000979 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:19.000944    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-b5998f/auth-test-model-b5998f-kserve-79b66ddd9c-56qmh"]
Oct 30 07:33:21.269045 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:21.269012    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-ace8fb-5bc7f544dc-nbdvs"]
Oct 30 07:33:21.269673 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:21.269656    6224 cpu_manager.go:414] "containerMap: removing container" podUID="72f1915d-8b19-4459-8450-436b4c30ba38" containerName="main"
Oct 30 07:33:21.269731 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:21.269674    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72f1915d-8b19-4459-8450-436b4c30ba38" containerName="main"
Oct 30 07:33:21.269800 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:21.269790    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="72f1915d-8b19-4459-8450-436b4c30ba38" containerName="main"
Oct 30 07:33:21.273132 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:21.273105    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ace8fb-5bc7f544dc-nbdvs"
Oct 30 07:33:21.293148 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:21.293115    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-ace8fb-5bc7f544dc-nbdvs"]
Oct 30 07:33:21.303446 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:21.303421    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-ace8fb-dockercfg-78c4c\""
Oct 30 07:33:21.329305 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:21.329277    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-ace8fb\""
Oct 30 07:33:21.346202 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:21.337933    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpcj4\" (UniqueName: \"kubernetes.io/projected/cce353ad-244f-40c3-95e8-9dc84e054f89-kube-api-access-wpcj4\") pod \"payload-pre-processing-e2e-extmod-b-ace8fb-5bc7f544dc-nbdvs\" (UID: \"cce353ad-244f-40c3-95e8-9dc84e054f89\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ace8fb-5bc7f544dc-nbdvs"
Oct 30 07:33:21.346202 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:21.338060    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cce353ad-244f-40c3-95e8-9dc84e054f89-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-ace8fb-5bc7f544dc-nbdvs\" (UID: \"cce353ad-244f-40c3-95e8-9dc84e054f89\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ace8fb-5bc7f544dc-nbdvs"
Oct 30 07:33:21.454800 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:21.441638    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpcj4\" (UniqueName: \"kubernetes.io/projected/cce353ad-244f-40c3-95e8-9dc84e054f89-kube-api-access-wpcj4\") pod \"payload-pre-processing-e2e-extmod-b-ace8fb-5bc7f544dc-nbdvs\" (UID: \"cce353ad-244f-40c3-95e8-9dc84e054f89\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ace8fb-5bc7f544dc-nbdvs"
Oct 30 07:33:21.454800 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:21.441759    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cce353ad-244f-40c3-95e8-9dc84e054f89-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-ace8fb-5bc7f544dc-nbdvs\" (UID: \"cce353ad-244f-40c3-95e8-9dc84e054f89\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ace8fb-5bc7f544dc-nbdvs"
Oct 30 07:33:21.454800 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:21.442913    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cce353ad-244f-40c3-95e8-9dc84e054f89-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-ace8fb-5bc7f544dc-nbdvs\" (UID: \"cce353ad-244f-40c3-95e8-9dc84e054f89\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ace8fb-5bc7f544dc-nbdvs"
Oct 30 07:33:21.476943 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:21.476911    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpcj4\" (UniqueName: \"kubernetes.io/projected/cce353ad-244f-40c3-95e8-9dc84e054f89-kube-api-access-wpcj4\") pod \"payload-pre-processing-e2e-extmod-b-ace8fb-5bc7f544dc-nbdvs\" (UID: \"cce353ad-244f-40c3-95e8-9dc84e054f89\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ace8fb-5bc7f544dc-nbdvs"
Oct 30 07:33:21.586497 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:21.586465    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ace8fb-5bc7f544dc-nbdvs"
Oct 30 07:33:21.918636 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:33:21.918600    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcce353ad_244f_40c3_95e8_9dc84e054f89.slice/crio-c7e145dd053cd95c59733562e25fdd1a4d3a351d4fe148b017db9c172e85e2d4 WatchSource:0}: Error finding container c7e145dd053cd95c59733562e25fdd1a4d3a351d4fe148b017db9c172e85e2d4: Status 404 returned error can't find the container with id c7e145dd053cd95c59733562e25fdd1a4d3a351d4fe148b017db9c172e85e2d4
Oct 30 07:33:21.920693 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:21.920468    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-ace8fb-5bc7f544dc-nbdvs"]
Oct 30 07:33:22.755226 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:22.755183    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ace8fb-5bc7f544dc-nbdvs" event={"ID":"cce353ad-244f-40c3-95e8-9dc84e054f89","Type":"ContainerStarted","Data":"0b6aafcc3e765018c05c237267e31cea7d47c5014aa6789cde4a90abc38844df"}
Oct 30 07:33:22.755655 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:22.755247    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ace8fb-5bc7f544dc-nbdvs"
Oct 30 07:33:22.755655 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:22.755264    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ace8fb-5bc7f544dc-nbdvs" event={"ID":"cce353ad-244f-40c3-95e8-9dc84e054f89","Type":"ContainerStarted","Data":"c7e145dd053cd95c59733562e25fdd1a4d3a351d4fe148b017db9c172e85e2d4"}
Oct 30 07:33:22.807656 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:22.807606    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ace8fb-5bc7f544dc-nbdvs" podStartSLOduration=1.807592014 podStartE2EDuration="1.807592014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:33:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:33:22.806215217 +0000 UTC m=+2161.415622188" watchObservedRunningTime="2026-10-30 07:33:22.807592014 +0000 UTC m=+2161.416999003"
Oct 30 07:33:25.543066 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:25.543029    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs"]
Oct 30 07:33:25.543646 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:25.543277    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs" podUID="454321e8-5bec-497e-9789-7ed6604ed72c" containerName="istio-proxy" containerID="cri-o://b9556ab6882d2a50c45cc2cb15b9772a45b75b6ab3042e6e5d1029600136791d" gracePeriod=30
Oct 30 07:33:25.557998 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:25.557969    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg"]
Oct 30 07:33:25.558204 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:25.558179    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg" podUID="3c257ea9-cd39-419d-8960-43069d3eaecc" containerName="istio-proxy" containerID="cri-o://c1d89ca8a7acf24ba597ebbfe9247c1d752ac7a493d0a9339d94b2f630bd49ce" gracePeriod=30
Oct 30 07:33:26.422290 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:26.422256    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-1f34aa-6c5cc4f795-pqzbx"]
Oct 30 07:33:26.428084 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:26.428063    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1f34aa-6c5cc4f795-pqzbx"
Oct 30 07:33:26.445299 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:26.445258    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-1f34aa-dockercfg-65p6p\""
Oct 30 07:33:26.446477 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:26.446453    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-1f34aa-6c5cc4f795-pqzbx"]
Oct 30 07:33:26.486016 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:26.485989    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-1f34aa\""
Oct 30 07:33:26.568843 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:26.565397    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ea09206-913c-471c-b942-4a26878ee5e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-1f34aa-6c5cc4f795-pqzbx\" (UID: \"1ea09206-913c-471c-b942-4a26878ee5e6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1f34aa-6c5cc4f795-pqzbx"
Oct 30 07:33:26.568843 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:26.565718    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgx6f\" (UniqueName: \"kubernetes.io/projected/1ea09206-913c-471c-b942-4a26878ee5e6-kube-api-access-wgx6f\") pod \"payload-pre-processing-e2e-extmod-b-1f34aa-6c5cc4f795-pqzbx\" (UID: \"1ea09206-913c-471c-b942-4a26878ee5e6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1f34aa-6c5cc4f795-pqzbx"
Oct 30 07:33:26.666541 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:26.666508    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ea09206-913c-471c-b942-4a26878ee5e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-1f34aa-6c5cc4f795-pqzbx\" (UID: \"1ea09206-913c-471c-b942-4a26878ee5e6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1f34aa-6c5cc4f795-pqzbx"
Oct 30 07:33:26.666848 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:26.666829    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgx6f\" (UniqueName: \"kubernetes.io/projected/1ea09206-913c-471c-b942-4a26878ee5e6-kube-api-access-wgx6f\") pod \"payload-pre-processing-e2e-extmod-b-1f34aa-6c5cc4f795-pqzbx\" (UID: \"1ea09206-913c-471c-b942-4a26878ee5e6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1f34aa-6c5cc4f795-pqzbx"
Oct 30 07:33:26.667314 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:26.667298    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ea09206-913c-471c-b942-4a26878ee5e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-1f34aa-6c5cc4f795-pqzbx\" (UID: \"1ea09206-913c-471c-b942-4a26878ee5e6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1f34aa-6c5cc4f795-pqzbx"
Oct 30 07:33:26.694480 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:26.694425    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgx6f\" (UniqueName: \"kubernetes.io/projected/1ea09206-913c-471c-b942-4a26878ee5e6-kube-api-access-wgx6f\") pod \"payload-pre-processing-e2e-extmod-b-1f34aa-6c5cc4f795-pqzbx\" (UID: \"1ea09206-913c-471c-b942-4a26878ee5e6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1f34aa-6c5cc4f795-pqzbx"
Oct 30 07:33:26.742279 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:26.742253    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1f34aa-6c5cc4f795-pqzbx"
Oct 30 07:33:27.044318 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:33:27.044289    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ea09206_913c_471c_b942_4a26878ee5e6.slice/crio-ad3ce3c71e4195aba0cee6b1f5bb0ebf339240efe0eb7dcc0b77ddb186427b5f WatchSource:0}: Error finding container ad3ce3c71e4195aba0cee6b1f5bb0ebf339240efe0eb7dcc0b77ddb186427b5f: Status 404 returned error can't find the container with id ad3ce3c71e4195aba0cee6b1f5bb0ebf339240efe0eb7dcc0b77ddb186427b5f
Oct 30 07:33:27.045372 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:27.045347    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-1f34aa-6c5cc4f795-pqzbx"]
Oct 30 07:33:27.851101 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:27.851059    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1f34aa-6c5cc4f795-pqzbx" event={"ID":"1ea09206-913c-471c-b942-4a26878ee5e6","Type":"ContainerStarted","Data":"3a39589607e717e403a08cb455e2d6b8de2e16c8b3edf6287415780351c62b4c"}
Oct 30 07:33:27.851635 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:27.851133    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1f34aa-6c5cc4f795-pqzbx" event={"ID":"1ea09206-913c-471c-b942-4a26878ee5e6","Type":"ContainerStarted","Data":"ad3ce3c71e4195aba0cee6b1f5bb0ebf339240efe0eb7dcc0b77ddb186427b5f"}
Oct 30 07:33:27.852229 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:27.852204    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1f34aa-6c5cc4f795-pqzbx"
Oct 30 07:33:27.900991 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:27.900939    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1f34aa-6c5cc4f795-pqzbx" podStartSLOduration=1.9009265819999999 podStartE2EDuration="1.900926582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:33:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:33:27.898106152 +0000 UTC m=+2166.507513121" watchObservedRunningTime="2026-10-30 07:33:27.900926582 +0000 UTC m=+2166.510333572"
Oct 30 07:33:30.842771 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:30.842720    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs"
Oct 30 07:33:30.876874 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:30.876844    6224 generic.go:356] "Generic (PLEG): container finished" podID="3c257ea9-cd39-419d-8960-43069d3eaecc" containerID="c1d89ca8a7acf24ba597ebbfe9247c1d752ac7a493d0a9339d94b2f630bd49ce" exitCode=0
Oct 30 07:33:30.877027 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:30.876850    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg" event={"ID":"3c257ea9-cd39-419d-8960-43069d3eaecc","Type":"ContainerDied","Data":"c1d89ca8a7acf24ba597ebbfe9247c1d752ac7a493d0a9339d94b2f630bd49ce"}
Oct 30 07:33:30.878370 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:30.878347    6224 generic.go:356] "Generic (PLEG): container finished" podID="454321e8-5bec-497e-9789-7ed6604ed72c" containerID="b9556ab6882d2a50c45cc2cb15b9772a45b75b6ab3042e6e5d1029600136791d" exitCode=0
Oct 30 07:33:30.878496 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:30.878392    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs"
Oct 30 07:33:30.878496 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:30.878435    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs" event={"ID":"454321e8-5bec-497e-9789-7ed6604ed72c","Type":"ContainerDied","Data":"b9556ab6882d2a50c45cc2cb15b9772a45b75b6ab3042e6e5d1029600136791d"}
Oct 30 07:33:30.878584 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:30.878495    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs" event={"ID":"454321e8-5bec-497e-9789-7ed6604ed72c","Type":"ContainerDied","Data":"af37ff448b9c70fae180276f7d982c22420e433b1da9f1bc0695f3ac5f342d69"}
Oct 30 07:33:30.878584 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:30.878524    6224 scope.go:122] "RemoveContainer" containerID="b9556ab6882d2a50c45cc2cb15b9772a45b75b6ab3042e6e5d1029600136791d"
Oct 30 07:33:30.899172 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:30.899155    6224 scope.go:122] "RemoveContainer" containerID="b9556ab6882d2a50c45cc2cb15b9772a45b75b6ab3042e6e5d1029600136791d"
Oct 30 07:33:30.899454 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:33:30.899434    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9556ab6882d2a50c45cc2cb15b9772a45b75b6ab3042e6e5d1029600136791d\": container with ID starting with b9556ab6882d2a50c45cc2cb15b9772a45b75b6ab3042e6e5d1029600136791d not found: ID does not exist" containerID="b9556ab6882d2a50c45cc2cb15b9772a45b75b6ab3042e6e5d1029600136791d"
Oct 30 07:33:30.899520 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:30.899464    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9556ab6882d2a50c45cc2cb15b9772a45b75b6ab3042e6e5d1029600136791d"} err="failed to get container status \"b9556ab6882d2a50c45cc2cb15b9772a45b75b6ab3042e6e5d1029600136791d\": rpc error: code = NotFound desc = could not find container \"b9556ab6882d2a50c45cc2cb15b9772a45b75b6ab3042e6e5d1029600136791d\": container with ID starting with b9556ab6882d2a50c45cc2cb15b9772a45b75b6ab3042e6e5d1029600136791d not found: ID does not exist"
Oct 30 07:33:30.902778 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:30.902745    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg"
Oct 30 07:33:30.928696 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:30.928668    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/454321e8-5bec-497e-9789-7ed6604ed72c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/454321e8-5bec-497e-9789-7ed6604ed72c-istiod-ca-cert\") pod \"454321e8-5bec-497e-9789-7ed6604ed72c\" (UID: \"454321e8-5bec-497e-9789-7ed6604ed72c\") "
Oct 30 07:33:30.928863 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:30.928703    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c257ea9-cd39-419d-8960-43069d3eaecc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c257ea9-cd39-419d-8960-43069d3eaecc-istio-podinfo\") pod \"3c257ea9-cd39-419d-8960-43069d3eaecc\" (UID: \"3c257ea9-cd39-419d-8960-43069d3eaecc\") "
Oct 30 07:33:30.928863 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:30.928747    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c257ea9-cd39-419d-8960-43069d3eaecc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c257ea9-cd39-419d-8960-43069d3eaecc-istiod-ca-cert\") pod \"3c257ea9-cd39-419d-8960-43069d3eaecc\" (UID: \"3c257ea9-cd39-419d-8960-43069d3eaecc\") "
Oct 30 07:33:30.928863 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:30.928783    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/454321e8-5bec-497e-9789-7ed6604ed72c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/454321e8-5bec-497e-9789-7ed6604ed72c-istio-envoy\") pod \"454321e8-5bec-497e-9789-7ed6604ed72c\" (UID: \"454321e8-5bec-497e-9789-7ed6604ed72c\") "
Oct 30 07:33:30.928863 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:30.928844    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/454321e8-5bec-497e-9789-7ed6604ed72c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/454321e8-5bec-497e-9789-7ed6604ed72c-workload-socket\") pod \"454321e8-5bec-497e-9789-7ed6604ed72c\" (UID: \"454321e8-5bec-497e-9789-7ed6604ed72c\") "
Oct 30 07:33:30.929020 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:30.928885    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c257ea9-cd39-419d-8960-43069d3eaecc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c257ea9-cd39-419d-8960-43069d3eaecc-istio-data\") pod \"3c257ea9-cd39-419d-8960-43069d3eaecc\" (UID: \"3c257ea9-cd39-419d-8960-43069d3eaecc\") "
Oct 30 07:33:30.929020 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:30.928942    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/454321e8-5bec-497e-9789-7ed6604ed72c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/454321e8-5bec-497e-9789-7ed6604ed72c-istio-podinfo\") pod \"454321e8-5bec-497e-9789-7ed6604ed72c\" (UID: \"454321e8-5bec-497e-9789-7ed6604ed72c\") "
Oct 30 07:33:30.929020 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:30.928997    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c257ea9-cd39-419d-8960-43069d3eaecc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c257ea9-cd39-419d-8960-43069d3eaecc-credential-socket\") pod \"3c257ea9-cd39-419d-8960-43069d3eaecc\" (UID: \"3c257ea9-cd39-419d-8960-43069d3eaecc\") "
Oct 30 07:33:30.929131 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:30.929084    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/454321e8-5bec-497e-9789-7ed6604ed72c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/454321e8-5bec-497e-9789-7ed6604ed72c-workload-certs\") pod \"454321e8-5bec-497e-9789-7ed6604ed72c\" (UID: \"454321e8-5bec-497e-9789-7ed6604ed72c\") "
Oct 30 07:33:30.929189 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:30.929129    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/454321e8-5bec-497e-9789-7ed6604ed72c-kube-api-access-rwjg6\" (UniqueName: \"kubernetes.io/projected/454321e8-5bec-497e-9789-7ed6604ed72c-kube-api-access-rwjg6\") pod \"454321e8-5bec-497e-9789-7ed6604ed72c\" (UID: \"454321e8-5bec-497e-9789-7ed6604ed72c\") "
Oct 30 07:33:30.929189 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:30.929163    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/454321e8-5bec-497e-9789-7ed6604ed72c-istiod-ca-cert" pod "454321e8-5bec-497e-9789-7ed6604ed72c" (UID: "454321e8-5bec-497e-9789-7ed6604ed72c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:33:30.929298 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:30.929198    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/454321e8-5bec-497e-9789-7ed6604ed72c-istio-token\" (UniqueName: \"kubernetes.io/projected/454321e8-5bec-497e-9789-7ed6604ed72c-istio-token\") pod \"454321e8-5bec-497e-9789-7ed6604ed72c\" (UID: \"454321e8-5bec-497e-9789-7ed6604ed72c\") "
Oct 30 07:33:30.929298 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:30.929239    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c257ea9-cd39-419d-8960-43069d3eaecc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c257ea9-cd39-419d-8960-43069d3eaecc-workload-socket\") pod \"3c257ea9-cd39-419d-8960-43069d3eaecc\" (UID: \"3c257ea9-cd39-419d-8960-43069d3eaecc\") "
Oct 30 07:33:30.929401 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:30.929299    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c257ea9-cd39-419d-8960-43069d3eaecc-istio-data" pod "3c257ea9-cd39-419d-8960-43069d3eaecc" (UID: "3c257ea9-cd39-419d-8960-43069d3eaecc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:33:30.929401 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:30.929222    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c257ea9-cd39-419d-8960-43069d3eaecc-istiod-ca-cert" pod "3c257ea9-cd39-419d-8960-43069d3eaecc" (UID: "3c257ea9-cd39-419d-8960-43069d3eaecc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:33:30.929401 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:30.929368    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c257ea9-cd39-419d-8960-43069d3eaecc-credential-socket" pod "3c257ea9-cd39-419d-8960-43069d3eaecc" (UID: "3c257ea9-cd39-419d-8960-43069d3eaecc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:33:30.929401 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:30.929388    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c257ea9-cd39-419d-8960-43069d3eaecc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c257ea9-cd39-419d-8960-43069d3eaecc-workload-certs\") pod \"3c257ea9-cd39-419d-8960-43069d3eaecc\" (UID: \"3c257ea9-cd39-419d-8960-43069d3eaecc\") "
Oct 30 07:33:30.929614 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:30.929384    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/454321e8-5bec-497e-9789-7ed6604ed72c-workload-socket" pod "454321e8-5bec-497e-9789-7ed6604ed72c" (UID: "454321e8-5bec-497e-9789-7ed6604ed72c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:33:30.929614 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:30.929419    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/454321e8-5bec-497e-9789-7ed6604ed72c-workload-certs" pod "454321e8-5bec-497e-9789-7ed6604ed72c" (UID: "454321e8-5bec-497e-9789-7ed6604ed72c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:33:30.929614 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:30.929446    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c257ea9-cd39-419d-8960-43069d3eaecc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c257ea9-cd39-419d-8960-43069d3eaecc-istio-envoy\") pod \"3c257ea9-cd39-419d-8960-43069d3eaecc\" (UID: \"3c257ea9-cd39-419d-8960-43069d3eaecc\") "
Oct 30 07:33:30.929614 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:30.929473    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/454321e8-5bec-497e-9789-7ed6604ed72c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/454321e8-5bec-497e-9789-7ed6604ed72c-credential-socket\") pod \"454321e8-5bec-497e-9789-7ed6604ed72c\" (UID: \"454321e8-5bec-497e-9789-7ed6604ed72c\") "
Oct 30 07:33:30.929614 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:30.929511    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c257ea9-cd39-419d-8960-43069d3eaecc-kube-api-access-dsjlk\" (UniqueName: \"kubernetes.io/projected/3c257ea9-cd39-419d-8960-43069d3eaecc-kube-api-access-dsjlk\") pod \"3c257ea9-cd39-419d-8960-43069d3eaecc\" (UID: \"3c257ea9-cd39-419d-8960-43069d3eaecc\") "
Oct 30 07:33:30.929614 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:30.929543    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c257ea9-cd39-419d-8960-43069d3eaecc-workload-socket" pod "3c257ea9-cd39-419d-8960-43069d3eaecc" (UID: "3c257ea9-cd39-419d-8960-43069d3eaecc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:33:30.929614 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:30.929551    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c257ea9-cd39-419d-8960-43069d3eaecc-istio-token\" (UniqueName: \"kubernetes.io/projected/3c257ea9-cd39-419d-8960-43069d3eaecc-istio-token\") pod \"3c257ea9-cd39-419d-8960-43069d3eaecc\" (UID: \"3c257ea9-cd39-419d-8960-43069d3eaecc\") "
Oct 30 07:33:30.929614 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:30.929600    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/454321e8-5bec-497e-9789-7ed6604ed72c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/454321e8-5bec-497e-9789-7ed6604ed72c-istio-data\") pod \"454321e8-5bec-497e-9789-7ed6604ed72c\" (UID: \"454321e8-5bec-497e-9789-7ed6604ed72c\") "
Oct 30 07:33:30.930719 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:30.929680    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c257ea9-cd39-419d-8960-43069d3eaecc-workload-certs" pod "3c257ea9-cd39-419d-8960-43069d3eaecc" (UID: "3c257ea9-cd39-419d-8960-43069d3eaecc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:33:30.930719 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:30.929945    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/454321e8-5bec-497e-9789-7ed6604ed72c-credential-socket" pod "454321e8-5bec-497e-9789-7ed6604ed72c" (UID: "454321e8-5bec-497e-9789-7ed6604ed72c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:33:30.930719 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:30.930147    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/454321e8-5bec-497e-9789-7ed6604ed72c-istio-data" pod "454321e8-5bec-497e-9789-7ed6604ed72c" (UID: "454321e8-5bec-497e-9789-7ed6604ed72c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:33:30.930719 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:30.930396    6224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/454321e8-5bec-497e-9789-7ed6604ed72c-istiod-ca-cert\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:33:30.930719 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:30.930416    6224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c257ea9-cd39-419d-8960-43069d3eaecc-istiod-ca-cert\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:33:30.930719 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:30.930430    6224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/454321e8-5bec-497e-9789-7ed6604ed72c-workload-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:33:30.930719 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:30.930444    6224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c257ea9-cd39-419d-8960-43069d3eaecc-istio-data\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:33:30.930719 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:30.930458    6224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c257ea9-cd39-419d-8960-43069d3eaecc-credential-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:33:30.930719 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:30.930471    6224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/454321e8-5bec-497e-9789-7ed6604ed72c-workload-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:33:30.930719 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:30.930484    6224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c257ea9-cd39-419d-8960-43069d3eaecc-workload-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:33:30.930719 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:30.930498    6224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c257ea9-cd39-419d-8960-43069d3eaecc-workload-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:33:30.930719 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:30.930511    6224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/454321e8-5bec-497e-9789-7ed6604ed72c-credential-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:33:30.930719 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:30.930523    6224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/454321e8-5bec-497e-9789-7ed6604ed72c-istio-data\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:33:30.931789 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:30.931519    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c257ea9-cd39-419d-8960-43069d3eaecc-istio-podinfo" pod "3c257ea9-cd39-419d-8960-43069d3eaecc" (UID: "3c257ea9-cd39-419d-8960-43069d3eaecc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:33:30.932365 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:30.932340    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/454321e8-5bec-497e-9789-7ed6604ed72c-istio-token" pod "454321e8-5bec-497e-9789-7ed6604ed72c" (UID: "454321e8-5bec-497e-9789-7ed6604ed72c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:33:30.932575 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:30.932558    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c257ea9-cd39-419d-8960-43069d3eaecc-kube-api-access-dsjlk" pod "3c257ea9-cd39-419d-8960-43069d3eaecc" (UID: "3c257ea9-cd39-419d-8960-43069d3eaecc"). InnerVolumeSpecName "kube-api-access-dsjlk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:33:30.932888 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:30.932855    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c257ea9-cd39-419d-8960-43069d3eaecc-istio-envoy" pod "3c257ea9-cd39-419d-8960-43069d3eaecc" (UID: "3c257ea9-cd39-419d-8960-43069d3eaecc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:33:30.935158 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:30.935134    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c257ea9-cd39-419d-8960-43069d3eaecc-istio-token" pod "3c257ea9-cd39-419d-8960-43069d3eaecc" (UID: "3c257ea9-cd39-419d-8960-43069d3eaecc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:33:30.935297 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:30.935159    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/454321e8-5bec-497e-9789-7ed6604ed72c-kube-api-access-rwjg6" pod "454321e8-5bec-497e-9789-7ed6604ed72c" (UID: "454321e8-5bec-497e-9789-7ed6604ed72c"). InnerVolumeSpecName "kube-api-access-rwjg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:33:30.935297 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:30.935160    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/454321e8-5bec-497e-9789-7ed6604ed72c-istio-envoy" pod "454321e8-5bec-497e-9789-7ed6604ed72c" (UID: "454321e8-5bec-497e-9789-7ed6604ed72c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:33:30.936516 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:30.936484    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/454321e8-5bec-497e-9789-7ed6604ed72c-istio-podinfo" pod "454321e8-5bec-497e-9789-7ed6604ed72c" (UID: "454321e8-5bec-497e-9789-7ed6604ed72c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:33:31.031607 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:31.031573    6224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c257ea9-cd39-419d-8960-43069d3eaecc-istio-envoy\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:33:31.031607 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:31.031601    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsjlk\" (UniqueName: \"kubernetes.io/projected/3c257ea9-cd39-419d-8960-43069d3eaecc-kube-api-access-dsjlk\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:33:31.031607 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:31.031612    6224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c257ea9-cd39-419d-8960-43069d3eaecc-istio-token\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:33:31.031842 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:31.031621    6224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c257ea9-cd39-419d-8960-43069d3eaecc-istio-podinfo\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:33:31.031842 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:31.031630    6224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/454321e8-5bec-497e-9789-7ed6604ed72c-istio-envoy\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:33:31.031842 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:31.031638    6224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/454321e8-5bec-497e-9789-7ed6604ed72c-istio-podinfo\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:33:31.031842 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:31.031646    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwjg6\" (UniqueName: \"kubernetes.io/projected/454321e8-5bec-497e-9789-7ed6604ed72c-kube-api-access-rwjg6\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:33:31.031842 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:31.031653    6224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/454321e8-5bec-497e-9789-7ed6604ed72c-istio-token\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:33:31.230053 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:31.230025    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs"]
Oct 30 07:33:31.238742 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:31.238716    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-hmwvs"]
Oct 30 07:33:31.505972 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:31.505899    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="454321e8-5bec-497e-9789-7ed6604ed72c" path="/var/lib/kubelet/pods/454321e8-5bec-497e-9789-7ed6604ed72c/volumes"
Oct 30 07:33:31.860923 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:33:31.860882    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:33:31.886939 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:31.886915    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg"
Oct 30 07:33:31.887099 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:31.886954    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg" event={"ID":"3c257ea9-cd39-419d-8960-43069d3eaecc","Type":"ContainerDied","Data":"af44eae60be019f8527384d1932d28752af473ca7492799b61acd8d763b1e968"}
Oct 30 07:33:31.887099 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:31.887012    6224 scope.go:122] "RemoveContainer" containerID="c1d89ca8a7acf24ba597ebbfe9247c1d752ac7a493d0a9339d94b2f630bd49ce"
Oct 30 07:33:31.945199 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:31.945118    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg"]
Oct 30 07:33:31.959307 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:31.959281    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-2bkwg"]
Oct 30 07:33:33.504365 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:33.504287    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c257ea9-cd39-419d-8960-43069d3eaecc" path="/var/lib/kubelet/pods/3c257ea9-cd39-419d-8960-43069d3eaecc/volumes"
Oct 30 07:33:33.765615 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:33.765544    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ace8fb-5bc7f544dc-nbdvs"
Oct 30 07:33:34.023856 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:34.023745    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-b5998f/auth-test-model-b5998f-kserve-79b66ddd9c-56qmh" podUID="2a61cd57-5e12-4851-b981-3dd6c7ba0a66" containerName="main" containerID="cri-o://af99855d52056775fbfebb5e33e0ff272df9d42739a38fb2b3ac5df01bb6a2ad" gracePeriod=45
Oct 30 07:33:34.299693 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:34.299661    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-b5998f/auth-test-model-b5998f-kserve-79b66ddd9c-56qmh"
Oct 30 07:33:34.393350 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:34.393320    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a61cd57-5e12-4851-b981-3dd6c7ba0a66-home\" (UniqueName: \"kubernetes.io/empty-dir/2a61cd57-5e12-4851-b981-3dd6c7ba0a66-home\") pod \"2a61cd57-5e12-4851-b981-3dd6c7ba0a66\" (UID: \"2a61cd57-5e12-4851-b981-3dd6c7ba0a66\") "
Oct 30 07:33:34.393350 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:34.393356    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a61cd57-5e12-4851-b981-3dd6c7ba0a66-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a61cd57-5e12-4851-b981-3dd6c7ba0a66-model-cache\") pod \"2a61cd57-5e12-4851-b981-3dd6c7ba0a66\" (UID: \"2a61cd57-5e12-4851-b981-3dd6c7ba0a66\") "
Oct 30 07:33:34.393616 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:34.393442    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a61cd57-5e12-4851-b981-3dd6c7ba0a66-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a61cd57-5e12-4851-b981-3dd6c7ba0a66-tmp-dir\") pod \"2a61cd57-5e12-4851-b981-3dd6c7ba0a66\" (UID: \"2a61cd57-5e12-4851-b981-3dd6c7ba0a66\") "
Oct 30 07:33:34.393616 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:34.393503    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a61cd57-5e12-4851-b981-3dd6c7ba0a66-kube-api-access-5wbfm\" (UniqueName: \"kubernetes.io/projected/2a61cd57-5e12-4851-b981-3dd6c7ba0a66-kube-api-access-5wbfm\") pod \"2a61cd57-5e12-4851-b981-3dd6c7ba0a66\" (UID: \"2a61cd57-5e12-4851-b981-3dd6c7ba0a66\") "
Oct 30 07:33:34.393616 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:34.393547    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a61cd57-5e12-4851-b981-3dd6c7ba0a66-tls-certs\" (UniqueName: \"kubernetes.io/secret/2a61cd57-5e12-4851-b981-3dd6c7ba0a66-tls-certs\") pod \"2a61cd57-5e12-4851-b981-3dd6c7ba0a66\" (UID: \"2a61cd57-5e12-4851-b981-3dd6c7ba0a66\") "
Oct 30 07:33:34.393616 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:34.393591    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a61cd57-5e12-4851-b981-3dd6c7ba0a66-home" pod "2a61cd57-5e12-4851-b981-3dd6c7ba0a66" (UID: "2a61cd57-5e12-4851-b981-3dd6c7ba0a66"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:33:34.393616 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:34.393617    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a61cd57-5e12-4851-b981-3dd6c7ba0a66-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a61cd57-5e12-4851-b981-3dd6c7ba0a66-dshm\") pod \"2a61cd57-5e12-4851-b981-3dd6c7ba0a66\" (UID: \"2a61cd57-5e12-4851-b981-3dd6c7ba0a66\") "
Oct 30 07:33:34.393874 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:34.393685    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a61cd57-5e12-4851-b981-3dd6c7ba0a66-model-cache" pod "2a61cd57-5e12-4851-b981-3dd6c7ba0a66" (UID: "2a61cd57-5e12-4851-b981-3dd6c7ba0a66"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:33:34.393874 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:34.393739    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a61cd57-5e12-4851-b981-3dd6c7ba0a66-tmp-dir" pod "2a61cd57-5e12-4851-b981-3dd6c7ba0a66" (UID: "2a61cd57-5e12-4851-b981-3dd6c7ba0a66"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:33:34.394215 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:34.394193    6224 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a61cd57-5e12-4851-b981-3dd6c7ba0a66-home\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:33:34.394215 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:34.394217    6224 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a61cd57-5e12-4851-b981-3dd6c7ba0a66-model-cache\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:33:34.394404 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:34.394231    6224 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a61cd57-5e12-4851-b981-3dd6c7ba0a66-tmp-dir\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:33:34.396008 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:34.395965    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a61cd57-5e12-4851-b981-3dd6c7ba0a66-kube-api-access-5wbfm" pod "2a61cd57-5e12-4851-b981-3dd6c7ba0a66" (UID: "2a61cd57-5e12-4851-b981-3dd6c7ba0a66"). InnerVolumeSpecName "kube-api-access-5wbfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:33:34.397139 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:34.397066    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a61cd57-5e12-4851-b981-3dd6c7ba0a66-tls-certs" pod "2a61cd57-5e12-4851-b981-3dd6c7ba0a66" (UID: "2a61cd57-5e12-4851-b981-3dd6c7ba0a66"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:33:34.397139 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:34.397099    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a61cd57-5e12-4851-b981-3dd6c7ba0a66-dshm" pod "2a61cd57-5e12-4851-b981-3dd6c7ba0a66" (UID: "2a61cd57-5e12-4851-b981-3dd6c7ba0a66"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:33:34.495384 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:34.495354    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wbfm\" (UniqueName: \"kubernetes.io/projected/2a61cd57-5e12-4851-b981-3dd6c7ba0a66-kube-api-access-5wbfm\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:33:34.495384 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:34.495384    6224 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a61cd57-5e12-4851-b981-3dd6c7ba0a66-tls-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:33:34.495587 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:34.495393    6224 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a61cd57-5e12-4851-b981-3dd6c7ba0a66-dshm\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:33:34.910627 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:34.910594    6224 generic.go:356] "Generic (PLEG): container finished" podID="2a61cd57-5e12-4851-b981-3dd6c7ba0a66" containerID="af99855d52056775fbfebb5e33e0ff272df9d42739a38fb2b3ac5df01bb6a2ad" exitCode=0
Oct 30 07:33:34.910627 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:34.910635    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-b5998f/auth-test-model-b5998f-kserve-79b66ddd9c-56qmh"
Oct 30 07:33:34.919036 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:34.910681    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-b5998f/auth-test-model-b5998f-kserve-79b66ddd9c-56qmh" event={"ID":"2a61cd57-5e12-4851-b981-3dd6c7ba0a66","Type":"ContainerDied","Data":"af99855d52056775fbfebb5e33e0ff272df9d42739a38fb2b3ac5df01bb6a2ad"}
Oct 30 07:33:34.919036 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:34.910721    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-b5998f/auth-test-model-b5998f-kserve-79b66ddd9c-56qmh" event={"ID":"2a61cd57-5e12-4851-b981-3dd6c7ba0a66","Type":"ContainerDied","Data":"72d5f979daea9da44f4ce0d222a3065658a520af9fa7a89e191d5ec173bd3b74"}
Oct 30 07:33:34.919036 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:34.910760    6224 scope.go:122] "RemoveContainer" containerID="af99855d52056775fbfebb5e33e0ff272df9d42739a38fb2b3ac5df01bb6a2ad"
Oct 30 07:33:34.927031 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:34.927014    6224 scope.go:122] "RemoveContainer" containerID="af99855d52056775fbfebb5e33e0ff272df9d42739a38fb2b3ac5df01bb6a2ad"
Oct 30 07:33:34.927285 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:33:34.927266    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af99855d52056775fbfebb5e33e0ff272df9d42739a38fb2b3ac5df01bb6a2ad\": container with ID starting with af99855d52056775fbfebb5e33e0ff272df9d42739a38fb2b3ac5df01bb6a2ad not found: ID does not exist" containerID="af99855d52056775fbfebb5e33e0ff272df9d42739a38fb2b3ac5df01bb6a2ad"
Oct 30 07:33:34.927357 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:34.927292    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af99855d52056775fbfebb5e33e0ff272df9d42739a38fb2b3ac5df01bb6a2ad"} err="failed to get container status \"af99855d52056775fbfebb5e33e0ff272df9d42739a38fb2b3ac5df01bb6a2ad\": rpc error: code = NotFound desc = could not find container \"af99855d52056775fbfebb5e33e0ff272df9d42739a38fb2b3ac5df01bb6a2ad\": container with ID starting with af99855d52056775fbfebb5e33e0ff272df9d42739a38fb2b3ac5df01bb6a2ad not found: ID does not exist"
Oct 30 07:33:34.978144 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:34.978109    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-b5998f/auth-test-model-b5998f-kserve-79b66ddd9c-56qmh"]
Oct 30 07:33:34.991266 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:34.991241    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-b5998f/auth-test-model-b5998f-kserve-79b66ddd9c-56qmh"]
Oct 30 07:33:35.209859 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:35.209776    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-b5998f/auth-test-model-b5998f-kserve-79b66ddd9c-56qmh" podUID="2a61cd57-5e12-4851-b981-3dd6c7ba0a66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.126:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:33:35.503773 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:35.503693    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a61cd57-5e12-4851-b981-3dd6c7ba0a66" path="/var/lib/kubelet/pods/2a61cd57-5e12-4851-b981-3dd6c7ba0a66/volumes"
Oct 30 07:33:39.885831 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:39.885105    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1f34aa-6c5cc4f795-pqzbx"
Oct 30 07:33:40.709448 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:40.709416    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-25f44e/noten-model-25f44e-kserve-544947d8f7-fw8hb"]
Oct 30 07:33:40.710482 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:40.710454    6224 cpu_manager.go:414] "containerMap: removing container" podUID="3c257ea9-cd39-419d-8960-43069d3eaecc" containerName="istio-proxy"
Oct 30 07:33:40.710652 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:40.710628    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c257ea9-cd39-419d-8960-43069d3eaecc" containerName="istio-proxy"
Oct 30 07:33:40.710780 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:40.710663    6224 cpu_manager.go:414] "containerMap: removing container" podUID="454321e8-5bec-497e-9789-7ed6604ed72c" containerName="istio-proxy"
Oct 30 07:33:40.710780 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:40.710671    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="454321e8-5bec-497e-9789-7ed6604ed72c" containerName="istio-proxy"
Oct 30 07:33:40.710780 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:40.710716    6224 cpu_manager.go:414] "containerMap: removing container" podUID="2a61cd57-5e12-4851-b981-3dd6c7ba0a66" containerName="main"
Oct 30 07:33:40.710780 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:40.710724    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a61cd57-5e12-4851-b981-3dd6c7ba0a66" containerName="main"
Oct 30 07:33:40.711025 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:40.710917    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="454321e8-5bec-497e-9789-7ed6604ed72c" containerName="istio-proxy"
Oct 30 07:33:40.711025 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:40.710937    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a61cd57-5e12-4851-b981-3dd6c7ba0a66" containerName="main"
Oct 30 07:33:40.711025 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:40.710955    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c257ea9-cd39-419d-8960-43069d3eaecc" containerName="istio-proxy"
Oct 30 07:33:40.727281 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:40.727253    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-25f44e/noten-model-25f44e-kserve-544947d8f7-fw8hb"
Oct 30 07:33:40.731885 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:40.731861    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-25f44e/noten-model-25f44e-kserve-544947d8f7-fw8hb"]
Oct 30 07:33:40.758366 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:40.758343    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-25f44e\"/\"openshift-service-ca.crt\""
Oct 30 07:33:40.758490 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:40.758399    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-25f44e\"/\"default-dockercfg-z6c57\""
Oct 30 07:33:40.758624 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:40.758610    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-25f44e\"/\"kube-root-ca.crt\""
Oct 30 07:33:40.758688 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:40.758678    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-25f44e\"/\"noten-model-25f44e-kserve-self-signed-certs\""
Oct 30 07:33:40.874209 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:40.874168    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0d444b9-e96c-48f0-9cd2-09610027cdc9-dshm\") pod \"noten-model-25f44e-kserve-544947d8f7-fw8hb\" (UID: \"d0d444b9-e96c-48f0-9cd2-09610027cdc9\") " pod="ai-tenant-e2e-noten-25f44e/noten-model-25f44e-kserve-544947d8f7-fw8hb"
Oct 30 07:33:40.874410 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:40.874234    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0d444b9-e96c-48f0-9cd2-09610027cdc9-tmp-dir\") pod \"noten-model-25f44e-kserve-544947d8f7-fw8hb\" (UID: \"d0d444b9-e96c-48f0-9cd2-09610027cdc9\") " pod="ai-tenant-e2e-noten-25f44e/noten-model-25f44e-kserve-544947d8f7-fw8hb"
Oct 30 07:33:40.874410 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:40.874342    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0d444b9-e96c-48f0-9cd2-09610027cdc9-home\") pod \"noten-model-25f44e-kserve-544947d8f7-fw8hb\" (UID: \"d0d444b9-e96c-48f0-9cd2-09610027cdc9\") " pod="ai-tenant-e2e-noten-25f44e/noten-model-25f44e-kserve-544947d8f7-fw8hb"
Oct 30 07:33:40.874538 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:40.874435    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxhwn\" (UniqueName: \"kubernetes.io/projected/d0d444b9-e96c-48f0-9cd2-09610027cdc9-kube-api-access-mxhwn\") pod \"noten-model-25f44e-kserve-544947d8f7-fw8hb\" (UID: \"d0d444b9-e96c-48f0-9cd2-09610027cdc9\") " pod="ai-tenant-e2e-noten-25f44e/noten-model-25f44e-kserve-544947d8f7-fw8hb"
Oct 30 07:33:40.874601 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:40.874588    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0d444b9-e96c-48f0-9cd2-09610027cdc9-model-cache\") pod \"noten-model-25f44e-kserve-544947d8f7-fw8hb\" (UID: \"d0d444b9-e96c-48f0-9cd2-09610027cdc9\") " pod="ai-tenant-e2e-noten-25f44e/noten-model-25f44e-kserve-544947d8f7-fw8hb"
Oct 30 07:33:40.874656 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:40.874619    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0d444b9-e96c-48f0-9cd2-09610027cdc9-tls-certs\") pod \"noten-model-25f44e-kserve-544947d8f7-fw8hb\" (UID: \"d0d444b9-e96c-48f0-9cd2-09610027cdc9\") " pod="ai-tenant-e2e-noten-25f44e/noten-model-25f44e-kserve-544947d8f7-fw8hb"
Oct 30 07:33:40.980443 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:40.978653    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0d444b9-e96c-48f0-9cd2-09610027cdc9-model-cache\") pod \"noten-model-25f44e-kserve-544947d8f7-fw8hb\" (UID: \"d0d444b9-e96c-48f0-9cd2-09610027cdc9\") " pod="ai-tenant-e2e-noten-25f44e/noten-model-25f44e-kserve-544947d8f7-fw8hb"
Oct 30 07:33:40.980443 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:40.978712    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0d444b9-e96c-48f0-9cd2-09610027cdc9-tls-certs\") pod \"noten-model-25f44e-kserve-544947d8f7-fw8hb\" (UID: \"d0d444b9-e96c-48f0-9cd2-09610027cdc9\") " pod="ai-tenant-e2e-noten-25f44e/noten-model-25f44e-kserve-544947d8f7-fw8hb"
Oct 30 07:33:40.980443 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:40.978839    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0d444b9-e96c-48f0-9cd2-09610027cdc9-dshm\") pod \"noten-model-25f44e-kserve-544947d8f7-fw8hb\" (UID: \"d0d444b9-e96c-48f0-9cd2-09610027cdc9\") " pod="ai-tenant-e2e-noten-25f44e/noten-model-25f44e-kserve-544947d8f7-fw8hb"
Oct 30 07:33:40.980443 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:40.978896    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0d444b9-e96c-48f0-9cd2-09610027cdc9-tmp-dir\") pod \"noten-model-25f44e-kserve-544947d8f7-fw8hb\" (UID: \"d0d444b9-e96c-48f0-9cd2-09610027cdc9\") " pod="ai-tenant-e2e-noten-25f44e/noten-model-25f44e-kserve-544947d8f7-fw8hb"
Oct 30 07:33:40.980443 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:40.978941    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0d444b9-e96c-48f0-9cd2-09610027cdc9-home\") pod \"noten-model-25f44e-kserve-544947d8f7-fw8hb\" (UID: \"d0d444b9-e96c-48f0-9cd2-09610027cdc9\") " pod="ai-tenant-e2e-noten-25f44e/noten-model-25f44e-kserve-544947d8f7-fw8hb"
Oct 30 07:33:40.980443 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:40.979008    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxhwn\" (UniqueName: \"kubernetes.io/projected/d0d444b9-e96c-48f0-9cd2-09610027cdc9-kube-api-access-mxhwn\") pod \"noten-model-25f44e-kserve-544947d8f7-fw8hb\" (UID: \"d0d444b9-e96c-48f0-9cd2-09610027cdc9\") " pod="ai-tenant-e2e-noten-25f44e/noten-model-25f44e-kserve-544947d8f7-fw8hb"
Oct 30 07:33:40.980443 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:40.979867    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0d444b9-e96c-48f0-9cd2-09610027cdc9-model-cache\") pod \"noten-model-25f44e-kserve-544947d8f7-fw8hb\" (UID: \"d0d444b9-e96c-48f0-9cd2-09610027cdc9\") " pod="ai-tenant-e2e-noten-25f44e/noten-model-25f44e-kserve-544947d8f7-fw8hb"
Oct 30 07:33:40.981698 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:40.981669    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0d444b9-e96c-48f0-9cd2-09610027cdc9-tmp-dir\") pod \"noten-model-25f44e-kserve-544947d8f7-fw8hb\" (UID: \"d0d444b9-e96c-48f0-9cd2-09610027cdc9\") " pod="ai-tenant-e2e-noten-25f44e/noten-model-25f44e-kserve-544947d8f7-fw8hb"
Oct 30 07:33:41.006708 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:40.987745    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0d444b9-e96c-48f0-9cd2-09610027cdc9-tls-certs\") pod \"noten-model-25f44e-kserve-544947d8f7-fw8hb\" (UID: \"d0d444b9-e96c-48f0-9cd2-09610027cdc9\") " pod="ai-tenant-e2e-noten-25f44e/noten-model-25f44e-kserve-544947d8f7-fw8hb"
Oct 30 07:33:41.006708 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:40.988030    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0d444b9-e96c-48f0-9cd2-09610027cdc9-home\") pod \"noten-model-25f44e-kserve-544947d8f7-fw8hb\" (UID: \"d0d444b9-e96c-48f0-9cd2-09610027cdc9\") " pod="ai-tenant-e2e-noten-25f44e/noten-model-25f44e-kserve-544947d8f7-fw8hb"
Oct 30 07:33:41.006708 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:41.003666    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0d444b9-e96c-48f0-9cd2-09610027cdc9-dshm\") pod \"noten-model-25f44e-kserve-544947d8f7-fw8hb\" (UID: \"d0d444b9-e96c-48f0-9cd2-09610027cdc9\") " pod="ai-tenant-e2e-noten-25f44e/noten-model-25f44e-kserve-544947d8f7-fw8hb"
Oct 30 07:33:41.018826 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:41.014461    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxhwn\" (UniqueName: \"kubernetes.io/projected/d0d444b9-e96c-48f0-9cd2-09610027cdc9-kube-api-access-mxhwn\") pod \"noten-model-25f44e-kserve-544947d8f7-fw8hb\" (UID: \"d0d444b9-e96c-48f0-9cd2-09610027cdc9\") " pod="ai-tenant-e2e-noten-25f44e/noten-model-25f44e-kserve-544947d8f7-fw8hb"
Oct 30 07:33:41.044830 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:41.043191    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-25f44e/noten-model-25f44e-kserve-544947d8f7-fw8hb"
Oct 30 07:33:41.361514 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:33:41.361479    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0d444b9_e96c_48f0_9cd2_09610027cdc9.slice/crio-10f739c0e36b2c276037b474880d9fad8548c0fda98925ea5ac95bda60d36684 WatchSource:0}: Error finding container 10f739c0e36b2c276037b474880d9fad8548c0fda98925ea5ac95bda60d36684: Status 404 returned error can't find the container with id 10f739c0e36b2c276037b474880d9fad8548c0fda98925ea5ac95bda60d36684
Oct 30 07:33:41.361782 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:41.361758    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-25f44e/noten-model-25f44e-kserve-544947d8f7-fw8hb"]
Oct 30 07:33:41.973159 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:41.973100    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-25f44e/noten-model-25f44e-kserve-544947d8f7-fw8hb" event={"ID":"d0d444b9-e96c-48f0-9cd2-09610027cdc9","Type":"ContainerStarted","Data":"ec8d686853e9cd6e47d0e0ad677df45be40ec1edc31ca21f9a6453cb22d505bb"}
Oct 30 07:33:41.973159 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:41.973166    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-25f44e/noten-model-25f44e-kserve-544947d8f7-fw8hb" event={"ID":"d0d444b9-e96c-48f0-9cd2-09610027cdc9","Type":"ContainerStarted","Data":"10f739c0e36b2c276037b474880d9fad8548c0fda98925ea5ac95bda60d36684"}
Oct 30 07:33:42.018583 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:42.018533    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-25f44e/noten-model-25f44e-kserve-544947d8f7-fw8hb" podStartSLOduration=2.01852002 podStartE2EDuration="2.01852002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:33:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:33:42.017144671 +0000 UTC m=+2180.626551641" watchObservedRunningTime="2026-10-30 07:33:42.01852002 +0000 UTC m=+2180.627927011"
Oct 30 07:33:42.043303 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:42.043277    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-25f44e/noten-model-25f44e-kserve-544947d8f7-fw8hb"
Oct 30 07:33:44.337576 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:44.337530    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh"]
Oct 30 07:33:44.352682 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:44.352656    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh"
Oct 30 07:33:44.371263 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:44.371235    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh"]
Oct 30 07:33:44.376416 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:44.376374    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-f1f700-openshift-default-dockercfg-2bv85\""
Oct 30 07:33:44.528462 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:44.528418    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f5bab32-7cdb-471b-9bd0-38b205ace2e3-credential-socket\") pod \"e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh\" (UID: \"6f5bab32-7cdb-471b-9bd0-38b205ace2e3\") " pod="openshift-ingress/e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh"
Oct 30 07:33:44.529138 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:44.529006    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f5bab32-7cdb-471b-9bd0-38b205ace2e3-istiod-ca-cert\") pod \"e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh\" (UID: \"6f5bab32-7cdb-471b-9bd0-38b205ace2e3\") " pod="openshift-ingress/e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh"
Oct 30 07:33:44.529524 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:44.529507    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f5bab32-7cdb-471b-9bd0-38b205ace2e3-istio-envoy\") pod \"e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh\" (UID: \"6f5bab32-7cdb-471b-9bd0-38b205ace2e3\") " pod="openshift-ingress/e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh"
Oct 30 07:33:44.529696 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:44.529674    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6cv4\" (UniqueName: \"kubernetes.io/projected/6f5bab32-7cdb-471b-9bd0-38b205ace2e3-kube-api-access-t6cv4\") pod \"e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh\" (UID: \"6f5bab32-7cdb-471b-9bd0-38b205ace2e3\") " pod="openshift-ingress/e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh"
Oct 30 07:33:44.530270 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:44.530251    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f5bab32-7cdb-471b-9bd0-38b205ace2e3-workload-socket\") pod \"e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh\" (UID: \"6f5bab32-7cdb-471b-9bd0-38b205ace2e3\") " pod="openshift-ingress/e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh"
Oct 30 07:33:44.530576 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:44.530559    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f5bab32-7cdb-471b-9bd0-38b205ace2e3-istio-podinfo\") pod \"e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh\" (UID: \"6f5bab32-7cdb-471b-9bd0-38b205ace2e3\") " pod="openshift-ingress/e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh"
Oct 30 07:33:44.530877 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:44.530859    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f5bab32-7cdb-471b-9bd0-38b205ace2e3-workload-certs\") pod \"e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh\" (UID: \"6f5bab32-7cdb-471b-9bd0-38b205ace2e3\") " pod="openshift-ingress/e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh"
Oct 30 07:33:44.531112 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:44.531092    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f5bab32-7cdb-471b-9bd0-38b205ace2e3-istio-data\") pod \"e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh\" (UID: \"6f5bab32-7cdb-471b-9bd0-38b205ace2e3\") " pod="openshift-ingress/e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh"
Oct 30 07:33:44.531522 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:44.531503    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f5bab32-7cdb-471b-9bd0-38b205ace2e3-istio-token\") pod \"e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh\" (UID: \"6f5bab32-7cdb-471b-9bd0-38b205ace2e3\") " pod="openshift-ingress/e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh"
Oct 30 07:33:44.638605 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:44.638501    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f5bab32-7cdb-471b-9bd0-38b205ace2e3-workload-socket\") pod \"e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh\" (UID: \"6f5bab32-7cdb-471b-9bd0-38b205ace2e3\") " pod="openshift-ingress/e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh"
Oct 30 07:33:44.640121 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:44.640086    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f5bab32-7cdb-471b-9bd0-38b205ace2e3-istio-podinfo\") pod \"e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh\" (UID: \"6f5bab32-7cdb-471b-9bd0-38b205ace2e3\") " pod="openshift-ingress/e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh"
Oct 30 07:33:44.640284 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:44.640201    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f5bab32-7cdb-471b-9bd0-38b205ace2e3-workload-certs\") pod \"e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh\" (UID: \"6f5bab32-7cdb-471b-9bd0-38b205ace2e3\") " pod="openshift-ingress/e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh"
Oct 30 07:33:44.640284 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:44.640237    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f5bab32-7cdb-471b-9bd0-38b205ace2e3-istio-data\") pod \"e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh\" (UID: \"6f5bab32-7cdb-471b-9bd0-38b205ace2e3\") " pod="openshift-ingress/e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh"
Oct 30 07:33:44.640284 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:44.640272    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f5bab32-7cdb-471b-9bd0-38b205ace2e3-istio-token\") pod \"e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh\" (UID: \"6f5bab32-7cdb-471b-9bd0-38b205ace2e3\") " pod="openshift-ingress/e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh"
Oct 30 07:33:44.640447 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:44.640306    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f5bab32-7cdb-471b-9bd0-38b205ace2e3-credential-socket\") pod \"e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh\" (UID: \"6f5bab32-7cdb-471b-9bd0-38b205ace2e3\") " pod="openshift-ingress/e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh"
Oct 30 07:33:44.640447 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:44.640405    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f5bab32-7cdb-471b-9bd0-38b205ace2e3-istiod-ca-cert\") pod \"e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh\" (UID: \"6f5bab32-7cdb-471b-9bd0-38b205ace2e3\") " pod="openshift-ingress/e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh"
Oct 30 07:33:44.640557 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:44.640510    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f5bab32-7cdb-471b-9bd0-38b205ace2e3-istio-envoy\") pod \"e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh\" (UID: \"6f5bab32-7cdb-471b-9bd0-38b205ace2e3\") " pod="openshift-ingress/e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh"
Oct 30 07:33:44.640610 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:44.640574    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6cv4\" (UniqueName: \"kubernetes.io/projected/6f5bab32-7cdb-471b-9bd0-38b205ace2e3-kube-api-access-t6cv4\") pod \"e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh\" (UID: \"6f5bab32-7cdb-471b-9bd0-38b205ace2e3\") " pod="openshift-ingress/e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh"
Oct 30 07:33:44.641048 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:44.639222    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f5bab32-7cdb-471b-9bd0-38b205ace2e3-workload-socket\") pod \"e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh\" (UID: \"6f5bab32-7cdb-471b-9bd0-38b205ace2e3\") " pod="openshift-ingress/e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh"
Oct 30 07:33:44.642281 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:44.642253    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f5bab32-7cdb-471b-9bd0-38b205ace2e3-istio-data\") pod \"e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh\" (UID: \"6f5bab32-7cdb-471b-9bd0-38b205ace2e3\") " pod="openshift-ingress/e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh"
Oct 30 07:33:44.642661 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:44.642636    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f5bab32-7cdb-471b-9bd0-38b205ace2e3-credential-socket\") pod \"e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh\" (UID: \"6f5bab32-7cdb-471b-9bd0-38b205ace2e3\") " pod="openshift-ingress/e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh"
Oct 30 07:33:44.643034 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:44.642857    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f5bab32-7cdb-471b-9bd0-38b205ace2e3-istiod-ca-cert\") pod \"e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh\" (UID: \"6f5bab32-7cdb-471b-9bd0-38b205ace2e3\") " pod="openshift-ingress/e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh"
Oct 30 07:33:44.643164 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:44.643110    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f5bab32-7cdb-471b-9bd0-38b205ace2e3-workload-certs\") pod \"e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh\" (UID: \"6f5bab32-7cdb-471b-9bd0-38b205ace2e3\") " pod="openshift-ingress/e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh"
Oct 30 07:33:44.646505 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:44.646044    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f5bab32-7cdb-471b-9bd0-38b205ace2e3-istio-podinfo\") pod \"e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh\" (UID: \"6f5bab32-7cdb-471b-9bd0-38b205ace2e3\") " pod="openshift-ingress/e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh"
Oct 30 07:33:44.649284 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:44.649261    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f5bab32-7cdb-471b-9bd0-38b205ace2e3-istio-envoy\") pod \"e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh\" (UID: \"6f5bab32-7cdb-471b-9bd0-38b205ace2e3\") " pod="openshift-ingress/e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh"
Oct 30 07:33:44.668911 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:44.667697    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f5bab32-7cdb-471b-9bd0-38b205ace2e3-istio-token\") pod \"e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh\" (UID: \"6f5bab32-7cdb-471b-9bd0-38b205ace2e3\") " pod="openshift-ingress/e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh"
Oct 30 07:33:44.669072 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:44.668913    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6cv4\" (UniqueName: \"kubernetes.io/projected/6f5bab32-7cdb-471b-9bd0-38b205ace2e3-kube-api-access-t6cv4\") pod \"e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh\" (UID: \"6f5bab32-7cdb-471b-9bd0-38b205ace2e3\") " pod="openshift-ingress/e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh"
Oct 30 07:33:44.672531 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:44.672496    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh"
Oct 30 07:33:45.011514 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:33:45.011485    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f5bab32_7cdb_471b_9bd0_38b205ace2e3.slice/crio-431abbf7a4f5d83a5934f24b81cb09bfb4b86da9b5dd7c1da6371979e8571b29 WatchSource:0}: Error finding container 431abbf7a4f5d83a5934f24b81cb09bfb4b86da9b5dd7c1da6371979e8571b29: Status 404 returned error can't find the container with id 431abbf7a4f5d83a5934f24b81cb09bfb4b86da9b5dd7c1da6371979e8571b29
Oct 30 07:33:45.012341 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:45.012318    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh"]
Oct 30 07:33:45.015018 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:45.014973    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:33:45.015091 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:45.015052    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:33:45.015091 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:45.015085    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:33:45.293264 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:45.293228    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-25f44e/noten-model-25f44e-kserve-544947d8f7-fw8hb"]
Oct 30 07:33:46.005029 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:46.004986    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh" event={"ID":"6f5bab32-7cdb-471b-9bd0-38b205ace2e3","Type":"ContainerStarted","Data":"cb68c675228dc2925720b0e9ca5511cb2a4d461c920b79a8684c7137f0385a91"}
Oct 30 07:33:46.005495 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:46.005090    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh" event={"ID":"6f5bab32-7cdb-471b-9bd0-38b205ace2e3","Type":"ContainerStarted","Data":"431abbf7a4f5d83a5934f24b81cb09bfb4b86da9b5dd7c1da6371979e8571b29"}
Oct 30 07:33:46.673286 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:46.673250    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh"
Oct 30 07:33:46.679084 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:46.679060    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh"
Oct 30 07:33:46.729969 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:46.729920    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh" podStartSLOduration=2.72990464 podStartE2EDuration="2.72990464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:33:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:33:46.0822151 +0000 UTC m=+2184.691622075" watchObservedRunningTime="2026-10-30 07:33:46.72990464 +0000 UTC m=+2185.339311611"
Oct 30 07:33:47.013377 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:47.013237    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh"
Oct 30 07:33:47.014341 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:47.014321    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh"
Oct 30 07:33:50.474241 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:50.474200    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-f1f700-cccdc9f5-wn4nt"]
Oct 30 07:33:50.478744 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:50.478720    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f1f700-cccdc9f5-wn4nt"
Oct 30 07:33:50.498581 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:50.498554    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-f1f700-cccdc9f5-wn4nt"]
Oct 30 07:33:50.508420 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:50.508248    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-f1f700\""
Oct 30 07:33:50.508420 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:50.508372    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-f1f700-dockercfg-gfv5j\""
Oct 30 07:33:50.530563 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:50.530533    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mv6k\" (UniqueName: \"kubernetes.io/projected/0d55b266-9ae5-4b44-85e3-ea14553e4e27-kube-api-access-4mv6k\") pod \"payload-pre-processing-e2e-extmod-b-f1f700-cccdc9f5-wn4nt\" (UID: \"0d55b266-9ae5-4b44-85e3-ea14553e4e27\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f1f700-cccdc9f5-wn4nt"
Oct 30 07:33:50.530687 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:50.530654    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d55b266-9ae5-4b44-85e3-ea14553e4e27-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-f1f700-cccdc9f5-wn4nt\" (UID: \"0d55b266-9ae5-4b44-85e3-ea14553e4e27\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f1f700-cccdc9f5-wn4nt"
Oct 30 07:33:50.621243 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:50.610314    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-f1f700-6db48b9ff4-chndk"]
Oct 30 07:33:50.649039 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:50.640631    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d55b266-9ae5-4b44-85e3-ea14553e4e27-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-f1f700-cccdc9f5-wn4nt\" (UID: \"0d55b266-9ae5-4b44-85e3-ea14553e4e27\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f1f700-cccdc9f5-wn4nt"
Oct 30 07:33:50.649499 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:50.649473    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mv6k\" (UniqueName: \"kubernetes.io/projected/0d55b266-9ae5-4b44-85e3-ea14553e4e27-kube-api-access-4mv6k\") pod \"payload-pre-processing-e2e-extmod-b-f1f700-cccdc9f5-wn4nt\" (UID: \"0d55b266-9ae5-4b44-85e3-ea14553e4e27\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f1f700-cccdc9f5-wn4nt"
Oct 30 07:33:50.650174 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:50.641553    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d55b266-9ae5-4b44-85e3-ea14553e4e27-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-f1f700-cccdc9f5-wn4nt\" (UID: \"0d55b266-9ae5-4b44-85e3-ea14553e4e27\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f1f700-cccdc9f5-wn4nt"
Oct 30 07:33:50.672828 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:50.655285    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-f1f700-6db48b9ff4-chndk"]
Oct 30 07:33:50.672828 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:50.655403    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-f1f700-6db48b9ff4-chndk"
Oct 30 07:33:50.712828 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:50.711747    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mv6k\" (UniqueName: \"kubernetes.io/projected/0d55b266-9ae5-4b44-85e3-ea14553e4e27-kube-api-access-4mv6k\") pod \"payload-pre-processing-e2e-extmod-b-f1f700-cccdc9f5-wn4nt\" (UID: \"0d55b266-9ae5-4b44-85e3-ea14553e4e27\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f1f700-cccdc9f5-wn4nt"
Oct 30 07:33:50.751586 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:50.751557    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1825dd2c-f162-4d90-8ebd-a6923e990abd-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-f1f700-6db48b9ff4-chndk\" (UID: \"1825dd2c-f162-4d90-8ebd-a6923e990abd\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-f1f700-6db48b9ff4-chndk"
Oct 30 07:33:50.751774 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:50.751755    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpvs7\" (UniqueName: \"kubernetes.io/projected/1825dd2c-f162-4d90-8ebd-a6923e990abd-kube-api-access-gpvs7\") pod \"payload-processing-e2e-extmod-b-f1f700-6db48b9ff4-chndk\" (UID: \"1825dd2c-f162-4d90-8ebd-a6923e990abd\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-f1f700-6db48b9ff4-chndk"
Oct 30 07:33:50.792433 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:50.792407    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f1f700-cccdc9f5-wn4nt"
Oct 30 07:33:50.853018 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:50.852982    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1825dd2c-f162-4d90-8ebd-a6923e990abd-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-f1f700-6db48b9ff4-chndk\" (UID: \"1825dd2c-f162-4d90-8ebd-a6923e990abd\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-f1f700-6db48b9ff4-chndk"
Oct 30 07:33:50.853244 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:50.853227    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpvs7\" (UniqueName: \"kubernetes.io/projected/1825dd2c-f162-4d90-8ebd-a6923e990abd-kube-api-access-gpvs7\") pod \"payload-processing-e2e-extmod-b-f1f700-6db48b9ff4-chndk\" (UID: \"1825dd2c-f162-4d90-8ebd-a6923e990abd\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-f1f700-6db48b9ff4-chndk"
Oct 30 07:33:50.854002 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:50.853975    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1825dd2c-f162-4d90-8ebd-a6923e990abd-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-f1f700-6db48b9ff4-chndk\" (UID: \"1825dd2c-f162-4d90-8ebd-a6923e990abd\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-f1f700-6db48b9ff4-chndk"
Oct 30 07:33:50.880285 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:50.880015    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpvs7\" (UniqueName: \"kubernetes.io/projected/1825dd2c-f162-4d90-8ebd-a6923e990abd-kube-api-access-gpvs7\") pod \"payload-processing-e2e-extmod-b-f1f700-6db48b9ff4-chndk\" (UID: \"1825dd2c-f162-4d90-8ebd-a6923e990abd\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-f1f700-6db48b9ff4-chndk"
Oct 30 07:33:51.037374 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:51.036515    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-f1f700-6db48b9ff4-chndk"
Oct 30 07:33:51.108686 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:51.108644    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-f1f700-cccdc9f5-wn4nt"]
Oct 30 07:33:51.379422 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:33:51.379394    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1825dd2c_f162_4d90_8ebd_a6923e990abd.slice/crio-bb0d12de5e286e32e80a5fc564567f997c3f0eed00673a2ce2e54145fbcaf841 WatchSource:0}: Error finding container bb0d12de5e286e32e80a5fc564567f997c3f0eed00673a2ce2e54145fbcaf841: Status 404 returned error can't find the container with id bb0d12de5e286e32e80a5fc564567f997c3f0eed00673a2ce2e54145fbcaf841
Oct 30 07:33:51.380661 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:51.380637    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-f1f700-6db48b9ff4-chndk"]
Oct 30 07:33:52.073778 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:52.073741    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-f1f700-6db48b9ff4-chndk" event={"ID":"1825dd2c-f162-4d90-8ebd-a6923e990abd","Type":"ContainerStarted","Data":"d3ebd0183d13705987a8aea6c939610fd9a359c1a85393f2dd0e50b5044a5f2e"}
Oct 30 07:33:52.074214 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:52.073801    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-f1f700-6db48b9ff4-chndk" event={"ID":"1825dd2c-f162-4d90-8ebd-a6923e990abd","Type":"ContainerStarted","Data":"bb0d12de5e286e32e80a5fc564567f997c3f0eed00673a2ce2e54145fbcaf841"}
Oct 30 07:33:52.074214 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:52.073866    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-f1f700-6db48b9ff4-chndk"
Oct 30 07:33:52.075484 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:52.075461    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f1f700-cccdc9f5-wn4nt" event={"ID":"0d55b266-9ae5-4b44-85e3-ea14553e4e27","Type":"ContainerStarted","Data":"40c1243ac13cc5bd03a5727bed1733e95368ba674d9c8b98b1c4041449f77b3f"}
Oct 30 07:33:52.075586 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:52.075504    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f1f700-cccdc9f5-wn4nt" event={"ID":"0d55b266-9ae5-4b44-85e3-ea14553e4e27","Type":"ContainerStarted","Data":"c6efc81f1f2bbf26200a3d715a5f7e509eea535f87447b035f0c31f9e192a7f1"}
Oct 30 07:33:52.075586 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:52.075523    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f1f700-cccdc9f5-wn4nt"
Oct 30 07:33:52.121742 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:52.121698    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-f1f700-6db48b9ff4-chndk" podStartSLOduration=2.121685556 podStartE2EDuration="2.121685556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:33:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:33:52.118175019 +0000 UTC m=+2190.727581989" watchObservedRunningTime="2026-10-30 07:33:52.121685556 +0000 UTC m=+2190.731092548"
Oct 30 07:33:52.164286 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:52.164244    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f1f700-cccdc9f5-wn4nt" podStartSLOduration=2.164232708 podStartE2EDuration="2.164232708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:33:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:33:52.162493749 +0000 UTC m=+2190.771900722" watchObservedRunningTime="2026-10-30 07:33:52.164232708 +0000 UTC m=+2190.773639678"
Oct 30 07:33:57.056196 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:33:57.056153    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:34:00.319920 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:00.319882    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-25f44e/noten-model-25f44e-kserve-544947d8f7-fw8hb" podUID="d0d444b9-e96c-48f0-9cd2-09610027cdc9" containerName="main" containerID="cri-o://ec8d686853e9cd6e47d0e0ad677df45be40ec1edc31ca21f9a6453cb22d505bb" gracePeriod=45
Oct 30 07:34:00.644075 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:00.644047    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-25f44e/noten-model-25f44e-kserve-544947d8f7-fw8hb"
Oct 30 07:34:00.724856 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:00.724819    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0d444b9-e96c-48f0-9cd2-09610027cdc9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0d444b9-e96c-48f0-9cd2-09610027cdc9-model-cache\") pod \"d0d444b9-e96c-48f0-9cd2-09610027cdc9\" (UID: \"d0d444b9-e96c-48f0-9cd2-09610027cdc9\") "
Oct 30 07:34:00.725021 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:00.724898    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0d444b9-e96c-48f0-9cd2-09610027cdc9-home\" (UniqueName: \"kubernetes.io/empty-dir/d0d444b9-e96c-48f0-9cd2-09610027cdc9-home\") pod \"d0d444b9-e96c-48f0-9cd2-09610027cdc9\" (UID: \"d0d444b9-e96c-48f0-9cd2-09610027cdc9\") "
Oct 30 07:34:00.725021 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:00.724937    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0d444b9-e96c-48f0-9cd2-09610027cdc9-tls-certs\" (UniqueName: \"kubernetes.io/secret/d0d444b9-e96c-48f0-9cd2-09610027cdc9-tls-certs\") pod \"d0d444b9-e96c-48f0-9cd2-09610027cdc9\" (UID: \"d0d444b9-e96c-48f0-9cd2-09610027cdc9\") "
Oct 30 07:34:00.725021 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:00.724965    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0d444b9-e96c-48f0-9cd2-09610027cdc9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0d444b9-e96c-48f0-9cd2-09610027cdc9-tmp-dir\") pod \"d0d444b9-e96c-48f0-9cd2-09610027cdc9\" (UID: \"d0d444b9-e96c-48f0-9cd2-09610027cdc9\") "
Oct 30 07:34:00.725158 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:00.725069    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0d444b9-e96c-48f0-9cd2-09610027cdc9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0d444b9-e96c-48f0-9cd2-09610027cdc9-dshm\") pod \"d0d444b9-e96c-48f0-9cd2-09610027cdc9\" (UID: \"d0d444b9-e96c-48f0-9cd2-09610027cdc9\") "
Oct 30 07:34:00.725158 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:00.725105    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0d444b9-e96c-48f0-9cd2-09610027cdc9-kube-api-access-mxhwn\" (UniqueName: \"kubernetes.io/projected/d0d444b9-e96c-48f0-9cd2-09610027cdc9-kube-api-access-mxhwn\") pod \"d0d444b9-e96c-48f0-9cd2-09610027cdc9\" (UID: \"d0d444b9-e96c-48f0-9cd2-09610027cdc9\") "
Oct 30 07:34:00.726160 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:00.726068    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0d444b9-e96c-48f0-9cd2-09610027cdc9-model-cache" pod "d0d444b9-e96c-48f0-9cd2-09610027cdc9" (UID: "d0d444b9-e96c-48f0-9cd2-09610027cdc9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:34:00.726160 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:00.726081    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0d444b9-e96c-48f0-9cd2-09610027cdc9-tmp-dir" pod "d0d444b9-e96c-48f0-9cd2-09610027cdc9" (UID: "d0d444b9-e96c-48f0-9cd2-09610027cdc9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:34:00.726338 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:00.726160    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0d444b9-e96c-48f0-9cd2-09610027cdc9-home" pod "d0d444b9-e96c-48f0-9cd2-09610027cdc9" (UID: "d0d444b9-e96c-48f0-9cd2-09610027cdc9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:34:00.729347 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:00.729318    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0d444b9-e96c-48f0-9cd2-09610027cdc9-dshm" pod "d0d444b9-e96c-48f0-9cd2-09610027cdc9" (UID: "d0d444b9-e96c-48f0-9cd2-09610027cdc9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:34:00.729621 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:00.729603    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0d444b9-e96c-48f0-9cd2-09610027cdc9-kube-api-access-mxhwn" pod "d0d444b9-e96c-48f0-9cd2-09610027cdc9" (UID: "d0d444b9-e96c-48f0-9cd2-09610027cdc9"). InnerVolumeSpecName "kube-api-access-mxhwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:34:00.730918 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:00.730891    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0d444b9-e96c-48f0-9cd2-09610027cdc9-tls-certs" pod "d0d444b9-e96c-48f0-9cd2-09610027cdc9" (UID: "d0d444b9-e96c-48f0-9cd2-09610027cdc9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:34:00.826516 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:00.826478    6224 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0d444b9-e96c-48f0-9cd2-09610027cdc9-model-cache\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:34:00.826516 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:00.826507    6224 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0d444b9-e96c-48f0-9cd2-09610027cdc9-home\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:34:00.826516 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:00.826517    6224 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0d444b9-e96c-48f0-9cd2-09610027cdc9-tls-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:34:00.826516 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:00.826525    6224 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0d444b9-e96c-48f0-9cd2-09610027cdc9-tmp-dir\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:34:00.826801 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:00.826533    6224 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0d444b9-e96c-48f0-9cd2-09610027cdc9-dshm\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:34:00.826801 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:00.826541    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxhwn\" (UniqueName: \"kubernetes.io/projected/d0d444b9-e96c-48f0-9cd2-09610027cdc9-kube-api-access-mxhwn\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:34:01.191241 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:01.190871    6224 generic.go:356] "Generic (PLEG): container finished" podID="d0d444b9-e96c-48f0-9cd2-09610027cdc9" containerID="ec8d686853e9cd6e47d0e0ad677df45be40ec1edc31ca21f9a6453cb22d505bb" exitCode=0
Oct 30 07:34:01.191241 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:01.190989    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-25f44e/noten-model-25f44e-kserve-544947d8f7-fw8hb" event={"ID":"d0d444b9-e96c-48f0-9cd2-09610027cdc9","Type":"ContainerDied","Data":"ec8d686853e9cd6e47d0e0ad677df45be40ec1edc31ca21f9a6453cb22d505bb"}
Oct 30 07:34:01.191241 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:01.191011    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-25f44e/noten-model-25f44e-kserve-544947d8f7-fw8hb"
Oct 30 07:34:01.191241 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:01.191028    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-25f44e/noten-model-25f44e-kserve-544947d8f7-fw8hb" event={"ID":"d0d444b9-e96c-48f0-9cd2-09610027cdc9","Type":"ContainerDied","Data":"10f739c0e36b2c276037b474880d9fad8548c0fda98925ea5ac95bda60d36684"}
Oct 30 07:34:01.191241 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:01.191053    6224 scope.go:122] "RemoveContainer" containerID="ec8d686853e9cd6e47d0e0ad677df45be40ec1edc31ca21f9a6453cb22d505bb"
Oct 30 07:34:01.204658 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:01.204639    6224 scope.go:122] "RemoveContainer" containerID="ec8d686853e9cd6e47d0e0ad677df45be40ec1edc31ca21f9a6453cb22d505bb"
Oct 30 07:34:01.204971 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:34:01.204950    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec8d686853e9cd6e47d0e0ad677df45be40ec1edc31ca21f9a6453cb22d505bb\": container with ID starting with ec8d686853e9cd6e47d0e0ad677df45be40ec1edc31ca21f9a6453cb22d505bb not found: ID does not exist" containerID="ec8d686853e9cd6e47d0e0ad677df45be40ec1edc31ca21f9a6453cb22d505bb"
Oct 30 07:34:01.205024 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:01.204981    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec8d686853e9cd6e47d0e0ad677df45be40ec1edc31ca21f9a6453cb22d505bb"} err="failed to get container status \"ec8d686853e9cd6e47d0e0ad677df45be40ec1edc31ca21f9a6453cb22d505bb\": rpc error: code = NotFound desc = could not find container \"ec8d686853e9cd6e47d0e0ad677df45be40ec1edc31ca21f9a6453cb22d505bb\": container with ID starting with ec8d686853e9cd6e47d0e0ad677df45be40ec1edc31ca21f9a6453cb22d505bb not found: ID does not exist"
Oct 30 07:34:01.253049 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:01.253016    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-25f44e/noten-model-25f44e-kserve-544947d8f7-fw8hb"]
Oct 30 07:34:01.260110 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:01.260085    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-25f44e/noten-model-25f44e-kserve-544947d8f7-fw8hb"]
Oct 30 07:34:01.507770 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:01.507691    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0d444b9-e96c-48f0-9cd2-09610027cdc9" path="/var/lib/kubelet/pods/d0d444b9-e96c-48f0-9cd2-09610027cdc9/volumes"
Oct 30 07:34:01.858002 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:34:01.857970    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:34:03.088905 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:03.088583    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f1f700-cccdc9f5-wn4nt"
Oct 30 07:34:03.090276 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:03.090249    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-f1f700-6db48b9ff4-chndk"
Oct 30 07:34:06.541207 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:06.541174    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-1f34aa-5c4b4fdd4f-jwkrk"]
Oct 30 07:34:06.541789 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:06.541774    6224 cpu_manager.go:414] "containerMap: removing container" podUID="d0d444b9-e96c-48f0-9cd2-09610027cdc9" containerName="main"
Oct 30 07:34:06.541855 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:06.541790    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0d444b9-e96c-48f0-9cd2-09610027cdc9" containerName="main"
Oct 30 07:34:06.541917 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:06.541908    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0d444b9-e96c-48f0-9cd2-09610027cdc9" containerName="main"
Oct 30 07:34:06.547245 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:06.547220    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1f34aa-5c4b4fdd4f-jwkrk"
Oct 30 07:34:06.567216 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:06.567191    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-1f34aa-5c4b4fdd4f-jwkrk"]
Oct 30 07:34:06.576762 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:06.576736    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-1f34aa-serving-cert\""
Oct 30 07:34:06.615967 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:06.615927    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc9e4c3b-9885-4706-817b-4da954d01dfb-maas-api-tls\") pod \"maas-api-e2e-extmod-b-1f34aa-5c4b4fdd4f-jwkrk\" (UID: \"bc9e4c3b-9885-4706-817b-4da954d01dfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1f34aa-5c4b4fdd4f-jwkrk"
Oct 30 07:34:06.616145 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:06.616032    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc9e4c3b-9885-4706-817b-4da954d01dfb-metrics-certs\") pod \"maas-api-e2e-extmod-b-1f34aa-5c4b4fdd4f-jwkrk\" (UID: \"bc9e4c3b-9885-4706-817b-4da954d01dfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1f34aa-5c4b4fdd4f-jwkrk"
Oct 30 07:34:06.616145 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:06.616123    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc9e4c3b-9885-4706-817b-4da954d01dfb-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-1f34aa-5c4b4fdd4f-jwkrk\" (UID: \"bc9e4c3b-9885-4706-817b-4da954d01dfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1f34aa-5c4b4fdd4f-jwkrk"
Oct 30 07:34:06.618294 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:06.618263    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gh5q\" (UniqueName: \"kubernetes.io/projected/bc9e4c3b-9885-4706-817b-4da954d01dfb-kube-api-access-2gh5q\") pod \"maas-api-e2e-extmod-b-1f34aa-5c4b4fdd4f-jwkrk\" (UID: \"bc9e4c3b-9885-4706-817b-4da954d01dfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1f34aa-5c4b4fdd4f-jwkrk"
Oct 30 07:34:06.720004 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:06.719968    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc9e4c3b-9885-4706-817b-4da954d01dfb-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-1f34aa-5c4b4fdd4f-jwkrk\" (UID: \"bc9e4c3b-9885-4706-817b-4da954d01dfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1f34aa-5c4b4fdd4f-jwkrk"
Oct 30 07:34:06.720186 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:06.720023    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gh5q\" (UniqueName: \"kubernetes.io/projected/bc9e4c3b-9885-4706-817b-4da954d01dfb-kube-api-access-2gh5q\") pod \"maas-api-e2e-extmod-b-1f34aa-5c4b4fdd4f-jwkrk\" (UID: \"bc9e4c3b-9885-4706-817b-4da954d01dfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1f34aa-5c4b4fdd4f-jwkrk"
Oct 30 07:34:06.720186 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:06.720179    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc9e4c3b-9885-4706-817b-4da954d01dfb-maas-api-tls\") pod \"maas-api-e2e-extmod-b-1f34aa-5c4b4fdd4f-jwkrk\" (UID: \"bc9e4c3b-9885-4706-817b-4da954d01dfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1f34aa-5c4b4fdd4f-jwkrk"
Oct 30 07:34:06.720361 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:06.720265    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc9e4c3b-9885-4706-817b-4da954d01dfb-metrics-certs\") pod \"maas-api-e2e-extmod-b-1f34aa-5c4b4fdd4f-jwkrk\" (UID: \"bc9e4c3b-9885-4706-817b-4da954d01dfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1f34aa-5c4b4fdd4f-jwkrk"
Oct 30 07:34:06.720723 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:06.720690    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc9e4c3b-9885-4706-817b-4da954d01dfb-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-1f34aa-5c4b4fdd4f-jwkrk\" (UID: \"bc9e4c3b-9885-4706-817b-4da954d01dfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1f34aa-5c4b4fdd4f-jwkrk"
Oct 30 07:34:06.722723 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:06.722700    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc9e4c3b-9885-4706-817b-4da954d01dfb-metrics-certs\") pod \"maas-api-e2e-extmod-b-1f34aa-5c4b4fdd4f-jwkrk\" (UID: \"bc9e4c3b-9885-4706-817b-4da954d01dfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1f34aa-5c4b4fdd4f-jwkrk"
Oct 30 07:34:06.722826 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:06.722764    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc9e4c3b-9885-4706-817b-4da954d01dfb-maas-api-tls\") pod \"maas-api-e2e-extmod-b-1f34aa-5c4b4fdd4f-jwkrk\" (UID: \"bc9e4c3b-9885-4706-817b-4da954d01dfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1f34aa-5c4b4fdd4f-jwkrk"
Oct 30 07:34:06.745010 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:06.744987    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gh5q\" (UniqueName: \"kubernetes.io/projected/bc9e4c3b-9885-4706-817b-4da954d01dfb-kube-api-access-2gh5q\") pod \"maas-api-e2e-extmod-b-1f34aa-5c4b4fdd4f-jwkrk\" (UID: \"bc9e4c3b-9885-4706-817b-4da954d01dfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1f34aa-5c4b4fdd4f-jwkrk"
Oct 30 07:34:06.866219 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:06.866130    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1f34aa-5c4b4fdd4f-jwkrk"
Oct 30 07:34:07.177612 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:07.177585    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-1f34aa-5c4b4fdd4f-jwkrk"]
Oct 30 07:34:07.178071 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:34:07.178048    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc9e4c3b_9885_4706_817b_4da954d01dfb.slice/crio-7db87f50411d0136e2d95c008c663cafb7f9273056bc5b8c774c921544a27c6a WatchSource:0}: Error finding container 7db87f50411d0136e2d95c008c663cafb7f9273056bc5b8c774c921544a27c6a: Status 404 returned error can't find the container with id 7db87f50411d0136e2d95c008c663cafb7f9273056bc5b8c774c921544a27c6a
Oct 30 07:34:07.238288 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:07.238245    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1f34aa-5c4b4fdd4f-jwkrk" event={"ID":"bc9e4c3b-9885-4706-817b-4da954d01dfb","Type":"ContainerStarted","Data":"7db87f50411d0136e2d95c008c663cafb7f9273056bc5b8c774c921544a27c6a"}
Oct 30 07:34:08.246919 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:08.246879    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1f34aa-5c4b4fdd4f-jwkrk" event={"ID":"bc9e4c3b-9885-4706-817b-4da954d01dfb","Type":"ContainerStarted","Data":"96f4170e881ace5f96838eee8522340a6e7f8ecd851c9cea3a201e47e5fe0742"}
Oct 30 07:34:08.247436 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:08.246952    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1f34aa-5c4b4fdd4f-jwkrk"
Oct 30 07:34:08.248569 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:08.248544    6224 generic.go:356] "Generic (PLEG): container finished" podID="89326bce-8281-4747-aa3f-c1d37c615234" containerID="f83e71bb373bc44313b4a523c72c94150cc2ef2427a9d57a48c1de416ed47e29" exitCode=0
Oct 30 07:34:08.248682 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:08.248616    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf" event={"ID":"89326bce-8281-4747-aa3f-c1d37c615234","Type":"ContainerDied","Data":"f83e71bb373bc44313b4a523c72c94150cc2ef2427a9d57a48c1de416ed47e29"}
Oct 30 07:34:08.248968 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:08.248954    6224 scope.go:122] "RemoveContainer" containerID="f83e71bb373bc44313b4a523c72c94150cc2ef2427a9d57a48c1de416ed47e29"
Oct 30 07:34:08.250473 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:08.250448    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:34:08.250537 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:08.250514    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:34:08.250572 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:08.250546    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:34:08.296928 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:08.296886    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1f34aa-5c4b4fdd4f-jwkrk" podStartSLOduration=1.994061079 podStartE2EDuration="2.296872948s" totalImagesPullingTime="302.811869ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:34:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:34:08.293875075 +0000 UTC m=+2206.903282126" watchObservedRunningTime="2026-10-30 07:34:08.296872948 +0000 UTC m=+2206.906279917"
Oct 30 07:34:09.257042 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:09.257000    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf" event={"ID":"89326bce-8281-4747-aa3f-c1d37c615234","Type":"ContainerStarted","Data":"72cd4cbefbae77bd6a6bdd3ce4ee12ffabdc8d855eaebfa80a3ba7bca32fd5a7"}
Oct 30 07:34:09.651325 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:09.651297    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf"
Oct 30 07:34:09.652692 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:09.652660    6224 patch_prober.go:29] interesting pod/e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Oct 30 07:34:09.652842 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:09.652719    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf" podUID="89326bce-8281-4747-aa3f-c1d37c615234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Oct 30 07:34:10.650981 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:10.650940    6224 patch_prober.go:29] interesting pod/e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Oct 30 07:34:10.651462 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:10.651001    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf" podUID="89326bce-8281-4747-aa3f-c1d37c615234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Oct 30 07:34:11.443872 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:11.443835    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf"
Oct 30 07:34:11.651059 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:11.651019    6224 patch_prober.go:29] interesting pod/e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Oct 30 07:34:11.651545 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:11.651089    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf" podUID="89326bce-8281-4747-aa3f-c1d37c615234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Oct 30 07:34:12.651299 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:12.651264    6224 patch_prober.go:29] interesting pod/e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Oct 30 07:34:12.651785 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:12.651327    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf" podUID="89326bce-8281-4747-aa3f-c1d37c615234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Oct 30 07:34:13.651275 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:13.651240    6224 patch_prober.go:29] interesting pod/e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Oct 30 07:34:13.651451 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:13.651307    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf" podUID="89326bce-8281-4747-aa3f-c1d37c615234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Oct 30 07:34:14.263013 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:14.262981    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1f34aa-5c4b4fdd4f-jwkrk"
Oct 30 07:34:14.655361 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:14.655332    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf"
Oct 30 07:34:14.656888 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:14.656861    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf"
Oct 30 07:34:16.869468 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:16.869436    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c1b5d7-677bbc4b95-vlwb7"]
Oct 30 07:34:16.873679 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:16.873660    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c1b5d7-677bbc4b95-vlwb7"
Oct 30 07:34:16.890549 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:16.890521    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-c1b5d7-dockercfg-v5vsp\""
Oct 30 07:34:16.894768 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:16.894738    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c1b5d7-677bbc4b95-vlwb7"]
Oct 30 07:34:16.904364 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:16.904333    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-c1b5d7\""
Oct 30 07:34:17.008830 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:17.005566    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6knn\" (UniqueName: \"kubernetes.io/projected/ff56f991-7e93-4d06-8d87-c42cb72465e3-kube-api-access-t6knn\") pod \"payload-pre-processing-e2e-extmod-b-c1b5d7-677bbc4b95-vlwb7\" (UID: \"ff56f991-7e93-4d06-8d87-c42cb72465e3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c1b5d7-677bbc4b95-vlwb7"
Oct 30 07:34:17.008830 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:17.005765    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff56f991-7e93-4d06-8d87-c42cb72465e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-c1b5d7-677bbc4b95-vlwb7\" (UID: \"ff56f991-7e93-4d06-8d87-c42cb72465e3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c1b5d7-677bbc4b95-vlwb7"
Oct 30 07:34:17.029229 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:17.029191    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c1b5d7-6cb579c4-9h8vp"]
Oct 30 07:34:17.038830 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:17.037959    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-c1b5d7-6cb579c4-9h8vp"
Oct 30 07:34:17.077838 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:17.074789    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c1b5d7-6cb579c4-9h8vp"]
Oct 30 07:34:17.111605 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:17.111568    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1313ec66-e8a0-4845-9181-cf9fe5b2d8c9-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-c1b5d7-6cb579c4-9h8vp\" (UID: \"1313ec66-e8a0-4845-9181-cf9fe5b2d8c9\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c1b5d7-6cb579c4-9h8vp"
Oct 30 07:34:17.111849 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:17.111640    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff56f991-7e93-4d06-8d87-c42cb72465e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-c1b5d7-677bbc4b95-vlwb7\" (UID: \"ff56f991-7e93-4d06-8d87-c42cb72465e3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c1b5d7-677bbc4b95-vlwb7"
Oct 30 07:34:17.111849 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:17.111728    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6knn\" (UniqueName: \"kubernetes.io/projected/ff56f991-7e93-4d06-8d87-c42cb72465e3-kube-api-access-t6knn\") pod \"payload-pre-processing-e2e-extmod-b-c1b5d7-677bbc4b95-vlwb7\" (UID: \"ff56f991-7e93-4d06-8d87-c42cb72465e3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c1b5d7-677bbc4b95-vlwb7"
Oct 30 07:34:17.111849 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:17.111762    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g764\" (UniqueName: \"kubernetes.io/projected/1313ec66-e8a0-4845-9181-cf9fe5b2d8c9-kube-api-access-6g764\") pod \"payload-processing-e2e-extmod-b-c1b5d7-6cb579c4-9h8vp\" (UID: \"1313ec66-e8a0-4845-9181-cf9fe5b2d8c9\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c1b5d7-6cb579c4-9h8vp"
Oct 30 07:34:17.112629 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:17.112605    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff56f991-7e93-4d06-8d87-c42cb72465e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-c1b5d7-677bbc4b95-vlwb7\" (UID: \"ff56f991-7e93-4d06-8d87-c42cb72465e3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c1b5d7-677bbc4b95-vlwb7"
Oct 30 07:34:17.138002 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:17.137928    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6knn\" (UniqueName: \"kubernetes.io/projected/ff56f991-7e93-4d06-8d87-c42cb72465e3-kube-api-access-t6knn\") pod \"payload-pre-processing-e2e-extmod-b-c1b5d7-677bbc4b95-vlwb7\" (UID: \"ff56f991-7e93-4d06-8d87-c42cb72465e3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c1b5d7-677bbc4b95-vlwb7"
Oct 30 07:34:17.187795 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:17.187765    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c1b5d7-677bbc4b95-vlwb7"
Oct 30 07:34:17.213159 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:17.213128    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g764\" (UniqueName: \"kubernetes.io/projected/1313ec66-e8a0-4845-9181-cf9fe5b2d8c9-kube-api-access-6g764\") pod \"payload-processing-e2e-extmod-b-c1b5d7-6cb579c4-9h8vp\" (UID: \"1313ec66-e8a0-4845-9181-cf9fe5b2d8c9\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c1b5d7-6cb579c4-9h8vp"
Oct 30 07:34:17.213285 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:17.213232    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1313ec66-e8a0-4845-9181-cf9fe5b2d8c9-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-c1b5d7-6cb579c4-9h8vp\" (UID: \"1313ec66-e8a0-4845-9181-cf9fe5b2d8c9\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c1b5d7-6cb579c4-9h8vp"
Oct 30 07:34:17.213992 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:17.213971    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1313ec66-e8a0-4845-9181-cf9fe5b2d8c9-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-c1b5d7-6cb579c4-9h8vp\" (UID: \"1313ec66-e8a0-4845-9181-cf9fe5b2d8c9\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c1b5d7-6cb579c4-9h8vp"
Oct 30 07:34:17.238372 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:17.238348    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g764\" (UniqueName: \"kubernetes.io/projected/1313ec66-e8a0-4845-9181-cf9fe5b2d8c9-kube-api-access-6g764\") pod \"payload-processing-e2e-extmod-b-c1b5d7-6cb579c4-9h8vp\" (UID: \"1313ec66-e8a0-4845-9181-cf9fe5b2d8c9\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c1b5d7-6cb579c4-9h8vp"
Oct 30 07:34:17.354506 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:17.354474    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-c1b5d7-6cb579c4-9h8vp"
Oct 30 07:34:17.496083 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:17.496047    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c1b5d7-677bbc4b95-vlwb7"]
Oct 30 07:34:17.496983 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:34:17.496957    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff56f991_7e93_4d06_8d87_c42cb72465e3.slice/crio-e71089215cce4a8195082c169fedca3d453ad22f732c7251f47100e4999c0d2a WatchSource:0}: Error finding container e71089215cce4a8195082c169fedca3d453ad22f732c7251f47100e4999c0d2a: Status 404 returned error can't find the container with id e71089215cce4a8195082c169fedca3d453ad22f732c7251f47100e4999c0d2a
Oct 30 07:34:17.654597 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:34:17.654566    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1313ec66_e8a0_4845_9181_cf9fe5b2d8c9.slice/crio-4aef7ebf3a3cca83dab2158ce4df236aa752a1c7dc3c83712e9c2d63556e609b WatchSource:0}: Error finding container 4aef7ebf3a3cca83dab2158ce4df236aa752a1c7dc3c83712e9c2d63556e609b: Status 404 returned error can't find the container with id 4aef7ebf3a3cca83dab2158ce4df236aa752a1c7dc3c83712e9c2d63556e609b
Oct 30 07:34:17.656654 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:17.656627    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c1b5d7-6cb579c4-9h8vp"]
Oct 30 07:34:18.362187 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:18.362153    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-c1b5d7-6cb579c4-9h8vp" event={"ID":"1313ec66-e8a0-4845-9181-cf9fe5b2d8c9","Type":"ContainerStarted","Data":"4aaf9f666652e5731d15fa960fdc01b22c7ecea4c6a2f18aac1ee473d8690d1a"}
Oct 30 07:34:18.362647 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:18.362206    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-c1b5d7-6cb579c4-9h8vp" event={"ID":"1313ec66-e8a0-4845-9181-cf9fe5b2d8c9","Type":"ContainerStarted","Data":"4aef7ebf3a3cca83dab2158ce4df236aa752a1c7dc3c83712e9c2d63556e609b"}
Oct 30 07:34:18.362647 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:18.362240    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-c1b5d7-6cb579c4-9h8vp"
Oct 30 07:34:18.363975 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:18.363946    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c1b5d7-677bbc4b95-vlwb7" event={"ID":"ff56f991-7e93-4d06-8d87-c42cb72465e3","Type":"ContainerStarted","Data":"f77e0f6b2929f081d3d174b5dbb4ba39ee8f761c12c8aaf16bc41d13541891d5"}
Oct 30 07:34:18.364103 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:18.364002    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c1b5d7-677bbc4b95-vlwb7" event={"ID":"ff56f991-7e93-4d06-8d87-c42cb72465e3","Type":"ContainerStarted","Data":"e71089215cce4a8195082c169fedca3d453ad22f732c7251f47100e4999c0d2a"}
Oct 30 07:34:18.364103 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:18.364033    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c1b5d7-677bbc4b95-vlwb7"
Oct 30 07:34:18.408241 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:18.408187    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-c1b5d7-6cb579c4-9h8vp" podStartSLOduration=2.40817473 podStartE2EDuration="2.40817473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:34:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:34:18.405609586 +0000 UTC m=+2217.015016559" watchObservedRunningTime="2026-10-30 07:34:18.40817473 +0000 UTC m=+2217.017581694"
Oct 30 07:34:18.451181 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:18.451135    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c1b5d7-677bbc4b95-vlwb7" podStartSLOduration=2.451123519 podStartE2EDuration="2.451123519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:34:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:34:18.44946967 +0000 UTC m=+2217.058876641" watchObservedRunningTime="2026-10-30 07:34:18.451123519 +0000 UTC m=+2217.060530506"
Oct 30 07:34:29.373666 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:29.373635    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-c1b5d7-6cb579c4-9h8vp"
Oct 30 07:34:29.374275 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:29.373708    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c1b5d7-677bbc4b95-vlwb7"
Oct 30 07:34:31.861067 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:34:31.861019    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:34:36.066174 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:36.066136    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-8d0c2f-df895bbbb-gczsx"]
Oct 30 07:34:36.072465 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:36.072441    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-8d0c2f-df895bbbb-gczsx"
Oct 30 07:34:36.089503 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:36.089479    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-8d0c2f-df895bbbb-gczsx"]
Oct 30 07:34:36.099385 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:36.099360    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-8d0c2f\""
Oct 30 07:34:36.099529 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:36.099408    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-8d0c2f-dockercfg-rjszc\""
Oct 30 07:34:36.170289 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:36.170252    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/845c4712-065d-4f17-87c6-514dc1b5c1a8-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-8d0c2f-df895bbbb-gczsx\" (UID: \"845c4712-065d-4f17-87c6-514dc1b5c1a8\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-8d0c2f-df895bbbb-gczsx"
Oct 30 07:34:36.170484 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:36.170340    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn8ml\" (UniqueName: \"kubernetes.io/projected/845c4712-065d-4f17-87c6-514dc1b5c1a8-kube-api-access-wn8ml\") pod \"payload-processing-e2e-extmod-a-8d0c2f-df895bbbb-gczsx\" (UID: \"845c4712-065d-4f17-87c6-514dc1b5c1a8\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-8d0c2f-df895bbbb-gczsx"
Oct 30 07:34:36.276702 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:36.276659    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/845c4712-065d-4f17-87c6-514dc1b5c1a8-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-8d0c2f-df895bbbb-gczsx\" (UID: \"845c4712-065d-4f17-87c6-514dc1b5c1a8\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-8d0c2f-df895bbbb-gczsx"
Oct 30 07:34:36.276929 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:36.276736    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn8ml\" (UniqueName: \"kubernetes.io/projected/845c4712-065d-4f17-87c6-514dc1b5c1a8-kube-api-access-wn8ml\") pod \"payload-processing-e2e-extmod-a-8d0c2f-df895bbbb-gczsx\" (UID: \"845c4712-065d-4f17-87c6-514dc1b5c1a8\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-8d0c2f-df895bbbb-gczsx"
Oct 30 07:34:36.278101 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:36.278074    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/845c4712-065d-4f17-87c6-514dc1b5c1a8-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-8d0c2f-df895bbbb-gczsx\" (UID: \"845c4712-065d-4f17-87c6-514dc1b5c1a8\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-8d0c2f-df895bbbb-gczsx"
Oct 30 07:34:36.331204 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:36.328619    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn8ml\" (UniqueName: \"kubernetes.io/projected/845c4712-065d-4f17-87c6-514dc1b5c1a8-kube-api-access-wn8ml\") pod \"payload-processing-e2e-extmod-a-8d0c2f-df895bbbb-gczsx\" (UID: \"845c4712-065d-4f17-87c6-514dc1b5c1a8\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-8d0c2f-df895bbbb-gczsx"
Oct 30 07:34:36.386104 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:36.386073    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-8d0c2f-df895bbbb-gczsx"
Oct 30 07:34:36.689785 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:34:36.689752    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod845c4712_065d_4f17_87c6_514dc1b5c1a8.slice/crio-b603b070c4de2d9da68f76afadc1375f5fa0f8a6162abfbe291ac0e4f048d6e4 WatchSource:0}: Error finding container b603b070c4de2d9da68f76afadc1375f5fa0f8a6162abfbe291ac0e4f048d6e4: Status 404 returned error can't find the container with id b603b070c4de2d9da68f76afadc1375f5fa0f8a6162abfbe291ac0e4f048d6e4
Oct 30 07:34:36.690369 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:36.690346    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-8d0c2f-df895bbbb-gczsx"]
Oct 30 07:34:37.517450 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:37.517410    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-8d0c2f-df895bbbb-gczsx" event={"ID":"845c4712-065d-4f17-87c6-514dc1b5c1a8","Type":"ContainerStarted","Data":"b4d33d94102ed4318b4c7faef30925178a454b65e17ec002086a10d2e5d4ad20"}
Oct 30 07:34:37.517928 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:37.517466    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-8d0c2f-df895bbbb-gczsx" event={"ID":"845c4712-065d-4f17-87c6-514dc1b5c1a8","Type":"ContainerStarted","Data":"b603b070c4de2d9da68f76afadc1375f5fa0f8a6162abfbe291ac0e4f048d6e4"}
Oct 30 07:34:37.517928 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:37.517492    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-8d0c2f-df895bbbb-gczsx"
Oct 30 07:34:37.564673 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:37.564619    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-8d0c2f-df895bbbb-gczsx" podStartSLOduration=1.564604976 podStartE2EDuration="1.564604976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:34:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:34:37.561199368 +0000 UTC m=+2236.170606339" watchObservedRunningTime="2026-10-30 07:34:37.564604976 +0000 UTC m=+2236.174011946"
Oct 30 07:34:44.396834 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:44.386910    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:34:48.529384 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:48.529356    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-8d0c2f-df895bbbb-gczsx"
Oct 30 07:34:54.234666 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:54.234625    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v"]
Oct 30 07:34:54.235236 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:54.234915    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v" podUID="ea11f788-17b6-4758-945d-fbb0d273e840" containerName="istio-proxy" containerID="cri-o://a4b50c1feb0e7a59388b908e7a1a4da3d70d33efe27b59b3191567d8655cca46" gracePeriod=30
Oct 30 07:34:55.042295 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:55.042264    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-ace8fb/payload-processing-external-model-e2e-extmod-b-ace8fb-798c98nqf"]
Oct 30 07:34:55.047702 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:55.047681    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ace8fb/payload-processing-external-model-e2e-extmod-b-ace8fb-798c98nqf"
Oct 30 07:34:55.066624 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:55.066600    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ace8fb/payload-processing-external-model-e2e-extmod-b-ace8fb-798c98nqf"]
Oct 30 07:34:55.076609 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:55.076580    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ace8fb\"/\"openshift-service-ca.crt\""
Oct 30 07:34:55.076891 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:55.076871    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ace8fb\"/\"kube-root-ca.crt\""
Oct 30 07:34:55.077101 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:55.077083    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ace8fb\"/\"routing-overlay\""
Oct 30 07:34:55.082442 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:55.082414    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ace8fb\"/\"payload-processing-external-model-plugins-e2e-extmod-b-ace8fb\""
Oct 30 07:34:55.082555 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:55.082462    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ace8fb\"/\"agc-cred-769f1be413bb\""
Oct 30 07:34:55.082555 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:55.082518    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ace8fb\"/\"payload-processing-external-model-e2e--1b4b6e0b-dockercfg-gjdq2\""
Oct 30 07:34:55.106322 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:55.106293    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9f8346da-1bd9-4102-ace6-69aa26b9b2f6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ace8fb-798c98nqf\" (UID: \"9f8346da-1bd9-4102-ace6-69aa26b9b2f6\") " pod="ai-tenant-e2e-extmod-b-ace8fb/payload-processing-external-model-e2e-extmod-b-ace8fb-798c98nqf"
Oct 30 07:34:55.106534 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:55.106513    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9kmw\" (UniqueName: \"kubernetes.io/projected/9f8346da-1bd9-4102-ace6-69aa26b9b2f6-kube-api-access-d9kmw\") pod \"payload-processing-external-model-e2e-extmod-b-ace8fb-798c98nqf\" (UID: \"9f8346da-1bd9-4102-ace6-69aa26b9b2f6\") " pod="ai-tenant-e2e-extmod-b-ace8fb/payload-processing-external-model-e2e-extmod-b-ace8fb-798c98nqf"
Oct 30 07:34:55.106596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:55.106552    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f8346da-1bd9-4102-ace6-69aa26b9b2f6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ace8fb-798c98nqf\" (UID: \"9f8346da-1bd9-4102-ace6-69aa26b9b2f6\") " pod="ai-tenant-e2e-extmod-b-ace8fb/payload-processing-external-model-e2e-extmod-b-ace8fb-798c98nqf"
Oct 30 07:34:55.106596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:55.106578    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9f8346da-1bd9-4102-ace6-69aa26b9b2f6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ace8fb-798c98nqf\" (UID: \"9f8346da-1bd9-4102-ace6-69aa26b9b2f6\") " pod="ai-tenant-e2e-extmod-b-ace8fb/payload-processing-external-model-e2e-extmod-b-ace8fb-798c98nqf"
Oct 30 07:34:55.229836 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:55.227782    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9f8346da-1bd9-4102-ace6-69aa26b9b2f6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ace8fb-798c98nqf\" (UID: \"9f8346da-1bd9-4102-ace6-69aa26b9b2f6\") " pod="ai-tenant-e2e-extmod-b-ace8fb/payload-processing-external-model-e2e-extmod-b-ace8fb-798c98nqf"
Oct 30 07:34:55.243896 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:55.241984    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9kmw\" (UniqueName: \"kubernetes.io/projected/9f8346da-1bd9-4102-ace6-69aa26b9b2f6-kube-api-access-d9kmw\") pod \"payload-processing-external-model-e2e-extmod-b-ace8fb-798c98nqf\" (UID: \"9f8346da-1bd9-4102-ace6-69aa26b9b2f6\") " pod="ai-tenant-e2e-extmod-b-ace8fb/payload-processing-external-model-e2e-extmod-b-ace8fb-798c98nqf"
Oct 30 07:34:55.243896 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:55.242038    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f8346da-1bd9-4102-ace6-69aa26b9b2f6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ace8fb-798c98nqf\" (UID: \"9f8346da-1bd9-4102-ace6-69aa26b9b2f6\") " pod="ai-tenant-e2e-extmod-b-ace8fb/payload-processing-external-model-e2e-extmod-b-ace8fb-798c98nqf"
Oct 30 07:34:55.243896 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:55.242246    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9f8346da-1bd9-4102-ace6-69aa26b9b2f6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ace8fb-798c98nqf\" (UID: \"9f8346da-1bd9-4102-ace6-69aa26b9b2f6\") " pod="ai-tenant-e2e-extmod-b-ace8fb/payload-processing-external-model-e2e-extmod-b-ace8fb-798c98nqf"
Oct 30 07:34:55.243896 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:55.243117    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9f8346da-1bd9-4102-ace6-69aa26b9b2f6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ace8fb-798c98nqf\" (UID: \"9f8346da-1bd9-4102-ace6-69aa26b9b2f6\") " pod="ai-tenant-e2e-extmod-b-ace8fb/payload-processing-external-model-e2e-extmod-b-ace8fb-798c98nqf"
Oct 30 07:34:55.243896 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:55.243513    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f8346da-1bd9-4102-ace6-69aa26b9b2f6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ace8fb-798c98nqf\" (UID: \"9f8346da-1bd9-4102-ace6-69aa26b9b2f6\") " pod="ai-tenant-e2e-extmod-b-ace8fb/payload-processing-external-model-e2e-extmod-b-ace8fb-798c98nqf"
Oct 30 07:34:55.268307 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:55.268242    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9f8346da-1bd9-4102-ace6-69aa26b9b2f6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ace8fb-798c98nqf\" (UID: \"9f8346da-1bd9-4102-ace6-69aa26b9b2f6\") " pod="ai-tenant-e2e-extmod-b-ace8fb/payload-processing-external-model-e2e-extmod-b-ace8fb-798c98nqf"
Oct 30 07:34:55.275705 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:55.275641    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9kmw\" (UniqueName: \"kubernetes.io/projected/9f8346da-1bd9-4102-ace6-69aa26b9b2f6-kube-api-access-d9kmw\") pod \"payload-processing-external-model-e2e-extmod-b-ace8fb-798c98nqf\" (UID: \"9f8346da-1bd9-4102-ace6-69aa26b9b2f6\") " pod="ai-tenant-e2e-extmod-b-ace8fb/payload-processing-external-model-e2e-extmod-b-ace8fb-798c98nqf"
Oct 30 07:34:55.362841 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:55.362741    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ace8fb/payload-processing-external-model-e2e-extmod-b-ace8fb-798c98nqf"
Oct 30 07:34:55.690524 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:34:55.690488    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f8346da_1bd9_4102_ace6_69aa26b9b2f6.slice/crio-94cf023f8a1aa96b496fb5f801b1f28f4906ba393efdd3297da53ea26a7f1be9 WatchSource:0}: Error finding container 94cf023f8a1aa96b496fb5f801b1f28f4906ba393efdd3297da53ea26a7f1be9: Status 404 returned error can't find the container with id 94cf023f8a1aa96b496fb5f801b1f28f4906ba393efdd3297da53ea26a7f1be9
Oct 30 07:34:55.691485 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:55.691446    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ace8fb/payload-processing-external-model-e2e-extmod-b-ace8fb-798c98nqf"]
Oct 30 07:34:56.675746 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:56.675703    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ace8fb/payload-processing-external-model-e2e-extmod-b-ace8fb-798c98nqf" event={"ID":"9f8346da-1bd9-4102-ace6-69aa26b9b2f6","Type":"ContainerStarted","Data":"1ea0e7190d71c8921f93c4a645f5018b79e175b358f42947c204387a7ea6fb7a"}
Oct 30 07:34:56.676265 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:56.675769    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ace8fb/payload-processing-external-model-e2e-extmod-b-ace8fb-798c98nqf" event={"ID":"9f8346da-1bd9-4102-ace6-69aa26b9b2f6","Type":"ContainerStarted","Data":"94cf023f8a1aa96b496fb5f801b1f28f4906ba393efdd3297da53ea26a7f1be9"}
Oct 30 07:34:56.676265 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:56.675918    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-ace8fb/payload-processing-external-model-e2e-extmod-b-ace8fb-798c98nqf"
Oct 30 07:34:56.677566 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:56.677544    6224 generic.go:356] "Generic (PLEG): container finished" podID="6f5bab32-7cdb-471b-9bd0-38b205ace2e3" containerID="cb68c675228dc2925720b0e9ca5511cb2a4d461c920b79a8684c7137f0385a91" exitCode=0
Oct 30 07:34:56.677682 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:56.677613    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh" event={"ID":"6f5bab32-7cdb-471b-9bd0-38b205ace2e3","Type":"ContainerDied","Data":"cb68c675228dc2925720b0e9ca5511cb2a4d461c920b79a8684c7137f0385a91"}
Oct 30 07:34:56.677956 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:56.677943    6224 scope.go:122] "RemoveContainer" containerID="cb68c675228dc2925720b0e9ca5511cb2a4d461c920b79a8684c7137f0385a91"
Oct 30 07:34:56.679498 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:56.679470    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:34:56.679588 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:56.679535    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:34:56.679588 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:56.679569    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:34:56.727629 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:56.726141    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-ace8fb/payload-processing-external-model-e2e-extmod-b-ace8fb-798c98nqf" podStartSLOduration=1.726117625 podStartE2EDuration="1.726117625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:34:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:34:56.718628911 +0000 UTC m=+2255.328035922" watchObservedRunningTime="2026-10-30 07:34:56.726117625 +0000 UTC m=+2255.335524595"
Oct 30 07:34:57.696268 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:57.696228    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh" event={"ID":"6f5bab32-7cdb-471b-9bd0-38b205ace2e3","Type":"ContainerStarted","Data":"1961f45606306292f536917d1a90ba1d51c7d2670ab4a44d2ad2baa82f946ba7"}
Oct 30 07:34:58.685832 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:58.683875    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh"
Oct 30 07:34:58.687929 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:58.687890    6224 patch_prober.go:29] interesting pod/e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Oct 30 07:34:58.688083 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:58.687952    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh" podUID="6f5bab32-7cdb-471b-9bd0-38b205ace2e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Oct 30 07:34:58.918743 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:58.918709    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x"]
Oct 30 07:34:58.919280 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:58.918927    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x" podUID="883946b4-c9d3-4c98-904f-9076e7597a1f" containerName="istio-proxy" containerID="cri-o://7bbee109bf27e28dc0874b73c27b9315d236b89c6ea8ccf25c0e0aa5e3f720c4" gracePeriod=30
Oct 30 07:34:59.614926 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:59.614714    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v"
Oct 30 07:34:59.683834 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:59.681904    6224 patch_prober.go:29] interesting pod/e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Oct 30 07:34:59.683834 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:59.681961    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh" podUID="6f5bab32-7cdb-471b-9bd0-38b205ace2e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Oct 30 07:34:59.722869 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:59.716916    6224 generic.go:356] "Generic (PLEG): container finished" podID="ea11f788-17b6-4758-945d-fbb0d273e840" containerID="a4b50c1feb0e7a59388b908e7a1a4da3d70d33efe27b59b3191567d8655cca46" exitCode=0
Oct 30 07:34:59.722869 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:59.717061    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v" event={"ID":"ea11f788-17b6-4758-945d-fbb0d273e840","Type":"ContainerDied","Data":"a4b50c1feb0e7a59388b908e7a1a4da3d70d33efe27b59b3191567d8655cca46"}
Oct 30 07:34:59.722869 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:59.717124    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v" event={"ID":"ea11f788-17b6-4758-945d-fbb0d273e840","Type":"ContainerDied","Data":"5a71bdb4be531fba6bbd5464f88337fb5b2e77bda7eac1ca08d1ff86c0104002"}
Oct 30 07:34:59.722869 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:59.717157    6224 scope.go:122] "RemoveContainer" containerID="a4b50c1feb0e7a59388b908e7a1a4da3d70d33efe27b59b3191567d8655cca46"
Oct 30 07:34:59.722869 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:59.717315    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v"
Oct 30 07:34:59.723676 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:59.723639    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea11f788-17b6-4758-945d-fbb0d273e840-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea11f788-17b6-4758-945d-fbb0d273e840-workload-socket\") pod \"ea11f788-17b6-4758-945d-fbb0d273e840\" (UID: \"ea11f788-17b6-4758-945d-fbb0d273e840\") "
Oct 30 07:34:59.723855 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:59.723709    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea11f788-17b6-4758-945d-fbb0d273e840-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea11f788-17b6-4758-945d-fbb0d273e840-workload-certs\") pod \"ea11f788-17b6-4758-945d-fbb0d273e840\" (UID: \"ea11f788-17b6-4758-945d-fbb0d273e840\") "
Oct 30 07:34:59.723855 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:59.723799    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea11f788-17b6-4758-945d-fbb0d273e840-kube-api-access-h4pqm\" (UniqueName: \"kubernetes.io/projected/ea11f788-17b6-4758-945d-fbb0d273e840-kube-api-access-h4pqm\") pod \"ea11f788-17b6-4758-945d-fbb0d273e840\" (UID: \"ea11f788-17b6-4758-945d-fbb0d273e840\") "
Oct 30 07:34:59.723987 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:59.723859    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea11f788-17b6-4758-945d-fbb0d273e840-istio-token\" (UniqueName: \"kubernetes.io/projected/ea11f788-17b6-4758-945d-fbb0d273e840-istio-token\") pod \"ea11f788-17b6-4758-945d-fbb0d273e840\" (UID: \"ea11f788-17b6-4758-945d-fbb0d273e840\") "
Oct 30 07:34:59.723987 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:59.723978    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea11f788-17b6-4758-945d-fbb0d273e840-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea11f788-17b6-4758-945d-fbb0d273e840-istio-podinfo\") pod \"ea11f788-17b6-4758-945d-fbb0d273e840\" (UID: \"ea11f788-17b6-4758-945d-fbb0d273e840\") "
Oct 30 07:34:59.724093 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:59.724050    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea11f788-17b6-4758-945d-fbb0d273e840-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea11f788-17b6-4758-945d-fbb0d273e840-istiod-ca-cert\") pod \"ea11f788-17b6-4758-945d-fbb0d273e840\" (UID: \"ea11f788-17b6-4758-945d-fbb0d273e840\") "
Oct 30 07:34:59.724151 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:59.724114    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea11f788-17b6-4758-945d-fbb0d273e840-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea11f788-17b6-4758-945d-fbb0d273e840-credential-socket\") pod \"ea11f788-17b6-4758-945d-fbb0d273e840\" (UID: \"ea11f788-17b6-4758-945d-fbb0d273e840\") "
Oct 30 07:34:59.724205 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:59.724164    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea11f788-17b6-4758-945d-fbb0d273e840-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea11f788-17b6-4758-945d-fbb0d273e840-istio-envoy\") pod \"ea11f788-17b6-4758-945d-fbb0d273e840\" (UID: \"ea11f788-17b6-4758-945d-fbb0d273e840\") "
Oct 30 07:34:59.724263 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:59.724248    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea11f788-17b6-4758-945d-fbb0d273e840-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea11f788-17b6-4758-945d-fbb0d273e840-istio-data\") pod \"ea11f788-17b6-4758-945d-fbb0d273e840\" (UID: \"ea11f788-17b6-4758-945d-fbb0d273e840\") "
Oct 30 07:34:59.724926 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:59.724902    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea11f788-17b6-4758-945d-fbb0d273e840-istio-data" pod "ea11f788-17b6-4758-945d-fbb0d273e840" (UID: "ea11f788-17b6-4758-945d-fbb0d273e840"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:34:59.725119 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:59.725092    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea11f788-17b6-4758-945d-fbb0d273e840-workload-socket" pod "ea11f788-17b6-4758-945d-fbb0d273e840" (UID: "ea11f788-17b6-4758-945d-fbb0d273e840"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:34:59.726516 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:59.726472    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea11f788-17b6-4758-945d-fbb0d273e840-credential-socket" pod "ea11f788-17b6-4758-945d-fbb0d273e840" (UID: "ea11f788-17b6-4758-945d-fbb0d273e840"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:34:59.726854 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:59.726785    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea11f788-17b6-4758-945d-fbb0d273e840-istiod-ca-cert" pod "ea11f788-17b6-4758-945d-fbb0d273e840" (UID: "ea11f788-17b6-4758-945d-fbb0d273e840"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:34:59.727128 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:59.727104    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea11f788-17b6-4758-945d-fbb0d273e840-workload-certs" pod "ea11f788-17b6-4758-945d-fbb0d273e840" (UID: "ea11f788-17b6-4758-945d-fbb0d273e840"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:34:59.729730 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:59.729704    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea11f788-17b6-4758-945d-fbb0d273e840-kube-api-access-h4pqm" pod "ea11f788-17b6-4758-945d-fbb0d273e840" (UID: "ea11f788-17b6-4758-945d-fbb0d273e840"). InnerVolumeSpecName "kube-api-access-h4pqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:34:59.730973 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:59.730894    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea11f788-17b6-4758-945d-fbb0d273e840-istio-podinfo" pod "ea11f788-17b6-4758-945d-fbb0d273e840" (UID: "ea11f788-17b6-4758-945d-fbb0d273e840"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:34:59.731370 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:59.731344    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea11f788-17b6-4758-945d-fbb0d273e840-istio-envoy" pod "ea11f788-17b6-4758-945d-fbb0d273e840" (UID: "ea11f788-17b6-4758-945d-fbb0d273e840"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:34:59.732487 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:59.732117    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea11f788-17b6-4758-945d-fbb0d273e840-istio-token" pod "ea11f788-17b6-4758-945d-fbb0d273e840" (UID: "ea11f788-17b6-4758-945d-fbb0d273e840"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:34:59.743993 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:59.743950    6224 scope.go:122] "RemoveContainer" containerID="a4b50c1feb0e7a59388b908e7a1a4da3d70d33efe27b59b3191567d8655cca46"
Oct 30 07:34:59.751943 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:34:59.751910    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4b50c1feb0e7a59388b908e7a1a4da3d70d33efe27b59b3191567d8655cca46\": container with ID starting with a4b50c1feb0e7a59388b908e7a1a4da3d70d33efe27b59b3191567d8655cca46 not found: ID does not exist" containerID="a4b50c1feb0e7a59388b908e7a1a4da3d70d33efe27b59b3191567d8655cca46"
Oct 30 07:34:59.752057 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:59.751952    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4b50c1feb0e7a59388b908e7a1a4da3d70d33efe27b59b3191567d8655cca46"} err="failed to get container status \"a4b50c1feb0e7a59388b908e7a1a4da3d70d33efe27b59b3191567d8655cca46\": rpc error: code = NotFound desc = could not find container \"a4b50c1feb0e7a59388b908e7a1a4da3d70d33efe27b59b3191567d8655cca46\": container with ID starting with a4b50c1feb0e7a59388b908e7a1a4da3d70d33efe27b59b3191567d8655cca46 not found: ID does not exist"
Oct 30 07:34:59.825923 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:59.825848    6224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea11f788-17b6-4758-945d-fbb0d273e840-istio-data\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:34:59.825923 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:59.825875    6224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea11f788-17b6-4758-945d-fbb0d273e840-workload-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:34:59.825923 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:59.825886    6224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea11f788-17b6-4758-945d-fbb0d273e840-workload-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:34:59.825923 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:59.825896    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4pqm\" (UniqueName: \"kubernetes.io/projected/ea11f788-17b6-4758-945d-fbb0d273e840-kube-api-access-h4pqm\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:34:59.825923 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:59.825909    6224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea11f788-17b6-4758-945d-fbb0d273e840-istio-token\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:34:59.825923 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:59.825917    6224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea11f788-17b6-4758-945d-fbb0d273e840-istio-podinfo\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:34:59.825923 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:59.825924    6224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea11f788-17b6-4758-945d-fbb0d273e840-istiod-ca-cert\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:34:59.826255 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:59.825934    6224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea11f788-17b6-4758-945d-fbb0d273e840-credential-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:34:59.826255 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:34:59.825942    6224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea11f788-17b6-4758-945d-fbb0d273e840-istio-envoy\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:35:00.103856 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:00.103762    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v"]
Oct 30 07:35:00.108184 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:00.108146    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-nxm6v"]
Oct 30 07:35:00.684830 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:00.673124    6224 patch_prober.go:29] interesting pod/e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Oct 30 07:35:00.684830 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:00.673184    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh" podUID="6f5bab32-7cdb-471b-9bd0-38b205ace2e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Oct 30 07:35:01.116138 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:01.116100    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c1b5d7-666cdbf94c-kgdnv"]
Oct 30 07:35:01.116784 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:01.116767    6224 cpu_manager.go:414] "containerMap: removing container" podUID="ea11f788-17b6-4758-945d-fbb0d273e840" containerName="istio-proxy"
Oct 30 07:35:01.116868 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:01.116787    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea11f788-17b6-4758-945d-fbb0d273e840" containerName="istio-proxy"
Oct 30 07:35:01.116940 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:01.116929    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea11f788-17b6-4758-945d-fbb0d273e840" containerName="istio-proxy"
Oct 30 07:35:01.120238 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:01.120220    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c1b5d7-666cdbf94c-kgdnv"
Oct 30 07:35:01.146738 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:01.146710    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c1b5d7-666cdbf94c-kgdnv"]
Oct 30 07:35:01.159752 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:01.159718    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4444b346-84b4-4ac2-b445-0b43ec4d6fda-maas-api-tls\") pod \"maas-api-e2e-extmod-b-c1b5d7-666cdbf94c-kgdnv\" (UID: \"4444b346-84b4-4ac2-b445-0b43ec4d6fda\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c1b5d7-666cdbf94c-kgdnv"
Oct 30 07:35:01.159958 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:01.159934    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4444b346-84b4-4ac2-b445-0b43ec4d6fda-metrics-certs\") pod \"maas-api-e2e-extmod-b-c1b5d7-666cdbf94c-kgdnv\" (UID: \"4444b346-84b4-4ac2-b445-0b43ec4d6fda\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c1b5d7-666cdbf94c-kgdnv"
Oct 30 07:35:01.160017 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:01.159998    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x46cm\" (UniqueName: \"kubernetes.io/projected/4444b346-84b4-4ac2-b445-0b43ec4d6fda-kube-api-access-x46cm\") pod \"maas-api-e2e-extmod-b-c1b5d7-666cdbf94c-kgdnv\" (UID: \"4444b346-84b4-4ac2-b445-0b43ec4d6fda\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c1b5d7-666cdbf94c-kgdnv"
Oct 30 07:35:01.160151 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:01.160110    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4444b346-84b4-4ac2-b445-0b43ec4d6fda-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-c1b5d7-666cdbf94c-kgdnv\" (UID: \"4444b346-84b4-4ac2-b445-0b43ec4d6fda\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c1b5d7-666cdbf94c-kgdnv"
Oct 30 07:35:01.215488 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:01.215452    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-c1b5d7-serving-cert\""
Oct 30 07:35:01.261872 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:01.261799    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4444b346-84b4-4ac2-b445-0b43ec4d6fda-maas-api-tls\") pod \"maas-api-e2e-extmod-b-c1b5d7-666cdbf94c-kgdnv\" (UID: \"4444b346-84b4-4ac2-b445-0b43ec4d6fda\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c1b5d7-666cdbf94c-kgdnv"
Oct 30 07:35:01.262055 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:01.261901    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4444b346-84b4-4ac2-b445-0b43ec4d6fda-metrics-certs\") pod \"maas-api-e2e-extmod-b-c1b5d7-666cdbf94c-kgdnv\" (UID: \"4444b346-84b4-4ac2-b445-0b43ec4d6fda\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c1b5d7-666cdbf94c-kgdnv"
Oct 30 07:35:01.262055 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:01.261935    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x46cm\" (UniqueName: \"kubernetes.io/projected/4444b346-84b4-4ac2-b445-0b43ec4d6fda-kube-api-access-x46cm\") pod \"maas-api-e2e-extmod-b-c1b5d7-666cdbf94c-kgdnv\" (UID: \"4444b346-84b4-4ac2-b445-0b43ec4d6fda\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c1b5d7-666cdbf94c-kgdnv"
Oct 30 07:35:01.262055 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:01.262006    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4444b346-84b4-4ac2-b445-0b43ec4d6fda-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-c1b5d7-666cdbf94c-kgdnv\" (UID: \"4444b346-84b4-4ac2-b445-0b43ec4d6fda\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c1b5d7-666cdbf94c-kgdnv"
Oct 30 07:35:01.262827 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:01.262788    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4444b346-84b4-4ac2-b445-0b43ec4d6fda-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-c1b5d7-666cdbf94c-kgdnv\" (UID: \"4444b346-84b4-4ac2-b445-0b43ec4d6fda\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c1b5d7-666cdbf94c-kgdnv"
Oct 30 07:35:01.287827 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:01.287351    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4444b346-84b4-4ac2-b445-0b43ec4d6fda-metrics-certs\") pod \"maas-api-e2e-extmod-b-c1b5d7-666cdbf94c-kgdnv\" (UID: \"4444b346-84b4-4ac2-b445-0b43ec4d6fda\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c1b5d7-666cdbf94c-kgdnv"
Oct 30 07:35:01.291203 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:01.291147    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4444b346-84b4-4ac2-b445-0b43ec4d6fda-maas-api-tls\") pod \"maas-api-e2e-extmod-b-c1b5d7-666cdbf94c-kgdnv\" (UID: \"4444b346-84b4-4ac2-b445-0b43ec4d6fda\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c1b5d7-666cdbf94c-kgdnv"
Oct 30 07:35:01.294155 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:01.294130    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x46cm\" (UniqueName: \"kubernetes.io/projected/4444b346-84b4-4ac2-b445-0b43ec4d6fda-kube-api-access-x46cm\") pod \"maas-api-e2e-extmod-b-c1b5d7-666cdbf94c-kgdnv\" (UID: \"4444b346-84b4-4ac2-b445-0b43ec4d6fda\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c1b5d7-666cdbf94c-kgdnv"
Oct 30 07:35:01.432757 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:01.432675    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c1b5d7-666cdbf94c-kgdnv"
Oct 30 07:35:01.700599 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:01.679169    6224 patch_prober.go:29] interesting pod/e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Oct 30 07:35:01.700599 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:01.679247    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh" podUID="6f5bab32-7cdb-471b-9bd0-38b205ace2e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Oct 30 07:35:01.706544 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:01.706514    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea11f788-17b6-4758-945d-fbb0d273e840" path="/var/lib/kubelet/pods/ea11f788-17b6-4758-945d-fbb0d273e840/volumes"
Oct 30 07:35:01.802956 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:01.802746    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c1b5d7-666cdbf94c-kgdnv"]
Oct 30 07:35:01.859103 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:35:01.859051    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:35:02.013471 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:02.013378    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh"
Oct 30 07:35:02.694328 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:02.694290    6224 patch_prober.go:29] interesting pod/e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Oct 30 07:35:02.694954 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:02.694915    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh" podUID="6f5bab32-7cdb-471b-9bd0-38b205ace2e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Oct 30 07:35:02.741281 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:02.741210    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c1b5d7-666cdbf94c-kgdnv" event={"ID":"4444b346-84b4-4ac2-b445-0b43ec4d6fda","Type":"ContainerStarted","Data":"b44fa25c51d0540cb14035f5750c6c2b54b46351097175bd776336a1471720e5"}
Oct 30 07:35:02.741281 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:02.741260    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c1b5d7-666cdbf94c-kgdnv" event={"ID":"4444b346-84b4-4ac2-b445-0b43ec4d6fda","Type":"ContainerStarted","Data":"54836bac72de78c116ed0feff61a9be7c8c64a85e1437da1cba560419cd10c3c"}
Oct 30 07:35:02.741524 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:02.741420    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c1b5d7-666cdbf94c-kgdnv"
Oct 30 07:35:02.791834 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:02.791368    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c1b5d7-666cdbf94c-kgdnv" podStartSLOduration=1.357939332 podStartE2EDuration="1.791352047s" totalImagesPullingTime="433.412715ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:35:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:35:02.790240651 +0000 UTC m=+2261.399647622" watchObservedRunningTime="2026-10-30 07:35:02.791352047 +0000 UTC m=+2261.400759016"
Oct 30 07:35:03.263433 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:03.263394    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj"]
Oct 30 07:35:03.263658 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:03.263614    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj" podUID="1096c1ae-7e73-4107-ba58-bd4ea5096de2" containerName="istio-proxy" containerID="cri-o://b6496a9ec85face9542aee7bce22549c3304ddfb5d188a98dce113bd8ac4826b" gracePeriod=30
Oct 30 07:35:03.677742 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:03.677704    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh"
Oct 30 07:35:03.678671 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:03.678649    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh"
Oct 30 07:35:04.187208 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:04.187145    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x"
Oct 30 07:35:04.242252 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:04.242221    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/883946b4-c9d3-4c98-904f-9076e7597a1f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/883946b4-c9d3-4c98-904f-9076e7597a1f-istiod-ca-cert\") pod \"883946b4-c9d3-4c98-904f-9076e7597a1f\" (UID: \"883946b4-c9d3-4c98-904f-9076e7597a1f\") "
Oct 30 07:35:04.242427 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:04.242266    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/883946b4-c9d3-4c98-904f-9076e7597a1f-istio-token\" (UniqueName: \"kubernetes.io/projected/883946b4-c9d3-4c98-904f-9076e7597a1f-istio-token\") pod \"883946b4-c9d3-4c98-904f-9076e7597a1f\" (UID: \"883946b4-c9d3-4c98-904f-9076e7597a1f\") "
Oct 30 07:35:04.242427 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:04.242309    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/883946b4-c9d3-4c98-904f-9076e7597a1f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/883946b4-c9d3-4c98-904f-9076e7597a1f-istio-envoy\") pod \"883946b4-c9d3-4c98-904f-9076e7597a1f\" (UID: \"883946b4-c9d3-4c98-904f-9076e7597a1f\") "
Oct 30 07:35:04.242427 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:04.242403    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/883946b4-c9d3-4c98-904f-9076e7597a1f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/883946b4-c9d3-4c98-904f-9076e7597a1f-istio-data\") pod \"883946b4-c9d3-4c98-904f-9076e7597a1f\" (UID: \"883946b4-c9d3-4c98-904f-9076e7597a1f\") "
Oct 30 07:35:04.242601 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:04.242439    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/883946b4-c9d3-4c98-904f-9076e7597a1f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/883946b4-c9d3-4c98-904f-9076e7597a1f-credential-socket\") pod \"883946b4-c9d3-4c98-904f-9076e7597a1f\" (UID: \"883946b4-c9d3-4c98-904f-9076e7597a1f\") "
Oct 30 07:35:04.242601 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:04.242474    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/883946b4-c9d3-4c98-904f-9076e7597a1f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/883946b4-c9d3-4c98-904f-9076e7597a1f-workload-certs\") pod \"883946b4-c9d3-4c98-904f-9076e7597a1f\" (UID: \"883946b4-c9d3-4c98-904f-9076e7597a1f\") "
Oct 30 07:35:04.242601 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:04.242593    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/883946b4-c9d3-4c98-904f-9076e7597a1f-kube-api-access-4866t\" (UniqueName: \"kubernetes.io/projected/883946b4-c9d3-4c98-904f-9076e7597a1f-kube-api-access-4866t\") pod \"883946b4-c9d3-4c98-904f-9076e7597a1f\" (UID: \"883946b4-c9d3-4c98-904f-9076e7597a1f\") "
Oct 30 07:35:04.242754 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:04.242649    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/883946b4-c9d3-4c98-904f-9076e7597a1f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/883946b4-c9d3-4c98-904f-9076e7597a1f-workload-socket\") pod \"883946b4-c9d3-4c98-904f-9076e7597a1f\" (UID: \"883946b4-c9d3-4c98-904f-9076e7597a1f\") "
Oct 30 07:35:04.242754 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:04.242688    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/883946b4-c9d3-4c98-904f-9076e7597a1f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/883946b4-c9d3-4c98-904f-9076e7597a1f-istio-podinfo\") pod \"883946b4-c9d3-4c98-904f-9076e7597a1f\" (UID: \"883946b4-c9d3-4c98-904f-9076e7597a1f\") "
Oct 30 07:35:04.242754 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:04.242727    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/883946b4-c9d3-4c98-904f-9076e7597a1f-credential-socket" pod "883946b4-c9d3-4c98-904f-9076e7597a1f" (UID: "883946b4-c9d3-4c98-904f-9076e7597a1f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:35:04.242959 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:04.242745    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/883946b4-c9d3-4c98-904f-9076e7597a1f-istio-data" pod "883946b4-c9d3-4c98-904f-9076e7597a1f" (UID: "883946b4-c9d3-4c98-904f-9076e7597a1f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:35:04.242959 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:04.242647    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/883946b4-c9d3-4c98-904f-9076e7597a1f-istiod-ca-cert" pod "883946b4-c9d3-4c98-904f-9076e7597a1f" (UID: "883946b4-c9d3-4c98-904f-9076e7597a1f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:35:04.242959 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:04.242947    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/883946b4-c9d3-4c98-904f-9076e7597a1f-workload-socket" pod "883946b4-c9d3-4c98-904f-9076e7597a1f" (UID: "883946b4-c9d3-4c98-904f-9076e7597a1f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:35:04.243124 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:04.243097    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/883946b4-c9d3-4c98-904f-9076e7597a1f-workload-certs" pod "883946b4-c9d3-4c98-904f-9076e7597a1f" (UID: "883946b4-c9d3-4c98-904f-9076e7597a1f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:35:04.243470 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:04.243446    6224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/883946b4-c9d3-4c98-904f-9076e7597a1f-istio-data\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:35:04.243586 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:04.243476    6224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/883946b4-c9d3-4c98-904f-9076e7597a1f-credential-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:35:04.243586 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:04.243492    6224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/883946b4-c9d3-4c98-904f-9076e7597a1f-workload-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:35:04.243586 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:04.243507    6224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/883946b4-c9d3-4c98-904f-9076e7597a1f-workload-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:35:04.243586 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:04.243521    6224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/883946b4-c9d3-4c98-904f-9076e7597a1f-istiod-ca-cert\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:35:04.245065 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:04.245035    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/883946b4-c9d3-4c98-904f-9076e7597a1f-kube-api-access-4866t" pod "883946b4-c9d3-4c98-904f-9076e7597a1f" (UID: "883946b4-c9d3-4c98-904f-9076e7597a1f"). InnerVolumeSpecName "kube-api-access-4866t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:35:04.245172 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:04.245071    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/883946b4-c9d3-4c98-904f-9076e7597a1f-istio-token" pod "883946b4-c9d3-4c98-904f-9076e7597a1f" (UID: "883946b4-c9d3-4c98-904f-9076e7597a1f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:35:04.245172 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:04.245136    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/883946b4-c9d3-4c98-904f-9076e7597a1f-istio-podinfo" pod "883946b4-c9d3-4c98-904f-9076e7597a1f" (UID: "883946b4-c9d3-4c98-904f-9076e7597a1f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:35:04.245566 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:04.245544    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/883946b4-c9d3-4c98-904f-9076e7597a1f-istio-envoy" pod "883946b4-c9d3-4c98-904f-9076e7597a1f" (UID: "883946b4-c9d3-4c98-904f-9076e7597a1f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:35:04.349408 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:04.349371    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4866t\" (UniqueName: \"kubernetes.io/projected/883946b4-c9d3-4c98-904f-9076e7597a1f-kube-api-access-4866t\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:35:04.349408 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:04.349411    6224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/883946b4-c9d3-4c98-904f-9076e7597a1f-istio-podinfo\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:35:04.349658 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:04.349426    6224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/883946b4-c9d3-4c98-904f-9076e7597a1f-istio-token\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:35:04.349658 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:04.349438    6224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/883946b4-c9d3-4c98-904f-9076e7597a1f-istio-envoy\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:35:04.756424 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:04.756389    6224 generic.go:356] "Generic (PLEG): container finished" podID="883946b4-c9d3-4c98-904f-9076e7597a1f" containerID="7bbee109bf27e28dc0874b73c27b9315d236b89c6ea8ccf25c0e0aa5e3f720c4" exitCode=0
Oct 30 07:35:04.756617 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:04.756451    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x"
Oct 30 07:35:04.756617 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:04.756469    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x" event={"ID":"883946b4-c9d3-4c98-904f-9076e7597a1f","Type":"ContainerDied","Data":"7bbee109bf27e28dc0874b73c27b9315d236b89c6ea8ccf25c0e0aa5e3f720c4"}
Oct 30 07:35:04.756617 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:04.756523    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x" event={"ID":"883946b4-c9d3-4c98-904f-9076e7597a1f","Type":"ContainerDied","Data":"61c902f661cf218bb202d8e2e0e105c2eafde7d2365c9d4362fab8d9f3bc966b"}
Oct 30 07:35:04.756617 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:04.756548    6224 scope.go:122] "RemoveContainer" containerID="7bbee109bf27e28dc0874b73c27b9315d236b89c6ea8ccf25c0e0aa5e3f720c4"
Oct 30 07:35:04.769078 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:04.769053    6224 scope.go:122] "RemoveContainer" containerID="7bbee109bf27e28dc0874b73c27b9315d236b89c6ea8ccf25c0e0aa5e3f720c4"
Oct 30 07:35:04.769356 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:35:04.769333    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bbee109bf27e28dc0874b73c27b9315d236b89c6ea8ccf25c0e0aa5e3f720c4\": container with ID starting with 7bbee109bf27e28dc0874b73c27b9315d236b89c6ea8ccf25c0e0aa5e3f720c4 not found: ID does not exist" containerID="7bbee109bf27e28dc0874b73c27b9315d236b89c6ea8ccf25c0e0aa5e3f720c4"
Oct 30 07:35:04.769464 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:04.769366    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bbee109bf27e28dc0874b73c27b9315d236b89c6ea8ccf25c0e0aa5e3f720c4"} err="failed to get container status \"7bbee109bf27e28dc0874b73c27b9315d236b89c6ea8ccf25c0e0aa5e3f720c4\": rpc error: code = NotFound desc = could not find container \"7bbee109bf27e28dc0874b73c27b9315d236b89c6ea8ccf25c0e0aa5e3f720c4\": container with ID starting with 7bbee109bf27e28dc0874b73c27b9315d236b89c6ea8ccf25c0e0aa5e3f720c4 not found: ID does not exist"
Oct 30 07:35:04.820792 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:04.820765    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x"]
Oct 30 07:35:04.834148 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:04.834119    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-xxb4x"]
Oct 30 07:35:05.504308 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:05.504277    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="883946b4-c9d3-4c98-904f-9076e7597a1f" path="/var/lib/kubelet/pods/883946b4-c9d3-4c98-904f-9076e7597a1f/volumes"
Oct 30 07:35:07.699738 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:07.699712    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-ace8fb/payload-processing-external-model-e2e-extmod-b-ace8fb-798c98nqf"
Oct 30 07:35:08.531001 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:08.530981    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj"
Oct 30 07:35:08.605501 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:08.605472    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1096c1ae-7e73-4107-ba58-bd4ea5096de2-istio-token\" (UniqueName: \"kubernetes.io/projected/1096c1ae-7e73-4107-ba58-bd4ea5096de2-istio-token\") pod \"1096c1ae-7e73-4107-ba58-bd4ea5096de2\" (UID: \"1096c1ae-7e73-4107-ba58-bd4ea5096de2\") "
Oct 30 07:35:08.605628 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:08.605549    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1096c1ae-7e73-4107-ba58-bd4ea5096de2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1096c1ae-7e73-4107-ba58-bd4ea5096de2-workload-certs\") pod \"1096c1ae-7e73-4107-ba58-bd4ea5096de2\" (UID: \"1096c1ae-7e73-4107-ba58-bd4ea5096de2\") "
Oct 30 07:35:08.605628 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:08.605573    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1096c1ae-7e73-4107-ba58-bd4ea5096de2-kube-api-access-ql2fq\" (UniqueName: \"kubernetes.io/projected/1096c1ae-7e73-4107-ba58-bd4ea5096de2-kube-api-access-ql2fq\") pod \"1096c1ae-7e73-4107-ba58-bd4ea5096de2\" (UID: \"1096c1ae-7e73-4107-ba58-bd4ea5096de2\") "
Oct 30 07:35:08.605758 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:08.605627    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1096c1ae-7e73-4107-ba58-bd4ea5096de2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1096c1ae-7e73-4107-ba58-bd4ea5096de2-credential-socket\") pod \"1096c1ae-7e73-4107-ba58-bd4ea5096de2\" (UID: \"1096c1ae-7e73-4107-ba58-bd4ea5096de2\") "
Oct 30 07:35:08.605758 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:08.605706    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1096c1ae-7e73-4107-ba58-bd4ea5096de2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1096c1ae-7e73-4107-ba58-bd4ea5096de2-istiod-ca-cert\") pod \"1096c1ae-7e73-4107-ba58-bd4ea5096de2\" (UID: \"1096c1ae-7e73-4107-ba58-bd4ea5096de2\") "
Oct 30 07:35:08.605758 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:08.605738    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1096c1ae-7e73-4107-ba58-bd4ea5096de2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1096c1ae-7e73-4107-ba58-bd4ea5096de2-istio-podinfo\") pod \"1096c1ae-7e73-4107-ba58-bd4ea5096de2\" (UID: \"1096c1ae-7e73-4107-ba58-bd4ea5096de2\") "
Oct 30 07:35:08.605937 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:08.605802    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1096c1ae-7e73-4107-ba58-bd4ea5096de2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1096c1ae-7e73-4107-ba58-bd4ea5096de2-workload-socket\") pod \"1096c1ae-7e73-4107-ba58-bd4ea5096de2\" (UID: \"1096c1ae-7e73-4107-ba58-bd4ea5096de2\") "
Oct 30 07:35:08.605937 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:08.605898    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1096c1ae-7e73-4107-ba58-bd4ea5096de2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1096c1ae-7e73-4107-ba58-bd4ea5096de2-istio-envoy\") pod \"1096c1ae-7e73-4107-ba58-bd4ea5096de2\" (UID: \"1096c1ae-7e73-4107-ba58-bd4ea5096de2\") "
Oct 30 07:35:08.605937 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:08.605898    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1096c1ae-7e73-4107-ba58-bd4ea5096de2-workload-certs" pod "1096c1ae-7e73-4107-ba58-bd4ea5096de2" (UID: "1096c1ae-7e73-4107-ba58-bd4ea5096de2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:35:08.606122 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:08.606001    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1096c1ae-7e73-4107-ba58-bd4ea5096de2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1096c1ae-7e73-4107-ba58-bd4ea5096de2-istio-data\") pod \"1096c1ae-7e73-4107-ba58-bd4ea5096de2\" (UID: \"1096c1ae-7e73-4107-ba58-bd4ea5096de2\") "
Oct 30 07:35:08.606693 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:08.606665    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1096c1ae-7e73-4107-ba58-bd4ea5096de2-istiod-ca-cert" pod "1096c1ae-7e73-4107-ba58-bd4ea5096de2" (UID: "1096c1ae-7e73-4107-ba58-bd4ea5096de2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:35:08.606693 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:08.606686    6224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1096c1ae-7e73-4107-ba58-bd4ea5096de2-workload-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:35:08.607220 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:08.607199    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1096c1ae-7e73-4107-ba58-bd4ea5096de2-istio-data" pod "1096c1ae-7e73-4107-ba58-bd4ea5096de2" (UID: "1096c1ae-7e73-4107-ba58-bd4ea5096de2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:35:08.607316 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:08.607274    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1096c1ae-7e73-4107-ba58-bd4ea5096de2-workload-socket" pod "1096c1ae-7e73-4107-ba58-bd4ea5096de2" (UID: "1096c1ae-7e73-4107-ba58-bd4ea5096de2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:35:08.607378 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:08.607348    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1096c1ae-7e73-4107-ba58-bd4ea5096de2-credential-socket" pod "1096c1ae-7e73-4107-ba58-bd4ea5096de2" (UID: "1096c1ae-7e73-4107-ba58-bd4ea5096de2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:35:08.608166 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:08.608138    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1096c1ae-7e73-4107-ba58-bd4ea5096de2-istio-token" pod "1096c1ae-7e73-4107-ba58-bd4ea5096de2" (UID: "1096c1ae-7e73-4107-ba58-bd4ea5096de2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:35:08.608781 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:08.608751    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1096c1ae-7e73-4107-ba58-bd4ea5096de2-istio-envoy" pod "1096c1ae-7e73-4107-ba58-bd4ea5096de2" (UID: "1096c1ae-7e73-4107-ba58-bd4ea5096de2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:35:08.608907 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:08.608854    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1096c1ae-7e73-4107-ba58-bd4ea5096de2-kube-api-access-ql2fq" pod "1096c1ae-7e73-4107-ba58-bd4ea5096de2" (UID: "1096c1ae-7e73-4107-ba58-bd4ea5096de2"). InnerVolumeSpecName "kube-api-access-ql2fq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:35:08.608996 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:08.608973    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1096c1ae-7e73-4107-ba58-bd4ea5096de2-istio-podinfo" pod "1096c1ae-7e73-4107-ba58-bd4ea5096de2" (UID: "1096c1ae-7e73-4107-ba58-bd4ea5096de2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:35:08.708650 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:08.708613    6224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1096c1ae-7e73-4107-ba58-bd4ea5096de2-istiod-ca-cert\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:35:08.708650 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:08.708651    6224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1096c1ae-7e73-4107-ba58-bd4ea5096de2-istio-podinfo\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:35:08.709174 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:08.708664    6224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1096c1ae-7e73-4107-ba58-bd4ea5096de2-workload-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:35:08.709174 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:08.708679    6224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1096c1ae-7e73-4107-ba58-bd4ea5096de2-istio-envoy\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:35:08.709174 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:08.708690    6224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1096c1ae-7e73-4107-ba58-bd4ea5096de2-istio-data\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:35:08.709174 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:08.708701    6224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1096c1ae-7e73-4107-ba58-bd4ea5096de2-istio-token\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:35:08.709174 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:08.708713    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql2fq\" (UniqueName: \"kubernetes.io/projected/1096c1ae-7e73-4107-ba58-bd4ea5096de2-kube-api-access-ql2fq\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:35:08.709174 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:08.708725    6224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1096c1ae-7e73-4107-ba58-bd4ea5096de2-credential-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:35:08.753446 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:08.753415    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c1b5d7-666cdbf94c-kgdnv"
Oct 30 07:35:08.795194 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:08.795123    6224 generic.go:356] "Generic (PLEG): container finished" podID="1096c1ae-7e73-4107-ba58-bd4ea5096de2" containerID="b6496a9ec85face9542aee7bce22549c3304ddfb5d188a98dce113bd8ac4826b" exitCode=0
Oct 30 07:35:08.795194 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:08.795164    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj"
Oct 30 07:35:08.795359 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:08.795208    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj" event={"ID":"1096c1ae-7e73-4107-ba58-bd4ea5096de2","Type":"ContainerDied","Data":"b6496a9ec85face9542aee7bce22549c3304ddfb5d188a98dce113bd8ac4826b"}
Oct 30 07:35:08.795359 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:08.795257    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj" event={"ID":"1096c1ae-7e73-4107-ba58-bd4ea5096de2","Type":"ContainerDied","Data":"2a5d536c33ae644a1445ebe4137807f162df4ede7995f15021cd59c8e028e8a6"}
Oct 30 07:35:08.795359 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:08.795280    6224 scope.go:122] "RemoveContainer" containerID="b6496a9ec85face9542aee7bce22549c3304ddfb5d188a98dce113bd8ac4826b"
Oct 30 07:35:08.808372 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:08.808350    6224 scope.go:122] "RemoveContainer" containerID="b6496a9ec85face9542aee7bce22549c3304ddfb5d188a98dce113bd8ac4826b"
Oct 30 07:35:08.808642 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:35:08.808622    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6496a9ec85face9542aee7bce22549c3304ddfb5d188a98dce113bd8ac4826b\": container with ID starting with b6496a9ec85face9542aee7bce22549c3304ddfb5d188a98dce113bd8ac4826b not found: ID does not exist" containerID="b6496a9ec85face9542aee7bce22549c3304ddfb5d188a98dce113bd8ac4826b"
Oct 30 07:35:08.808711 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:08.808659    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6496a9ec85face9542aee7bce22549c3304ddfb5d188a98dce113bd8ac4826b"} err="failed to get container status \"b6496a9ec85face9542aee7bce22549c3304ddfb5d188a98dce113bd8ac4826b\": rpc error: code = NotFound desc = could not find container \"b6496a9ec85face9542aee7bce22549c3304ddfb5d188a98dce113bd8ac4826b\": container with ID starting with b6496a9ec85face9542aee7bce22549c3304ddfb5d188a98dce113bd8ac4826b not found: ID does not exist"
Oct 30 07:35:08.869090 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:08.869060    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj"]
Oct 30 07:35:08.942393 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:08.942361    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-c22pj"]
Oct 30 07:35:09.504492 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:09.504460    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1096c1ae-7e73-4107-ba58-bd4ea5096de2" path="/var/lib/kubelet/pods/1096c1ae-7e73-4107-ba58-bd4ea5096de2/volumes"
Oct 30 07:35:18.105868 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:18.103321    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:35:22.597719 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:22.597687    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d0c2f-54d45b7497-c4bvr"]
Oct 30 07:35:22.598323 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:22.598308    6224 cpu_manager.go:414] "containerMap: removing container" podUID="883946b4-c9d3-4c98-904f-9076e7597a1f" containerName="istio-proxy"
Oct 30 07:35:22.598367 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:22.598324    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="883946b4-c9d3-4c98-904f-9076e7597a1f" containerName="istio-proxy"
Oct 30 07:35:22.598367 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:22.598336    6224 cpu_manager.go:414] "containerMap: removing container" podUID="1096c1ae-7e73-4107-ba58-bd4ea5096de2" containerName="istio-proxy"
Oct 30 07:35:22.598367 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:22.598342    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1096c1ae-7e73-4107-ba58-bd4ea5096de2" containerName="istio-proxy"
Oct 30 07:35:22.598489 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:22.598480    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="883946b4-c9d3-4c98-904f-9076e7597a1f" containerName="istio-proxy"
Oct 30 07:35:22.598524 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:22.598490    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="1096c1ae-7e73-4107-ba58-bd4ea5096de2" containerName="istio-proxy"
Oct 30 07:35:22.602249 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:22.602233    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d0c2f-54d45b7497-c4bvr"
Oct 30 07:35:22.618743 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:22.618720    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-8d0c2f-serving-cert\""
Oct 30 07:35:22.636599 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:22.636570    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d0c2f-54d45b7497-c4bvr"]
Oct 30 07:35:22.788595 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:22.788553    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkshr\" (UniqueName: \"kubernetes.io/projected/543279e1-d9a2-413b-a6ad-f76814756067-kube-api-access-mkshr\") pod \"maas-api-e2e-extmod-a-8d0c2f-54d45b7497-c4bvr\" (UID: \"543279e1-d9a2-413b-a6ad-f76814756067\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d0c2f-54d45b7497-c4bvr"
Oct 30 07:35:22.788751 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:22.788617    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/543279e1-d9a2-413b-a6ad-f76814756067-metrics-certs\") pod \"maas-api-e2e-extmod-a-8d0c2f-54d45b7497-c4bvr\" (UID: \"543279e1-d9a2-413b-a6ad-f76814756067\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d0c2f-54d45b7497-c4bvr"
Oct 30 07:35:22.788751 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:22.788696    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/543279e1-d9a2-413b-a6ad-f76814756067-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-8d0c2f-54d45b7497-c4bvr\" (UID: \"543279e1-d9a2-413b-a6ad-f76814756067\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d0c2f-54d45b7497-c4bvr"
Oct 30 07:35:22.788842 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:22.788817    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/543279e1-d9a2-413b-a6ad-f76814756067-maas-api-tls\") pod \"maas-api-e2e-extmod-a-8d0c2f-54d45b7497-c4bvr\" (UID: \"543279e1-d9a2-413b-a6ad-f76814756067\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d0c2f-54d45b7497-c4bvr"
Oct 30 07:35:22.894370 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:22.892706    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/543279e1-d9a2-413b-a6ad-f76814756067-maas-api-tls\") pod \"maas-api-e2e-extmod-a-8d0c2f-54d45b7497-c4bvr\" (UID: \"543279e1-d9a2-413b-a6ad-f76814756067\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d0c2f-54d45b7497-c4bvr"
Oct 30 07:35:22.894370 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:22.892899    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkshr\" (UniqueName: \"kubernetes.io/projected/543279e1-d9a2-413b-a6ad-f76814756067-kube-api-access-mkshr\") pod \"maas-api-e2e-extmod-a-8d0c2f-54d45b7497-c4bvr\" (UID: \"543279e1-d9a2-413b-a6ad-f76814756067\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d0c2f-54d45b7497-c4bvr"
Oct 30 07:35:22.894370 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:22.892965    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/543279e1-d9a2-413b-a6ad-f76814756067-metrics-certs\") pod \"maas-api-e2e-extmod-a-8d0c2f-54d45b7497-c4bvr\" (UID: \"543279e1-d9a2-413b-a6ad-f76814756067\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d0c2f-54d45b7497-c4bvr"
Oct 30 07:35:22.894370 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:22.893015    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/543279e1-d9a2-413b-a6ad-f76814756067-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-8d0c2f-54d45b7497-c4bvr\" (UID: \"543279e1-d9a2-413b-a6ad-f76814756067\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d0c2f-54d45b7497-c4bvr"
Oct 30 07:35:22.894370 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:22.893932    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/543279e1-d9a2-413b-a6ad-f76814756067-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-8d0c2f-54d45b7497-c4bvr\" (UID: \"543279e1-d9a2-413b-a6ad-f76814756067\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d0c2f-54d45b7497-c4bvr"
Oct 30 07:35:22.898544 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:22.898515    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/543279e1-d9a2-413b-a6ad-f76814756067-maas-api-tls\") pod \"maas-api-e2e-extmod-a-8d0c2f-54d45b7497-c4bvr\" (UID: \"543279e1-d9a2-413b-a6ad-f76814756067\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d0c2f-54d45b7497-c4bvr"
Oct 30 07:35:22.902943 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:22.902919    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/543279e1-d9a2-413b-a6ad-f76814756067-metrics-certs\") pod \"maas-api-e2e-extmod-a-8d0c2f-54d45b7497-c4bvr\" (UID: \"543279e1-d9a2-413b-a6ad-f76814756067\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d0c2f-54d45b7497-c4bvr"
Oct 30 07:35:22.923352 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:22.923324    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkshr\" (UniqueName: \"kubernetes.io/projected/543279e1-d9a2-413b-a6ad-f76814756067-kube-api-access-mkshr\") pod \"maas-api-e2e-extmod-a-8d0c2f-54d45b7497-c4bvr\" (UID: \"543279e1-d9a2-413b-a6ad-f76814756067\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d0c2f-54d45b7497-c4bvr"
Oct 30 07:35:23.215563 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:23.215479    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d0c2f-54d45b7497-c4bvr"
Oct 30 07:35:23.517223 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:35:23.517194    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod543279e1_d9a2_413b_a6ad_f76814756067.slice/crio-e903ed86356ad415ff98a14662b8a7033df2d69071fe68d9c6d1b5e865fa325f WatchSource:0}: Error finding container e903ed86356ad415ff98a14662b8a7033df2d69071fe68d9c6d1b5e865fa325f: Status 404 returned error can't find the container with id e903ed86356ad415ff98a14662b8a7033df2d69071fe68d9c6d1b5e865fa325f
Oct 30 07:35:23.518260 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:23.518235    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d0c2f-54d45b7497-c4bvr"]
Oct 30 07:35:23.919359 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:23.919325    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d0c2f-54d45b7497-c4bvr" event={"ID":"543279e1-d9a2-413b-a6ad-f76814756067","Type":"ContainerStarted","Data":"9659922acad6ef26ae72b22432c038f00364a386d9381e4a4b19d54ccc41abd1"}
Oct 30 07:35:23.919878 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:23.919382    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d0c2f-54d45b7497-c4bvr"
Oct 30 07:35:23.919878 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:23.919399    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d0c2f-54d45b7497-c4bvr" event={"ID":"543279e1-d9a2-413b-a6ad-f76814756067","Type":"ContainerStarted","Data":"e903ed86356ad415ff98a14662b8a7033df2d69071fe68d9c6d1b5e865fa325f"}
Oct 30 07:35:23.966317 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:23.966262    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d0c2f-54d45b7497-c4bvr" podStartSLOduration=1.703477156 podStartE2EDuration="1.966246818s" totalImagesPullingTime="262.769662ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:35:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:35:23.965314326 +0000 UTC m=+2282.574721308" watchObservedRunningTime="2026-10-30 07:35:23.966246818 +0000 UTC m=+2282.575653787"
Oct 30 07:35:29.931636 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:29.931609    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d0c2f-54d45b7497-c4bvr"
Oct 30 07:35:30.370101 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:30.370072    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-77f56l8gh"]
Oct 30 07:35:30.374716 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:30.374699    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-77f56l8gh"
Oct 30 07:35:30.407403 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:30.407383    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f1f700\"/\"payload-processing-external-model-plugins-e2e-extmod-b-f1f700\""
Oct 30 07:35:30.410691 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:30.410673    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-77f56l8gh"]
Oct 30 07:35:30.420222 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:30.420199    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f1f700\"/\"kube-root-ca.crt\""
Oct 30 07:35:30.420222 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:30.420209    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f1f700\"/\"routing-overlay\""
Oct 30 07:35:30.420403 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:30.420219    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f1f700\"/\"openshift-service-ca.crt\""
Oct 30 07:35:30.426146 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:30.426124    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-f1f700\"/\"agc-cred-2c0ab0fb80d7\""
Oct 30 07:35:30.426267 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:30.426174    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-f1f700\"/\"payload-processing-external-model-e2e--96fc5bc0-dockercfg-tlgdn\""
Oct 30 07:35:30.495182 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:30.495150    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cf51f674-5137-4dc3-a4a2-2914526f9d9a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1f700-77f56l8gh\" (UID: \"cf51f674-5137-4dc3-a4a2-2914526f9d9a\") " pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-77f56l8gh"
Oct 30 07:35:30.495311 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:30.495292    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82wgt\" (UniqueName: \"kubernetes.io/projected/cf51f674-5137-4dc3-a4a2-2914526f9d9a-kube-api-access-82wgt\") pod \"payload-processing-external-model-e2e-extmod-b-f1f700-77f56l8gh\" (UID: \"cf51f674-5137-4dc3-a4a2-2914526f9d9a\") " pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-77f56l8gh"
Oct 30 07:35:30.495356 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:30.495338    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cf51f674-5137-4dc3-a4a2-2914526f9d9a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f1f700-77f56l8gh\" (UID: \"cf51f674-5137-4dc3-a4a2-2914526f9d9a\") " pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-77f56l8gh"
Oct 30 07:35:30.495461 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:30.495448    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf51f674-5137-4dc3-a4a2-2914526f9d9a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1f700-77f56l8gh\" (UID: \"cf51f674-5137-4dc3-a4a2-2914526f9d9a\") " pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-77f56l8gh"
Oct 30 07:35:30.646600 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:30.609664    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cf51f674-5137-4dc3-a4a2-2914526f9d9a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1f700-77f56l8gh\" (UID: \"cf51f674-5137-4dc3-a4a2-2914526f9d9a\") " pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-77f56l8gh"
Oct 30 07:35:30.646600 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:30.609832    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82wgt\" (UniqueName: \"kubernetes.io/projected/cf51f674-5137-4dc3-a4a2-2914526f9d9a-kube-api-access-82wgt\") pod \"payload-processing-external-model-e2e-extmod-b-f1f700-77f56l8gh\" (UID: \"cf51f674-5137-4dc3-a4a2-2914526f9d9a\") " pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-77f56l8gh"
Oct 30 07:35:30.646600 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:30.609868    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cf51f674-5137-4dc3-a4a2-2914526f9d9a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f1f700-77f56l8gh\" (UID: \"cf51f674-5137-4dc3-a4a2-2914526f9d9a\") " pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-77f56l8gh"
Oct 30 07:35:30.646600 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:30.609982    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf51f674-5137-4dc3-a4a2-2914526f9d9a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1f700-77f56l8gh\" (UID: \"cf51f674-5137-4dc3-a4a2-2914526f9d9a\") " pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-77f56l8gh"
Oct 30 07:35:30.646600 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:30.611023    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf51f674-5137-4dc3-a4a2-2914526f9d9a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1f700-77f56l8gh\" (UID: \"cf51f674-5137-4dc3-a4a2-2914526f9d9a\") " pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-77f56l8gh"
Oct 30 07:35:30.646600 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:30.611865    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cf51f674-5137-4dc3-a4a2-2914526f9d9a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1f700-77f56l8gh\" (UID: \"cf51f674-5137-4dc3-a4a2-2914526f9d9a\") " pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-77f56l8gh"
Oct 30 07:35:30.648490 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:30.648416    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82wgt\" (UniqueName: \"kubernetes.io/projected/cf51f674-5137-4dc3-a4a2-2914526f9d9a-kube-api-access-82wgt\") pod \"payload-processing-external-model-e2e-extmod-b-f1f700-77f56l8gh\" (UID: \"cf51f674-5137-4dc3-a4a2-2914526f9d9a\") " pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-77f56l8gh"
Oct 30 07:35:30.649166 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:30.649142    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cf51f674-5137-4dc3-a4a2-2914526f9d9a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f1f700-77f56l8gh\" (UID: \"cf51f674-5137-4dc3-a4a2-2914526f9d9a\") " pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-77f56l8gh"
Oct 30 07:35:30.688899 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:30.688853    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-77f56l8gh"
Oct 30 07:35:31.000559 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:31.000530    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-77f56l8gh"]
Oct 30 07:35:31.000939 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:35:31.000912    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf51f674_5137_4dc3_a4a2_2914526f9d9a.slice/crio-c48ce44b549f884374288fe8a8ce5ff4a2af4e9da003d2de69471ff794fe3bce WatchSource:0}: Error finding container c48ce44b549f884374288fe8a8ce5ff4a2af4e9da003d2de69471ff794fe3bce: Status 404 returned error can't find the container with id c48ce44b549f884374288fe8a8ce5ff4a2af4e9da003d2de69471ff794fe3bce
Oct 30 07:35:31.857262 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:35:31.857220    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:35:31.986073 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:31.986038    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-77f56l8gh" event={"ID":"cf51f674-5137-4dc3-a4a2-2914526f9d9a","Type":"ContainerStarted","Data":"f0a6e0bb291d415c1dd7641b515a2b5d81cbf34af780eea8efef71d04c7d78c5"}
Oct 30 07:35:31.986246 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:31.986088    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-77f56l8gh" event={"ID":"cf51f674-5137-4dc3-a4a2-2914526f9d9a","Type":"ContainerStarted","Data":"c48ce44b549f884374288fe8a8ce5ff4a2af4e9da003d2de69471ff794fe3bce"}
Oct 30 07:35:31.986246 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:31.986119    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-77f56l8gh"
Oct 30 07:35:32.031413 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:32.031368    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-77f56l8gh" podStartSLOduration=2.031355466 podStartE2EDuration="2.031355466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:35:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:35:32.031148129 +0000 UTC m=+2290.640555101" watchObservedRunningTime="2026-10-30 07:35:32.031355466 +0000 UTC m=+2290.640762435"
Oct 30 07:35:42.996641 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:35:42.996610    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-77f56l8gh"
Oct 30 07:36:01.493324 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:01.493290    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-55c9vn487"]
Oct 30 07:36:01.500165 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:01.500137    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-55c9vn487"
Oct 30 07:36:01.521923 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:01.521886    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-55c9vn487"]
Oct 30 07:36:01.530149 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:01.530126    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-c1b5d7\"/\"agc-cred-ed7329a72f37\""
Oct 30 07:36:01.530335 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:01.530135    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c1b5d7\"/\"kube-root-ca.crt\""
Oct 30 07:36:01.530456 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:01.530177    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c1b5d7\"/\"routing-overlay\""
Oct 30 07:36:01.530456 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:01.530321    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c1b5d7\"/\"payload-processing-external-model-plugins-e2e-extmod-b-c1b5d7\""
Oct 30 07:36:01.530727 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:01.530112    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c1b5d7\"/\"openshift-service-ca.crt\""
Oct 30 07:36:01.533538 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:01.533519    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-c1b5d7\"/\"payload-processing-external-model-e2e--68f86622-dockercfg-nx9xx\""
Oct 30 07:36:01.614077 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:01.614044    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1cc06823-94c3-484d-bdcd-89e44e945500-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c1b5d7-55c9vn487\" (UID: \"1cc06823-94c3-484d-bdcd-89e44e945500\") " pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-55c9vn487"
Oct 30 07:36:01.614267 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:01.614192    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1cc06823-94c3-484d-bdcd-89e44e945500-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c1b5d7-55c9vn487\" (UID: \"1cc06823-94c3-484d-bdcd-89e44e945500\") " pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-55c9vn487"
Oct 30 07:36:01.614267 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:01.614249    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-548jr\" (UniqueName: \"kubernetes.io/projected/1cc06823-94c3-484d-bdcd-89e44e945500-kube-api-access-548jr\") pod \"payload-processing-external-model-e2e-extmod-b-c1b5d7-55c9vn487\" (UID: \"1cc06823-94c3-484d-bdcd-89e44e945500\") " pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-55c9vn487"
Oct 30 07:36:01.614507 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:01.614477    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cc06823-94c3-484d-bdcd-89e44e945500-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c1b5d7-55c9vn487\" (UID: \"1cc06823-94c3-484d-bdcd-89e44e945500\") " pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-55c9vn487"
Oct 30 07:36:01.717332 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:01.717297    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1cc06823-94c3-484d-bdcd-89e44e945500-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c1b5d7-55c9vn487\" (UID: \"1cc06823-94c3-484d-bdcd-89e44e945500\") " pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-55c9vn487"
Oct 30 07:36:01.718497 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:01.718364    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1cc06823-94c3-484d-bdcd-89e44e945500-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c1b5d7-55c9vn487\" (UID: \"1cc06823-94c3-484d-bdcd-89e44e945500\") " pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-55c9vn487"
Oct 30 07:36:01.718697 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:01.718486    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1cc06823-94c3-484d-bdcd-89e44e945500-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c1b5d7-55c9vn487\" (UID: \"1cc06823-94c3-484d-bdcd-89e44e945500\") " pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-55c9vn487"
Oct 30 07:36:01.718887 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:01.718865    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-548jr\" (UniqueName: \"kubernetes.io/projected/1cc06823-94c3-484d-bdcd-89e44e945500-kube-api-access-548jr\") pod \"payload-processing-external-model-e2e-extmod-b-c1b5d7-55c9vn487\" (UID: \"1cc06823-94c3-484d-bdcd-89e44e945500\") " pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-55c9vn487"
Oct 30 07:36:01.719133 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:01.719111    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cc06823-94c3-484d-bdcd-89e44e945500-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c1b5d7-55c9vn487\" (UID: \"1cc06823-94c3-484d-bdcd-89e44e945500\") " pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-55c9vn487"
Oct 30 07:36:01.720145 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:01.720118    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cc06823-94c3-484d-bdcd-89e44e945500-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c1b5d7-55c9vn487\" (UID: \"1cc06823-94c3-484d-bdcd-89e44e945500\") " pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-55c9vn487"
Oct 30 07:36:01.723368 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:01.723343    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1cc06823-94c3-484d-bdcd-89e44e945500-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c1b5d7-55c9vn487\" (UID: \"1cc06823-94c3-484d-bdcd-89e44e945500\") " pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-55c9vn487"
Oct 30 07:36:01.756321 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:01.756256    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-548jr\" (UniqueName: \"kubernetes.io/projected/1cc06823-94c3-484d-bdcd-89e44e945500-kube-api-access-548jr\") pod \"payload-processing-external-model-e2e-extmod-b-c1b5d7-55c9vn487\" (UID: \"1cc06823-94c3-484d-bdcd-89e44e945500\") " pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-55c9vn487"
Oct 30 07:36:01.817580 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:01.817540    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-55c9vn487"
Oct 30 07:36:01.884566 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:36:01.878877    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:36:02.130975 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:36:02.130943    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cc06823_94c3_484d_bdcd_89e44e945500.slice/crio-8dc2a8294722e3a05b099fcc942fb724ea097d7938f9b299719eaa567dc60245 WatchSource:0}: Error finding container 8dc2a8294722e3a05b099fcc942fb724ea097d7938f9b299719eaa567dc60245: Status 404 returned error can't find the container with id 8dc2a8294722e3a05b099fcc942fb724ea097d7938f9b299719eaa567dc60245
Oct 30 07:36:02.131215 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:02.131190    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-55c9vn487"]
Oct 30 07:36:02.245576 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:02.245542    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-55c9vn487" event={"ID":"1cc06823-94c3-484d-bdcd-89e44e945500","Type":"ContainerStarted","Data":"1435c033837bd89a781a9a748af833eb1f3792b42027ebe2b086b9214151d065"}
Oct 30 07:36:02.245760 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:02.245600    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-55c9vn487"
Oct 30 07:36:02.245760 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:02.245616    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-55c9vn487" event={"ID":"1cc06823-94c3-484d-bdcd-89e44e945500","Type":"ContainerStarted","Data":"8dc2a8294722e3a05b099fcc942fb724ea097d7938f9b299719eaa567dc60245"}
Oct 30 07:36:02.328692 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:02.328640    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-55c9vn487" podStartSLOduration=1.328626794 podStartE2EDuration="1.328626794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:36:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:36:02.326261912 +0000 UTC m=+2320.935668883" watchObservedRunningTime="2026-10-30 07:36:02.328626794 +0000 UTC m=+2320.938033763"
Oct 30 07:36:13.259149 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:13.259112    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-55c9vn487"
Oct 30 07:36:13.341415 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:13.341382    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:36:19.582346 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:19.582301    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-6b49z5qnn"]
Oct 30 07:36:19.587764 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:19.587741    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-6b49z5qnn"
Oct 30 07:36:19.605196 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:19.605169    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-6b49z5qnn"]
Oct 30 07:36:19.615994 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:19.615737    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1f34aa\"/\"agc-cred-dd6c04d9ae3b\""
Oct 30 07:36:19.615994 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:19.615851    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1f34aa\"/\"agc-cred-652ba7d1390f\""
Oct 30 07:36:19.615994 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:19.615888    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1f34aa\"/\"kube-root-ca.crt\""
Oct 30 07:36:19.615994 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:19.615966    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1f34aa\"/\"openshift-service-ca.crt\""
Oct 30 07:36:19.618180 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:19.618137    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1f34aa\"/\"routing-overlay\""
Oct 30 07:36:19.618180 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:19.618140    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1f34aa\"/\"payload-processing-external-model-plugins-e2e-extmod-b-1f34aa\""
Oct 30 07:36:19.618180 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:19.618141    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1f34aa\"/\"payload-processing-external-model-e2e--e56a4edc-dockercfg-s85b7\""
Oct 30 07:36:19.659060 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:19.659029    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/35f3e736-96db-420c-8baf-880e3c7ca7ed-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-6b49z5qnn\" (UID: \"35f3e736-96db-420c-8baf-880e3c7ca7ed\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-6b49z5qnn"
Oct 30 07:36:19.659228 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:19.659075    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/35f3e736-96db-420c-8baf-880e3c7ca7ed-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-6b49z5qnn\" (UID: \"35f3e736-96db-420c-8baf-880e3c7ca7ed\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-6b49z5qnn"
Oct 30 07:36:19.659228 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:19.659184    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35f3e736-96db-420c-8baf-880e3c7ca7ed-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-6b49z5qnn\" (UID: \"35f3e736-96db-420c-8baf-880e3c7ca7ed\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-6b49z5qnn"
Oct 30 07:36:19.659353 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:19.659238    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxv4p\" (UniqueName: \"kubernetes.io/projected/35f3e736-96db-420c-8baf-880e3c7ca7ed-kube-api-access-lxv4p\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-6b49z5qnn\" (UID: \"35f3e736-96db-420c-8baf-880e3c7ca7ed\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-6b49z5qnn"
Oct 30 07:36:19.760550 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:19.760516    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/35f3e736-96db-420c-8baf-880e3c7ca7ed-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-6b49z5qnn\" (UID: \"35f3e736-96db-420c-8baf-880e3c7ca7ed\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-6b49z5qnn"
Oct 30 07:36:19.760712 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:19.760567    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/35f3e736-96db-420c-8baf-880e3c7ca7ed-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-6b49z5qnn\" (UID: \"35f3e736-96db-420c-8baf-880e3c7ca7ed\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-6b49z5qnn"
Oct 30 07:36:19.760712 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:19.760610    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35f3e736-96db-420c-8baf-880e3c7ca7ed-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-6b49z5qnn\" (UID: \"35f3e736-96db-420c-8baf-880e3c7ca7ed\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-6b49z5qnn"
Oct 30 07:36:19.760712 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:19.760658    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxv4p\" (UniqueName: \"kubernetes.io/projected/35f3e736-96db-420c-8baf-880e3c7ca7ed-kube-api-access-lxv4p\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-6b49z5qnn\" (UID: \"35f3e736-96db-420c-8baf-880e3c7ca7ed\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-6b49z5qnn"
Oct 30 07:36:19.761317 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:19.761292    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/35f3e736-96db-420c-8baf-880e3c7ca7ed-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-6b49z5qnn\" (UID: \"35f3e736-96db-420c-8baf-880e3c7ca7ed\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-6b49z5qnn"
Oct 30 07:36:19.761420 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:19.761404    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35f3e736-96db-420c-8baf-880e3c7ca7ed-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-6b49z5qnn\" (UID: \"35f3e736-96db-420c-8baf-880e3c7ca7ed\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-6b49z5qnn"
Oct 30 07:36:19.763090 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:19.763068    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/35f3e736-96db-420c-8baf-880e3c7ca7ed-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-6b49z5qnn\" (UID: \"35f3e736-96db-420c-8baf-880e3c7ca7ed\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-6b49z5qnn"
Oct 30 07:36:19.786094 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:19.786071    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxv4p\" (UniqueName: \"kubernetes.io/projected/35f3e736-96db-420c-8baf-880e3c7ca7ed-kube-api-access-lxv4p\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-6b49z5qnn\" (UID: \"35f3e736-96db-420c-8baf-880e3c7ca7ed\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-6b49z5qnn"
Oct 30 07:36:19.903356 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:19.903281    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-6b49z5qnn"
Oct 30 07:36:20.230958 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:20.230928    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-6b49z5qnn"]
Oct 30 07:36:20.231406 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:36:20.231374    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35f3e736_96db_420c_8baf_880e3c7ca7ed.slice/crio-d20e5d0d7da72da5a0180eff334c7c2bd43a02c2e8d6bb1e8b77984b3e1e2f44 WatchSource:0}: Error finding container d20e5d0d7da72da5a0180eff334c7c2bd43a02c2e8d6bb1e8b77984b3e1e2f44: Status 404 returned error can't find the container with id d20e5d0d7da72da5a0180eff334c7c2bd43a02c2e8d6bb1e8b77984b3e1e2f44
Oct 30 07:36:20.382089 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:20.382056    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-6b49z5qnn" event={"ID":"35f3e736-96db-420c-8baf-880e3c7ca7ed","Type":"ContainerStarted","Data":"75826de9c2ec1150e98130206f759c09eda085d833a01378cc388c941a95c71f"}
Oct 30 07:36:20.382268 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:20.382115    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-6b49z5qnn"
Oct 30 07:36:20.382268 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:20.382135    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-6b49z5qnn" event={"ID":"35f3e736-96db-420c-8baf-880e3c7ca7ed","Type":"ContainerStarted","Data":"d20e5d0d7da72da5a0180eff334c7c2bd43a02c2e8d6bb1e8b77984b3e1e2f44"}
Oct 30 07:36:20.428409 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:20.428360    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-6b49z5qnn" podStartSLOduration=1.4283453179999999 podStartE2EDuration="1.428345318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:36:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:36:20.426674874 +0000 UTC m=+2339.036081843" watchObservedRunningTime="2026-10-30 07:36:20.428345318 +0000 UTC m=+2339.037752303"
Oct 30 07:36:24.215662 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:24.215619    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e27e0c-6996bc9b75-kmm89"]
Oct 30 07:36:24.220850 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:24.220824    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e27e0c-6996bc9b75-kmm89"
Oct 30 07:36:24.240111 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:24.240088    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-e27e0c\""
Oct 30 07:36:24.254292 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:24.254266    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-e27e0c-dockercfg-k9b7t\""
Oct 30 07:36:24.258858 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:24.258828    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e27e0c-6996bc9b75-kmm89"]
Oct 30 07:36:24.390642 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:24.390598    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b78cab03-456c-4ea9-93bb-e5c84f7d72a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-e27e0c-6996bc9b75-kmm89\" (UID: \"b78cab03-456c-4ea9-93bb-e5c84f7d72a6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e27e0c-6996bc9b75-kmm89"
Oct 30 07:36:24.390866 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:24.390735    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w4hn\" (UniqueName: \"kubernetes.io/projected/b78cab03-456c-4ea9-93bb-e5c84f7d72a6-kube-api-access-5w4hn\") pod \"payload-pre-processing-e2e-extmod-b-e27e0c-6996bc9b75-kmm89\" (UID: \"b78cab03-456c-4ea9-93bb-e5c84f7d72a6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e27e0c-6996bc9b75-kmm89"
Oct 30 07:36:24.491950 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:24.491859    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w4hn\" (UniqueName: \"kubernetes.io/projected/b78cab03-456c-4ea9-93bb-e5c84f7d72a6-kube-api-access-5w4hn\") pod \"payload-pre-processing-e2e-extmod-b-e27e0c-6996bc9b75-kmm89\" (UID: \"b78cab03-456c-4ea9-93bb-e5c84f7d72a6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e27e0c-6996bc9b75-kmm89"
Oct 30 07:36:24.492123 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:24.492097    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b78cab03-456c-4ea9-93bb-e5c84f7d72a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-e27e0c-6996bc9b75-kmm89\" (UID: \"b78cab03-456c-4ea9-93bb-e5c84f7d72a6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e27e0c-6996bc9b75-kmm89"
Oct 30 07:36:24.492766 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:24.492746    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b78cab03-456c-4ea9-93bb-e5c84f7d72a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-e27e0c-6996bc9b75-kmm89\" (UID: \"b78cab03-456c-4ea9-93bb-e5c84f7d72a6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e27e0c-6996bc9b75-kmm89"
Oct 30 07:36:24.517173 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:24.517148    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w4hn\" (UniqueName: \"kubernetes.io/projected/b78cab03-456c-4ea9-93bb-e5c84f7d72a6-kube-api-access-5w4hn\") pod \"payload-pre-processing-e2e-extmod-b-e27e0c-6996bc9b75-kmm89\" (UID: \"b78cab03-456c-4ea9-93bb-e5c84f7d72a6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e27e0c-6996bc9b75-kmm89"
Oct 30 07:36:24.535123 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:24.535100    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e27e0c-6996bc9b75-kmm89"
Oct 30 07:36:24.838487 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:36:24.838457    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb78cab03_456c_4ea9_93bb_e5c84f7d72a6.slice/crio-07466f76df80f1050880a43313579b63f92e0273da872f6be67c8130e8a31551 WatchSource:0}: Error finding container 07466f76df80f1050880a43313579b63f92e0273da872f6be67c8130e8a31551: Status 404 returned error can't find the container with id 07466f76df80f1050880a43313579b63f92e0273da872f6be67c8130e8a31551
Oct 30 07:36:24.839247 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:24.839222    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e27e0c-6996bc9b75-kmm89"]
Oct 30 07:36:25.439287 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:25.439249    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e27e0c-6996bc9b75-kmm89" event={"ID":"b78cab03-456c-4ea9-93bb-e5c84f7d72a6","Type":"ContainerStarted","Data":"e2903d2883e92218c1ec2b118815e042e99bbf154d027a7e13bea76583c1b221"}
Oct 30 07:36:25.439784 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:25.439312    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e27e0c-6996bc9b75-kmm89" event={"ID":"b78cab03-456c-4ea9-93bb-e5c84f7d72a6","Type":"ContainerStarted","Data":"07466f76df80f1050880a43313579b63f92e0273da872f6be67c8130e8a31551"}
Oct 30 07:36:25.439784 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:25.439339    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e27e0c-6996bc9b75-kmm89"
Oct 30 07:36:25.490644 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:25.490597    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e27e0c-6996bc9b75-kmm89" podStartSLOduration=1.490583534 podStartE2EDuration="1.490583534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:36:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:36:25.488204221 +0000 UTC m=+2344.097611194" watchObservedRunningTime="2026-10-30 07:36:25.490583534 +0000 UTC m=+2344.099990504"
Oct 30 07:36:30.390771 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:30.390737    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ace8fb/payload-processing-external-model-e2e-extmod-b-ace8fb-798c98nqf"]
Oct 30 07:36:31.395923 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:31.395895    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-6b49z5qnn"
Oct 30 07:36:31.860874 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:36:31.860837    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:36:35.391483 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:35.391393    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-ace8fb/payload-processing-external-model-e2e-extmod-b-ace8fb-798c98nqf" podUID="9f8346da-1bd9-4102-ace6-69aa26b9b2f6" containerName="payload-processing" containerID="cri-o://1ea0e7190d71c8921f93c4a645f5018b79e175b358f42947c204387a7ea6fb7a" gracePeriod=25
Oct 30 07:36:35.521559 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:35.521522    6224 generic.go:356] "Generic (PLEG): container finished" podID="9f8346da-1bd9-4102-ace6-69aa26b9b2f6" containerID="1ea0e7190d71c8921f93c4a645f5018b79e175b358f42947c204387a7ea6fb7a" exitCode=0
Oct 30 07:36:35.521702 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:35.521562    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ace8fb/payload-processing-external-model-e2e-extmod-b-ace8fb-798c98nqf" event={"ID":"9f8346da-1bd9-4102-ace6-69aa26b9b2f6","Type":"ContainerDied","Data":"1ea0e7190d71c8921f93c4a645f5018b79e175b358f42947c204387a7ea6fb7a"}
Oct 30 07:36:35.669337 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:35.669314    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ace8fb/payload-processing-external-model-e2e-extmod-b-ace8fb-798c98nqf"
Oct 30 07:36:35.738418 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:35.738386    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f8346da-1bd9-4102-ace6-69aa26b9b2f6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9f8346da-1bd9-4102-ace6-69aa26b9b2f6-provider-credentials\") pod \"9f8346da-1bd9-4102-ace6-69aa26b9b2f6\" (UID: \"9f8346da-1bd9-4102-ace6-69aa26b9b2f6\") "
Oct 30 07:36:35.738575 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:35.738521    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f8346da-1bd9-4102-ace6-69aa26b9b2f6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f8346da-1bd9-4102-ace6-69aa26b9b2f6-plugins-config-volume\") pod \"9f8346da-1bd9-4102-ace6-69aa26b9b2f6\" (UID: \"9f8346da-1bd9-4102-ace6-69aa26b9b2f6\") "
Oct 30 07:36:35.738629 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:35.738617    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f8346da-1bd9-4102-ace6-69aa26b9b2f6-kube-api-access-d9kmw\" (UniqueName: \"kubernetes.io/projected/9f8346da-1bd9-4102-ace6-69aa26b9b2f6-kube-api-access-d9kmw\") pod \"9f8346da-1bd9-4102-ace6-69aa26b9b2f6\" (UID: \"9f8346da-1bd9-4102-ace6-69aa26b9b2f6\") "
Oct 30 07:36:35.738678 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:35.738655    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f8346da-1bd9-4102-ace6-69aa26b9b2f6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9f8346da-1bd9-4102-ace6-69aa26b9b2f6-routing-overlay-volume\") pod \"9f8346da-1bd9-4102-ace6-69aa26b9b2f6\" (UID: \"9f8346da-1bd9-4102-ace6-69aa26b9b2f6\") "
Oct 30 07:36:35.739060 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:35.739032    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f8346da-1bd9-4102-ace6-69aa26b9b2f6-plugins-config-volume" pod "9f8346da-1bd9-4102-ace6-69aa26b9b2f6" (UID: "9f8346da-1bd9-4102-ace6-69aa26b9b2f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:36:35.739192 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:35.739106    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f8346da-1bd9-4102-ace6-69aa26b9b2f6-routing-overlay-volume" pod "9f8346da-1bd9-4102-ace6-69aa26b9b2f6" (UID: "9f8346da-1bd9-4102-ace6-69aa26b9b2f6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:36:35.739431 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:35.739409    6224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f8346da-1bd9-4102-ace6-69aa26b9b2f6-plugins-config-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:36:35.739520 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:35.739438    6224 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9f8346da-1bd9-4102-ace6-69aa26b9b2f6-routing-overlay-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:36:35.740822 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:35.740785    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f8346da-1bd9-4102-ace6-69aa26b9b2f6-provider-credentials" pod "9f8346da-1bd9-4102-ace6-69aa26b9b2f6" (UID: "9f8346da-1bd9-4102-ace6-69aa26b9b2f6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:36:35.741300 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:35.741281    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f8346da-1bd9-4102-ace6-69aa26b9b2f6-kube-api-access-d9kmw" pod "9f8346da-1bd9-4102-ace6-69aa26b9b2f6" (UID: "9f8346da-1bd9-4102-ace6-69aa26b9b2f6"). InnerVolumeSpecName "kube-api-access-d9kmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:36:35.840439 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:35.840415    6224 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9f8346da-1bd9-4102-ace6-69aa26b9b2f6-provider-credentials\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:36:35.840439 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:35.840441    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9kmw\" (UniqueName: \"kubernetes.io/projected/9f8346da-1bd9-4102-ace6-69aa26b9b2f6-kube-api-access-d9kmw\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:36:36.449752 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:36.449724    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e27e0c-6996bc9b75-kmm89"
Oct 30 07:36:36.532921 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:36.532891    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ace8fb/payload-processing-external-model-e2e-extmod-b-ace8fb-798c98nqf"
Oct 30 07:36:36.533144 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:36.532913    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ace8fb/payload-processing-external-model-e2e-extmod-b-ace8fb-798c98nqf" event={"ID":"9f8346da-1bd9-4102-ace6-69aa26b9b2f6","Type":"ContainerDied","Data":"94cf023f8a1aa96b496fb5f801b1f28f4906ba393efdd3297da53ea26a7f1be9"}
Oct 30 07:36:36.533144 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:36.532980    6224 scope.go:122] "RemoveContainer" containerID="1ea0e7190d71c8921f93c4a645f5018b79e175b358f42947c204387a7ea6fb7a"
Oct 30 07:36:36.594279 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:36.594253    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ace8fb/payload-processing-external-model-e2e-extmod-b-ace8fb-798c98nqf"]
Oct 30 07:36:36.597589 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:36.597564    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-ace8fb/payload-processing-external-model-e2e-extmod-b-ace8fb-798c98nqf"]
Oct 30 07:36:37.504597 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:37.504560    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f8346da-1bd9-4102-ace6-69aa26b9b2f6" path="/var/lib/kubelet/pods/9f8346da-1bd9-4102-ace6-69aa26b9b2f6/volumes"
Oct 30 07:36:46.666116 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:46.666086    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-586dxf8bw"]
Oct 30 07:36:46.666760 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:46.666746    6224 cpu_manager.go:414] "containerMap: removing container" podUID="9f8346da-1bd9-4102-ace6-69aa26b9b2f6" containerName="payload-processing"
Oct 30 07:36:46.666836 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:46.666760    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f8346da-1bd9-4102-ace6-69aa26b9b2f6" containerName="payload-processing"
Oct 30 07:36:46.666929 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:46.666919    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f8346da-1bd9-4102-ace6-69aa26b9b2f6" containerName="payload-processing"
Oct 30 07:36:46.671728 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:46.671710    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-586dxf8bw"
Oct 30 07:36:46.689363 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:46.689337    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-0f5081\"/\"e2e-cmp-legacy-d9897e-provider-cred\""
Oct 30 07:36:46.692209 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:46.692186    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-586dxf8bw"]
Oct 30 07:36:46.702132 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:46.702112    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0f5081\"/\"payload-processing-external-model-plugins-e2e-extmod-a-0f5081\""
Oct 30 07:36:46.702743 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:46.702721    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0f5081\"/\"openshift-service-ca.crt\""
Oct 30 07:36:46.702743 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:46.702733    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0f5081\"/\"kube-root-ca.crt\""
Oct 30 07:36:46.703344 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:46.703319    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0f5081\"/\"routing-overlay\""
Oct 30 07:36:46.754643 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:46.754619    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b88b16e6-cb7c-492a-be7c-a028acb1bf19-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0f5081-586dxf8bw\" (UID: \"b88b16e6-cb7c-492a-be7c-a028acb1bf19\") " pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-586dxf8bw"
Oct 30 07:36:46.754838 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:46.754798    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b88b16e6-cb7c-492a-be7c-a028acb1bf19-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0f5081-586dxf8bw\" (UID: \"b88b16e6-cb7c-492a-be7c-a028acb1bf19\") " pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-586dxf8bw"
Oct 30 07:36:46.754907 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:46.754860    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9l4d\" (UniqueName: \"kubernetes.io/projected/b88b16e6-cb7c-492a-be7c-a028acb1bf19-kube-api-access-v9l4d\") pod \"payload-processing-external-model-e2e-extmod-a-0f5081-586dxf8bw\" (UID: \"b88b16e6-cb7c-492a-be7c-a028acb1bf19\") " pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-586dxf8bw"
Oct 30 07:36:46.754950 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:46.754921    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b88b16e6-cb7c-492a-be7c-a028acb1bf19-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0f5081-586dxf8bw\" (UID: \"b88b16e6-cb7c-492a-be7c-a028acb1bf19\") " pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-586dxf8bw"
Oct 30 07:36:46.790236 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:46.790214    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-0f5081\"/\"payload-processing-external-model-e2e--8bb4c319-dockercfg-zw6ng\""
Oct 30 07:36:46.856362 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:46.856334    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b88b16e6-cb7c-492a-be7c-a028acb1bf19-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0f5081-586dxf8bw\" (UID: \"b88b16e6-cb7c-492a-be7c-a028acb1bf19\") " pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-586dxf8bw"
Oct 30 07:36:46.856491 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:46.856476    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b88b16e6-cb7c-492a-be7c-a028acb1bf19-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0f5081-586dxf8bw\" (UID: \"b88b16e6-cb7c-492a-be7c-a028acb1bf19\") " pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-586dxf8bw"
Oct 30 07:36:46.856538 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:46.856508    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9l4d\" (UniqueName: \"kubernetes.io/projected/b88b16e6-cb7c-492a-be7c-a028acb1bf19-kube-api-access-v9l4d\") pod \"payload-processing-external-model-e2e-extmod-a-0f5081-586dxf8bw\" (UID: \"b88b16e6-cb7c-492a-be7c-a028acb1bf19\") " pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-586dxf8bw"
Oct 30 07:36:46.856538 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:46.856527    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b88b16e6-cb7c-492a-be7c-a028acb1bf19-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0f5081-586dxf8bw\" (UID: \"b88b16e6-cb7c-492a-be7c-a028acb1bf19\") " pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-586dxf8bw"
Oct 30 07:36:46.857357 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:46.857333    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b88b16e6-cb7c-492a-be7c-a028acb1bf19-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0f5081-586dxf8bw\" (UID: \"b88b16e6-cb7c-492a-be7c-a028acb1bf19\") " pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-586dxf8bw"
Oct 30 07:36:46.857448 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:46.857393    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b88b16e6-cb7c-492a-be7c-a028acb1bf19-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0f5081-586dxf8bw\" (UID: \"b88b16e6-cb7c-492a-be7c-a028acb1bf19\") " pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-586dxf8bw"
Oct 30 07:36:46.858937 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:46.858918    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b88b16e6-cb7c-492a-be7c-a028acb1bf19-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0f5081-586dxf8bw\" (UID: \"b88b16e6-cb7c-492a-be7c-a028acb1bf19\") " pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-586dxf8bw"
Oct 30 07:36:46.881913 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:46.881891    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9l4d\" (UniqueName: \"kubernetes.io/projected/b88b16e6-cb7c-492a-be7c-a028acb1bf19-kube-api-access-v9l4d\") pod \"payload-processing-external-model-e2e-extmod-a-0f5081-586dxf8bw\" (UID: \"b88b16e6-cb7c-492a-be7c-a028acb1bf19\") " pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-586dxf8bw"
Oct 30 07:36:46.986878 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:46.985842    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-586dxf8bw"
Oct 30 07:36:47.341030 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:47.341004    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-586dxf8bw"]
Oct 30 07:36:47.342999 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:36:47.342966    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb88b16e6_cb7c_492a_be7c_a028acb1bf19.slice/crio-9fc9523cc8e9b41a5499fb5713327388bd0d1cbb31acaa6861841b0a1c31dc6e WatchSource:0}: Error finding container 9fc9523cc8e9b41a5499fb5713327388bd0d1cbb31acaa6861841b0a1c31dc6e: Status 404 returned error can't find the container with id 9fc9523cc8e9b41a5499fb5713327388bd0d1cbb31acaa6861841b0a1c31dc6e
Oct 30 07:36:47.653501 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:47.653400    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-586dxf8bw" event={"ID":"b88b16e6-cb7c-492a-be7c-a028acb1bf19","Type":"ContainerStarted","Data":"d30c091386108faa662544aea5670ecc03e0f91bb68796ce1880329912bab484"}
Oct 30 07:36:47.653501 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:47.653463    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-586dxf8bw" event={"ID":"b88b16e6-cb7c-492a-be7c-a028acb1bf19","Type":"ContainerStarted","Data":"9fc9523cc8e9b41a5499fb5713327388bd0d1cbb31acaa6861841b0a1c31dc6e"}
Oct 30 07:36:47.653501 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:47.653493    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-586dxf8bw"
Oct 30 07:36:47.700680 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:47.700628    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-586dxf8bw" podStartSLOduration=1.700613933 podStartE2EDuration="1.700613933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:36:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:36:47.698103057 +0000 UTC m=+2366.307510027" watchObservedRunningTime="2026-10-30 07:36:47.700613933 +0000 UTC m=+2366.310020903"
Oct 30 07:36:58.675290 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:36:58.675253    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-586dxf8bw"
Oct 30 07:37:01.564226 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:01.564181    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-6b49z5qnn"]
Oct 30 07:37:01.859887 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:37:01.859753    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:37:06.505690 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:06.505654    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:37:06.565306 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:06.565263    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-6b49z5qnn" podUID="35f3e736-96db-420c-8baf-880e3c7ca7ed" containerName="payload-processing" containerID="cri-o://75826de9c2ec1150e98130206f759c09eda085d833a01378cc388c941a95c71f" gracePeriod=25
Oct 30 07:37:06.823095 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:06.823063    6224 generic.go:356] "Generic (PLEG): container finished" podID="35f3e736-96db-420c-8baf-880e3c7ca7ed" containerID="75826de9c2ec1150e98130206f759c09eda085d833a01378cc388c941a95c71f" exitCode=0
Oct 30 07:37:06.823256 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:06.823137    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-6b49z5qnn" event={"ID":"35f3e736-96db-420c-8baf-880e3c7ca7ed","Type":"ContainerDied","Data":"75826de9c2ec1150e98130206f759c09eda085d833a01378cc388c941a95c71f"}
Oct 30 07:37:06.848181 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:06.848154    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-6b49z5qnn"
Oct 30 07:37:06.939046 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:06.939008    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35f3e736-96db-420c-8baf-880e3c7ca7ed-kube-api-access-lxv4p\" (UniqueName: \"kubernetes.io/projected/35f3e736-96db-420c-8baf-880e3c7ca7ed-kube-api-access-lxv4p\") pod \"35f3e736-96db-420c-8baf-880e3c7ca7ed\" (UID: \"35f3e736-96db-420c-8baf-880e3c7ca7ed\") "
Oct 30 07:37:06.939224 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:06.939084    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35f3e736-96db-420c-8baf-880e3c7ca7ed-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/35f3e736-96db-420c-8baf-880e3c7ca7ed-routing-overlay-volume\") pod \"35f3e736-96db-420c-8baf-880e3c7ca7ed\" (UID: \"35f3e736-96db-420c-8baf-880e3c7ca7ed\") "
Oct 30 07:37:06.939224 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:06.939160    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35f3e736-96db-420c-8baf-880e3c7ca7ed-provider-credentials\" (UniqueName: \"kubernetes.io/projected/35f3e736-96db-420c-8baf-880e3c7ca7ed-provider-credentials\") pod \"35f3e736-96db-420c-8baf-880e3c7ca7ed\" (UID: \"35f3e736-96db-420c-8baf-880e3c7ca7ed\") "
Oct 30 07:37:06.939326 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:06.939312    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35f3e736-96db-420c-8baf-880e3c7ca7ed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35f3e736-96db-420c-8baf-880e3c7ca7ed-plugins-config-volume\") pod \"35f3e736-96db-420c-8baf-880e3c7ca7ed\" (UID: \"35f3e736-96db-420c-8baf-880e3c7ca7ed\") "
Oct 30 07:37:06.939586 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:06.939555    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35f3e736-96db-420c-8baf-880e3c7ca7ed-routing-overlay-volume" pod "35f3e736-96db-420c-8baf-880e3c7ca7ed" (UID: "35f3e736-96db-420c-8baf-880e3c7ca7ed"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:37:06.939725 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:06.939707    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35f3e736-96db-420c-8baf-880e3c7ca7ed-plugins-config-volume" pod "35f3e736-96db-420c-8baf-880e3c7ca7ed" (UID: "35f3e736-96db-420c-8baf-880e3c7ca7ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:37:06.939780 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:06.939756    6224 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/35f3e736-96db-420c-8baf-880e3c7ca7ed-routing-overlay-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:37:06.941405 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:06.941382    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35f3e736-96db-420c-8baf-880e3c7ca7ed-kube-api-access-lxv4p" pod "35f3e736-96db-420c-8baf-880e3c7ca7ed" (UID: "35f3e736-96db-420c-8baf-880e3c7ca7ed"). InnerVolumeSpecName "kube-api-access-lxv4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:37:06.941520 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:06.941479    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35f3e736-96db-420c-8baf-880e3c7ca7ed-provider-credentials" pod "35f3e736-96db-420c-8baf-880e3c7ca7ed" (UID: "35f3e736-96db-420c-8baf-880e3c7ca7ed"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:37:07.055715 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:07.049205    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxv4p\" (UniqueName: \"kubernetes.io/projected/35f3e736-96db-420c-8baf-880e3c7ca7ed-kube-api-access-lxv4p\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:37:07.055715 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:07.049244    6224 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/35f3e736-96db-420c-8baf-880e3c7ca7ed-provider-credentials\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:37:07.055715 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:07.049260    6224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35f3e736-96db-420c-8baf-880e3c7ca7ed-plugins-config-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:37:07.832662 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:07.832576    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-6b49z5qnn"
Oct 30 07:37:07.832662 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:07.832607    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-6b49z5qnn" event={"ID":"35f3e736-96db-420c-8baf-880e3c7ca7ed","Type":"ContainerDied","Data":"d20e5d0d7da72da5a0180eff334c7c2bd43a02c2e8d6bb1e8b77984b3e1e2f44"}
Oct 30 07:37:07.833246 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:07.832670    6224 scope.go:122] "RemoveContainer" containerID="75826de9c2ec1150e98130206f759c09eda085d833a01378cc388c941a95c71f"
Oct 30 07:37:07.891227 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:07.891193    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-6b49z5qnn"]
Oct 30 07:37:07.905231 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:07.905199    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-6b49z5qnn"]
Oct 30 07:37:09.506919 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:09.506868    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35f3e736-96db-420c-8baf-880e3c7ca7ed" path="/var/lib/kubelet/pods/35f3e736-96db-420c-8baf-880e3c7ca7ed/volumes"
Oct 30 07:37:24.886883 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:24.886853    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-77f56l8gh"]
Oct 30 07:37:29.508620 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:29.508586    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-55c9vn487"]
Oct 30 07:37:29.887535 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:29.887466    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-77f56l8gh" podUID="cf51f674-5137-4dc3-a4a2-2914526f9d9a" containerName="payload-processing" containerID="cri-o://f0a6e0bb291d415c1dd7641b515a2b5d81cbf34af780eea8efef71d04c7d78c5" gracePeriod=25
Oct 30 07:37:30.157684 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:30.157627    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-77f56l8gh"
Oct 30 07:37:30.259006 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:30.258977    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf51f674-5137-4dc3-a4a2-2914526f9d9a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf51f674-5137-4dc3-a4a2-2914526f9d9a-plugins-config-volume\") pod \"cf51f674-5137-4dc3-a4a2-2914526f9d9a\" (UID: \"cf51f674-5137-4dc3-a4a2-2914526f9d9a\") "
Oct 30 07:37:30.259173 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:30.259045    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf51f674-5137-4dc3-a4a2-2914526f9d9a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/cf51f674-5137-4dc3-a4a2-2914526f9d9a-provider-credentials\") pod \"cf51f674-5137-4dc3-a4a2-2914526f9d9a\" (UID: \"cf51f674-5137-4dc3-a4a2-2914526f9d9a\") "
Oct 30 07:37:30.259173 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:30.259128    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf51f674-5137-4dc3-a4a2-2914526f9d9a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cf51f674-5137-4dc3-a4a2-2914526f9d9a-routing-overlay-volume\") pod \"cf51f674-5137-4dc3-a4a2-2914526f9d9a\" (UID: \"cf51f674-5137-4dc3-a4a2-2914526f9d9a\") "
Oct 30 07:37:30.259264 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:30.259187    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf51f674-5137-4dc3-a4a2-2914526f9d9a-kube-api-access-82wgt\" (UniqueName: \"kubernetes.io/projected/cf51f674-5137-4dc3-a4a2-2914526f9d9a-kube-api-access-82wgt\") pod \"cf51f674-5137-4dc3-a4a2-2914526f9d9a\" (UID: \"cf51f674-5137-4dc3-a4a2-2914526f9d9a\") "
Oct 30 07:37:30.259479 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:30.259448    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf51f674-5137-4dc3-a4a2-2914526f9d9a-plugins-config-volume" pod "cf51f674-5137-4dc3-a4a2-2914526f9d9a" (UID: "cf51f674-5137-4dc3-a4a2-2914526f9d9a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:37:30.259616 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:30.259479    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf51f674-5137-4dc3-a4a2-2914526f9d9a-routing-overlay-volume" pod "cf51f674-5137-4dc3-a4a2-2914526f9d9a" (UID: "cf51f674-5137-4dc3-a4a2-2914526f9d9a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:37:30.259616 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:30.259534    6224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf51f674-5137-4dc3-a4a2-2914526f9d9a-plugins-config-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:37:30.261350 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:30.261329    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf51f674-5137-4dc3-a4a2-2914526f9d9a-provider-credentials" pod "cf51f674-5137-4dc3-a4a2-2914526f9d9a" (UID: "cf51f674-5137-4dc3-a4a2-2914526f9d9a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:37:30.261463 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:30.261366    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf51f674-5137-4dc3-a4a2-2914526f9d9a-kube-api-access-82wgt" pod "cf51f674-5137-4dc3-a4a2-2914526f9d9a" (UID: "cf51f674-5137-4dc3-a4a2-2914526f9d9a"). InnerVolumeSpecName "kube-api-access-82wgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:37:30.360971 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:30.360941    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82wgt\" (UniqueName: \"kubernetes.io/projected/cf51f674-5137-4dc3-a4a2-2914526f9d9a-kube-api-access-82wgt\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:37:30.360971 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:30.360968    6224 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cf51f674-5137-4dc3-a4a2-2914526f9d9a-provider-credentials\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:37:30.361176 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:30.360978    6224 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cf51f674-5137-4dc3-a4a2-2914526f9d9a-routing-overlay-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:37:31.000313 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:31.000274    6224 generic.go:356] "Generic (PLEG): container finished" podID="cf51f674-5137-4dc3-a4a2-2914526f9d9a" containerID="f0a6e0bb291d415c1dd7641b515a2b5d81cbf34af780eea8efef71d04c7d78c5" exitCode=0
Oct 30 07:37:31.000753 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:31.000339    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-77f56l8gh"
Oct 30 07:37:31.000753 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:31.000361    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-77f56l8gh" event={"ID":"cf51f674-5137-4dc3-a4a2-2914526f9d9a","Type":"ContainerDied","Data":"f0a6e0bb291d415c1dd7641b515a2b5d81cbf34af780eea8efef71d04c7d78c5"}
Oct 30 07:37:31.000753 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:31.000442    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-77f56l8gh" event={"ID":"cf51f674-5137-4dc3-a4a2-2914526f9d9a","Type":"ContainerDied","Data":"c48ce44b549f884374288fe8a8ce5ff4a2af4e9da003d2de69471ff794fe3bce"}
Oct 30 07:37:31.000753 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:31.000464    6224 scope.go:122] "RemoveContainer" containerID="f0a6e0bb291d415c1dd7641b515a2b5d81cbf34af780eea8efef71d04c7d78c5"
Oct 30 07:37:31.011588 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:31.011570    6224 scope.go:122] "RemoveContainer" containerID="f0a6e0bb291d415c1dd7641b515a2b5d81cbf34af780eea8efef71d04c7d78c5"
Oct 30 07:37:31.011860 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:37:31.011842    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0a6e0bb291d415c1dd7641b515a2b5d81cbf34af780eea8efef71d04c7d78c5\": container with ID starting with f0a6e0bb291d415c1dd7641b515a2b5d81cbf34af780eea8efef71d04c7d78c5 not found: ID does not exist" containerID="f0a6e0bb291d415c1dd7641b515a2b5d81cbf34af780eea8efef71d04c7d78c5"
Oct 30 07:37:31.011915 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:31.011868    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0a6e0bb291d415c1dd7641b515a2b5d81cbf34af780eea8efef71d04c7d78c5"} err="failed to get container status \"f0a6e0bb291d415c1dd7641b515a2b5d81cbf34af780eea8efef71d04c7d78c5\": rpc error: code = NotFound desc = could not find container \"f0a6e0bb291d415c1dd7641b515a2b5d81cbf34af780eea8efef71d04c7d78c5\": container with ID starting with f0a6e0bb291d415c1dd7641b515a2b5d81cbf34af780eea8efef71d04c7d78c5 not found: ID does not exist"
Oct 30 07:37:31.058364 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:31.058335    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-77f56l8gh"]
Oct 30 07:37:31.072144 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:31.072116    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-77f56l8gh"]
Oct 30 07:37:31.505046 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:31.505009    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf51f674-5137-4dc3-a4a2-2914526f9d9a" path="/var/lib/kubelet/pods/cf51f674-5137-4dc3-a4a2-2914526f9d9a/volumes"
Oct 30 07:37:31.857405 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:37:31.857371    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:37:34.509762 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:34.509720    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-55c9vn487" podUID="1cc06823-94c3-484d-bdcd-89e44e945500" containerName="payload-processing" containerID="cri-o://1435c033837bd89a781a9a748af833eb1f3792b42027ebe2b086b9214151d065" gracePeriod=25
Oct 30 07:37:34.791504 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:34.791481    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-55c9vn487"
Oct 30 07:37:34.916616 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:34.916573    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cc06823-94c3-484d-bdcd-89e44e945500-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1cc06823-94c3-484d-bdcd-89e44e945500-provider-credentials\") pod \"1cc06823-94c3-484d-bdcd-89e44e945500\" (UID: \"1cc06823-94c3-484d-bdcd-89e44e945500\") "
Oct 30 07:37:34.916788 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:34.916636    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cc06823-94c3-484d-bdcd-89e44e945500-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1cc06823-94c3-484d-bdcd-89e44e945500-routing-overlay-volume\") pod \"1cc06823-94c3-484d-bdcd-89e44e945500\" (UID: \"1cc06823-94c3-484d-bdcd-89e44e945500\") "
Oct 30 07:37:34.916788 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:34.916655    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cc06823-94c3-484d-bdcd-89e44e945500-kube-api-access-548jr\" (UniqueName: \"kubernetes.io/projected/1cc06823-94c3-484d-bdcd-89e44e945500-kube-api-access-548jr\") pod \"1cc06823-94c3-484d-bdcd-89e44e945500\" (UID: \"1cc06823-94c3-484d-bdcd-89e44e945500\") "
Oct 30 07:37:34.916788 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:34.916677    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cc06823-94c3-484d-bdcd-89e44e945500-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cc06823-94c3-484d-bdcd-89e44e945500-plugins-config-volume\") pod \"1cc06823-94c3-484d-bdcd-89e44e945500\" (UID: \"1cc06823-94c3-484d-bdcd-89e44e945500\") "
Oct 30 07:37:34.917158 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:34.917110    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cc06823-94c3-484d-bdcd-89e44e945500-routing-overlay-volume" pod "1cc06823-94c3-484d-bdcd-89e44e945500" (UID: "1cc06823-94c3-484d-bdcd-89e44e945500"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:37:34.917279 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:34.917173    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cc06823-94c3-484d-bdcd-89e44e945500-plugins-config-volume" pod "1cc06823-94c3-484d-bdcd-89e44e945500" (UID: "1cc06823-94c3-484d-bdcd-89e44e945500"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:37:34.917415 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:34.917396    6224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cc06823-94c3-484d-bdcd-89e44e945500-plugins-config-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:37:34.917459 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:34.917418    6224 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1cc06823-94c3-484d-bdcd-89e44e945500-routing-overlay-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:37:34.919000 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:34.918978    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cc06823-94c3-484d-bdcd-89e44e945500-kube-api-access-548jr" pod "1cc06823-94c3-484d-bdcd-89e44e945500" (UID: "1cc06823-94c3-484d-bdcd-89e44e945500"). InnerVolumeSpecName "kube-api-access-548jr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:37:34.919085 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:34.919011    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cc06823-94c3-484d-bdcd-89e44e945500-provider-credentials" pod "1cc06823-94c3-484d-bdcd-89e44e945500" (UID: "1cc06823-94c3-484d-bdcd-89e44e945500"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:37:35.019108 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:35.019047    6224 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1cc06823-94c3-484d-bdcd-89e44e945500-provider-credentials\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:37:35.019108 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:35.019072    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-548jr\" (UniqueName: \"kubernetes.io/projected/1cc06823-94c3-484d-bdcd-89e44e945500-kube-api-access-548jr\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:37:35.031425 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:35.031395    6224 generic.go:356] "Generic (PLEG): container finished" podID="1cc06823-94c3-484d-bdcd-89e44e945500" containerID="1435c033837bd89a781a9a748af833eb1f3792b42027ebe2b086b9214151d065" exitCode=0
Oct 30 07:37:35.031536 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:35.031438    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-55c9vn487"
Oct 30 07:37:35.031536 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:35.031480    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-55c9vn487" event={"ID":"1cc06823-94c3-484d-bdcd-89e44e945500","Type":"ContainerDied","Data":"1435c033837bd89a781a9a748af833eb1f3792b42027ebe2b086b9214151d065"}
Oct 30 07:37:35.031627 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:35.031536    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-55c9vn487" event={"ID":"1cc06823-94c3-484d-bdcd-89e44e945500","Type":"ContainerDied","Data":"8dc2a8294722e3a05b099fcc942fb724ea097d7938f9b299719eaa567dc60245"}
Oct 30 07:37:35.031627 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:35.031563    6224 scope.go:122] "RemoveContainer" containerID="1435c033837bd89a781a9a748af833eb1f3792b42027ebe2b086b9214151d065"
Oct 30 07:37:35.042141 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:35.042124    6224 scope.go:122] "RemoveContainer" containerID="1435c033837bd89a781a9a748af833eb1f3792b42027ebe2b086b9214151d065"
Oct 30 07:37:35.042388 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:37:35.042368    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1435c033837bd89a781a9a748af833eb1f3792b42027ebe2b086b9214151d065\": container with ID starting with 1435c033837bd89a781a9a748af833eb1f3792b42027ebe2b086b9214151d065 not found: ID does not exist" containerID="1435c033837bd89a781a9a748af833eb1f3792b42027ebe2b086b9214151d065"
Oct 30 07:37:35.042450 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:35.042395    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1435c033837bd89a781a9a748af833eb1f3792b42027ebe2b086b9214151d065"} err="failed to get container status \"1435c033837bd89a781a9a748af833eb1f3792b42027ebe2b086b9214151d065\": rpc error: code = NotFound desc = could not find container \"1435c033837bd89a781a9a748af833eb1f3792b42027ebe2b086b9214151d065\": container with ID starting with 1435c033837bd89a781a9a748af833eb1f3792b42027ebe2b086b9214151d065 not found: ID does not exist"
Oct 30 07:37:35.090118 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:35.090094    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-55c9vn487"]
Oct 30 07:37:35.095914 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:35.095893    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-55c9vn487"]
Oct 30 07:37:35.507869 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:35.507834    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cc06823-94c3-484d-bdcd-89e44e945500" path="/var/lib/kubelet/pods/1cc06823-94c3-484d-bdcd-89e44e945500/volumes"
Oct 30 07:37:45.114675 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:45.114639    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-786dq29q8"]
Oct 30 07:37:45.115287 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:45.115273    6224 cpu_manager.go:414] "containerMap: removing container" podUID="cf51f674-5137-4dc3-a4a2-2914526f9d9a" containerName="payload-processing"
Oct 30 07:37:45.115336 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:45.115288    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf51f674-5137-4dc3-a4a2-2914526f9d9a" containerName="payload-processing"
Oct 30 07:37:45.115336 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:45.115322    6224 cpu_manager.go:414] "containerMap: removing container" podUID="1cc06823-94c3-484d-bdcd-89e44e945500" containerName="payload-processing"
Oct 30 07:37:45.115336 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:45.115327    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cc06823-94c3-484d-bdcd-89e44e945500" containerName="payload-processing"
Oct 30 07:37:45.115429 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:45.115342    6224 cpu_manager.go:414] "containerMap: removing container" podUID="35f3e736-96db-420c-8baf-880e3c7ca7ed" containerName="payload-processing"
Oct 30 07:37:45.115429 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:45.115347    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35f3e736-96db-420c-8baf-880e3c7ca7ed" containerName="payload-processing"
Oct 30 07:37:45.115496 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:45.115442    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cc06823-94c3-484d-bdcd-89e44e945500" containerName="payload-processing"
Oct 30 07:37:45.115496 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:45.115452    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="35f3e736-96db-420c-8baf-880e3c7ca7ed" containerName="payload-processing"
Oct 30 07:37:45.115496 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:45.115460    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf51f674-5137-4dc3-a4a2-2914526f9d9a" containerName="payload-processing"
Oct 30 07:37:45.120552 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:45.120533    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-786dq29q8"
Oct 30 07:37:45.140347 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:45.140323    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-786dq29q8"]
Oct 30 07:37:45.150802 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:45.150773    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-f1f700\"/\"agc-cred-2c0ab0fb80d7\""
Oct 30 07:37:45.150943 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:45.150783    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-f1f700\"/\"agc-cred-00a7322d0aa7\""
Oct 30 07:37:45.150943 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:45.150784    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-f1f700\"/\"agc-cred-c0ebddc52a78\""
Oct 30 07:37:45.151072 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:45.151051    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f1f700\"/\"routing-overlay\""
Oct 30 07:37:45.151129 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:45.151113    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-f1f700\"/\"payload-processing-external-model-e2e--96fc5bc0-dockercfg-tlgdn\""
Oct 30 07:37:45.153764 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:45.153723    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f1f700\"/\"payload-processing-external-model-plugins-e2e-extmod-b-f1f700\""
Oct 30 07:37:45.153764 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:45.153736    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f1f700\"/\"kube-root-ca.crt\""
Oct 30 07:37:45.153764 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:45.153752    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f1f700\"/\"openshift-service-ca.crt\""
Oct 30 07:37:45.282334 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:45.282303    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fafd3773-189e-4e9f-b4fa-c6bababa9a83-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f1f700-786dq29q8\" (UID: \"fafd3773-189e-4e9f-b4fa-c6bababa9a83\") " pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-786dq29q8"
Oct 30 07:37:45.282492 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:45.282444    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xgqb\" (UniqueName: \"kubernetes.io/projected/fafd3773-189e-4e9f-b4fa-c6bababa9a83-kube-api-access-9xgqb\") pod \"payload-processing-external-model-e2e-extmod-b-f1f700-786dq29q8\" (UID: \"fafd3773-189e-4e9f-b4fa-c6bababa9a83\") " pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-786dq29q8"
Oct 30 07:37:45.282596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:45.282577    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fafd3773-189e-4e9f-b4fa-c6bababa9a83-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1f700-786dq29q8\" (UID: \"fafd3773-189e-4e9f-b4fa-c6bababa9a83\") " pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-786dq29q8"
Oct 30 07:37:45.282638 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:45.282610    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fafd3773-189e-4e9f-b4fa-c6bababa9a83-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1f700-786dq29q8\" (UID: \"fafd3773-189e-4e9f-b4fa-c6bababa9a83\") " pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-786dq29q8"
Oct 30 07:37:45.383647 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:45.383568    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fafd3773-189e-4e9f-b4fa-c6bababa9a83-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1f700-786dq29q8\" (UID: \"fafd3773-189e-4e9f-b4fa-c6bababa9a83\") " pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-786dq29q8"
Oct 30 07:37:45.383647 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:45.383606    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fafd3773-189e-4e9f-b4fa-c6bababa9a83-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1f700-786dq29q8\" (UID: \"fafd3773-189e-4e9f-b4fa-c6bababa9a83\") " pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-786dq29q8"
Oct 30 07:37:45.383877 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:45.383656    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fafd3773-189e-4e9f-b4fa-c6bababa9a83-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f1f700-786dq29q8\" (UID: \"fafd3773-189e-4e9f-b4fa-c6bababa9a83\") " pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-786dq29q8"
Oct 30 07:37:45.383877 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:45.383741    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xgqb\" (UniqueName: \"kubernetes.io/projected/fafd3773-189e-4e9f-b4fa-c6bababa9a83-kube-api-access-9xgqb\") pod \"payload-processing-external-model-e2e-extmod-b-f1f700-786dq29q8\" (UID: \"fafd3773-189e-4e9f-b4fa-c6bababa9a83\") " pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-786dq29q8"
Oct 30 07:37:45.384333 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:45.384307    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fafd3773-189e-4e9f-b4fa-c6bababa9a83-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1f700-786dq29q8\" (UID: \"fafd3773-189e-4e9f-b4fa-c6bababa9a83\") " pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-786dq29q8"
Oct 30 07:37:45.384390 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:45.384359    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fafd3773-189e-4e9f-b4fa-c6bababa9a83-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1f700-786dq29q8\" (UID: \"fafd3773-189e-4e9f-b4fa-c6bababa9a83\") " pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-786dq29q8"
Oct 30 07:37:45.386439 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:45.386413    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fafd3773-189e-4e9f-b4fa-c6bababa9a83-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f1f700-786dq29q8\" (UID: \"fafd3773-189e-4e9f-b4fa-c6bababa9a83\") " pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-786dq29q8"
Oct 30 07:37:45.409219 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:45.409195    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xgqb\" (UniqueName: \"kubernetes.io/projected/fafd3773-189e-4e9f-b4fa-c6bababa9a83-kube-api-access-9xgqb\") pod \"payload-processing-external-model-e2e-extmod-b-f1f700-786dq29q8\" (UID: \"fafd3773-189e-4e9f-b4fa-c6bababa9a83\") " pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-786dq29q8"
Oct 30 07:37:45.435062 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:45.435036    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-786dq29q8"
Oct 30 07:37:45.758064 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:45.758033    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-786dq29q8"]
Oct 30 07:37:45.758197 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:37:45.758152    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfafd3773_189e_4e9f_b4fa_c6bababa9a83.slice/crio-e1c483f93dfb22b33e91c5cc9435ff3b150256dd30e2fcb62fb556ab19d1e29f WatchSource:0}: Error finding container e1c483f93dfb22b33e91c5cc9435ff3b150256dd30e2fcb62fb556ab19d1e29f: Status 404 returned error can't find the container with id e1c483f93dfb22b33e91c5cc9435ff3b150256dd30e2fcb62fb556ab19d1e29f
Oct 30 07:37:46.109004 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:46.108964    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-786dq29q8" event={"ID":"fafd3773-189e-4e9f-b4fa-c6bababa9a83","Type":"ContainerStarted","Data":"b87f9821901f630d0d17ea7651b6733e0534d064c60014725b83a383b1cc1bc7"}
Oct 30 07:37:46.109183 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:46.109037    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-786dq29q8"
Oct 30 07:37:46.109183 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:46.109060    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-786dq29q8" event={"ID":"fafd3773-189e-4e9f-b4fa-c6bababa9a83","Type":"ContainerStarted","Data":"e1c483f93dfb22b33e91c5cc9435ff3b150256dd30e2fcb62fb556ab19d1e29f"}
Oct 30 07:37:46.158008 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:46.157962    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-786dq29q8" podStartSLOduration=1.15794877 podStartE2EDuration="1.15794877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:37:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:37:46.154705951 +0000 UTC m=+2424.764112922" watchObservedRunningTime="2026-10-30 07:37:46.15794877 +0000 UTC m=+2424.767355739"
Oct 30 07:37:46.483386 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:46.483304    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-8e7948/payload-processing-external-model-e2e-extmod-a-8e7948-5dd8xq2fh"]
Oct 30 07:37:46.487771 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:46.487749    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8e7948/payload-processing-external-model-e2e-extmod-a-8e7948-5dd8xq2fh"
Oct 30 07:37:46.519340 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:46.519311    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8e7948\"/\"openshift-service-ca.crt\""
Oct 30 07:37:46.519340 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:46.519314    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8e7948\"/\"kube-root-ca.crt\""
Oct 30 07:37:46.519567 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:46.519466    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8e7948\"/\"agc-cred-fd3280f89628\""
Oct 30 07:37:46.519567 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:46.519494    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8e7948\"/\"payload-processing-external-model-e2e--8346c882-dockercfg-kwlft\""
Oct 30 07:37:46.519567 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:46.519550    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8e7948\"/\"routing-overlay\""
Oct 30 07:37:46.519723 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:46.519688    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8e7948\"/\"payload-processing-external-model-plugins-e2e-extmod-a-8e7948\""
Oct 30 07:37:46.523223 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:46.523199    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8e7948/payload-processing-external-model-e2e-extmod-a-8e7948-5dd8xq2fh"]
Oct 30 07:37:46.599338 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:46.599306    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b457056c-a22b-480a-8ea4-3dcd9a2b4dae-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8e7948-5dd8xq2fh\" (UID: \"b457056c-a22b-480a-8ea4-3dcd9a2b4dae\") " pod="ai-tenant-e2e-extmod-a-8e7948/payload-processing-external-model-e2e-extmod-a-8e7948-5dd8xq2fh"
Oct 30 07:37:46.599517 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:46.599381    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b457056c-a22b-480a-8ea4-3dcd9a2b4dae-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8e7948-5dd8xq2fh\" (UID: \"b457056c-a22b-480a-8ea4-3dcd9a2b4dae\") " pod="ai-tenant-e2e-extmod-a-8e7948/payload-processing-external-model-e2e-extmod-a-8e7948-5dd8xq2fh"
Oct 30 07:37:46.599613 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:46.599536    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b457056c-a22b-480a-8ea4-3dcd9a2b4dae-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8e7948-5dd8xq2fh\" (UID: \"b457056c-a22b-480a-8ea4-3dcd9a2b4dae\") " pod="ai-tenant-e2e-extmod-a-8e7948/payload-processing-external-model-e2e-extmod-a-8e7948-5dd8xq2fh"
Oct 30 07:37:46.599668 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:46.599652    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blk86\" (UniqueName: \"kubernetes.io/projected/b457056c-a22b-480a-8ea4-3dcd9a2b4dae-kube-api-access-blk86\") pod \"payload-processing-external-model-e2e-extmod-a-8e7948-5dd8xq2fh\" (UID: \"b457056c-a22b-480a-8ea4-3dcd9a2b4dae\") " pod="ai-tenant-e2e-extmod-a-8e7948/payload-processing-external-model-e2e-extmod-a-8e7948-5dd8xq2fh"
Oct 30 07:37:46.700737 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:46.700689    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b457056c-a22b-480a-8ea4-3dcd9a2b4dae-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8e7948-5dd8xq2fh\" (UID: \"b457056c-a22b-480a-8ea4-3dcd9a2b4dae\") " pod="ai-tenant-e2e-extmod-a-8e7948/payload-processing-external-model-e2e-extmod-a-8e7948-5dd8xq2fh"
Oct 30 07:37:46.701050 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:46.700838    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b457056c-a22b-480a-8ea4-3dcd9a2b4dae-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8e7948-5dd8xq2fh\" (UID: \"b457056c-a22b-480a-8ea4-3dcd9a2b4dae\") " pod="ai-tenant-e2e-extmod-a-8e7948/payload-processing-external-model-e2e-extmod-a-8e7948-5dd8xq2fh"
Oct 30 07:37:46.701050 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:46.700941    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b457056c-a22b-480a-8ea4-3dcd9a2b4dae-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8e7948-5dd8xq2fh\" (UID: \"b457056c-a22b-480a-8ea4-3dcd9a2b4dae\") " pod="ai-tenant-e2e-extmod-a-8e7948/payload-processing-external-model-e2e-extmod-a-8e7948-5dd8xq2fh"
Oct 30 07:37:46.701050 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:46.701016    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blk86\" (UniqueName: \"kubernetes.io/projected/b457056c-a22b-480a-8ea4-3dcd9a2b4dae-kube-api-access-blk86\") pod \"payload-processing-external-model-e2e-extmod-a-8e7948-5dd8xq2fh\" (UID: \"b457056c-a22b-480a-8ea4-3dcd9a2b4dae\") " pod="ai-tenant-e2e-extmod-a-8e7948/payload-processing-external-model-e2e-extmod-a-8e7948-5dd8xq2fh"
Oct 30 07:37:46.703874 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:46.702491    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b457056c-a22b-480a-8ea4-3dcd9a2b4dae-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8e7948-5dd8xq2fh\" (UID: \"b457056c-a22b-480a-8ea4-3dcd9a2b4dae\") " pod="ai-tenant-e2e-extmod-a-8e7948/payload-processing-external-model-e2e-extmod-a-8e7948-5dd8xq2fh"
Oct 30 07:37:46.703874 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:46.703673    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b457056c-a22b-480a-8ea4-3dcd9a2b4dae-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8e7948-5dd8xq2fh\" (UID: \"b457056c-a22b-480a-8ea4-3dcd9a2b4dae\") " pod="ai-tenant-e2e-extmod-a-8e7948/payload-processing-external-model-e2e-extmod-a-8e7948-5dd8xq2fh"
Oct 30 07:37:46.707244 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:46.706546    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b457056c-a22b-480a-8ea4-3dcd9a2b4dae-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8e7948-5dd8xq2fh\" (UID: \"b457056c-a22b-480a-8ea4-3dcd9a2b4dae\") " pod="ai-tenant-e2e-extmod-a-8e7948/payload-processing-external-model-e2e-extmod-a-8e7948-5dd8xq2fh"
Oct 30 07:37:46.741874 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:46.732569    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blk86\" (UniqueName: \"kubernetes.io/projected/b457056c-a22b-480a-8ea4-3dcd9a2b4dae-kube-api-access-blk86\") pod \"payload-processing-external-model-e2e-extmod-a-8e7948-5dd8xq2fh\" (UID: \"b457056c-a22b-480a-8ea4-3dcd9a2b4dae\") " pod="ai-tenant-e2e-extmod-a-8e7948/payload-processing-external-model-e2e-extmod-a-8e7948-5dd8xq2fh"
Oct 30 07:37:46.802264 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:46.802228    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8e7948/payload-processing-external-model-e2e-extmod-a-8e7948-5dd8xq2fh"
Oct 30 07:37:47.115164 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:37:47.115126    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb457056c_a22b_480a_8ea4_3dcd9a2b4dae.slice/crio-8282a43a291e4a028e1e64ab063b3d0a7b588e6c3810ca48867ea3194866cb61 WatchSource:0}: Error finding container 8282a43a291e4a028e1e64ab063b3d0a7b588e6c3810ca48867ea3194866cb61: Status 404 returned error can't find the container with id 8282a43a291e4a028e1e64ab063b3d0a7b588e6c3810ca48867ea3194866cb61
Oct 30 07:37:47.116555 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:47.116529    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8e7948/payload-processing-external-model-e2e-extmod-a-8e7948-5dd8xq2fh"]
Oct 30 07:37:48.126420 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:48.126378    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8e7948/payload-processing-external-model-e2e-extmod-a-8e7948-5dd8xq2fh" event={"ID":"b457056c-a22b-480a-8ea4-3dcd9a2b4dae","Type":"ContainerStarted","Data":"b73e70a1786e7d52c53fd5f1e9a9925badbe4656ae18a90ad709c38d3a6ea77a"}
Oct 30 07:37:48.126860 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:48.126474    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8e7948/payload-processing-external-model-e2e-extmod-a-8e7948-5dd8xq2fh" event={"ID":"b457056c-a22b-480a-8ea4-3dcd9a2b4dae","Type":"ContainerStarted","Data":"8282a43a291e4a028e1e64ab063b3d0a7b588e6c3810ca48867ea3194866cb61"}
Oct 30 07:37:48.126860 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:48.126627    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-8e7948/payload-processing-external-model-e2e-extmod-a-8e7948-5dd8xq2fh"
Oct 30 07:37:48.176180 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:48.176124    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-8e7948/payload-processing-external-model-e2e-extmod-a-8e7948-5dd8xq2fh" podStartSLOduration=2.17611154 podStartE2EDuration="2.17611154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:37:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:37:48.1725551 +0000 UTC m=+2426.781962070" watchObservedRunningTime="2026-10-30 07:37:48.17611154 +0000 UTC m=+2426.785518509"
Oct 30 07:37:48.658922 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:48.658889    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6855vnp8l"]
Oct 30 07:37:48.663082 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:48.663064    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6855vnp8l"
Oct 30 07:37:48.680967 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:48.680943    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9bec5c\"/\"routing-overlay\""
Oct 30 07:37:48.684437 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:48.684413    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6855vnp8l"]
Oct 30 07:37:48.694579 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:48.694555    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9bec5c\"/\"payload-processing-external-model-plugins-e2e-extmod-a-9bec5c\""
Oct 30 07:37:48.694715 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:48.694697    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9bec5c\"/\"kube-root-ca.crt\""
Oct 30 07:37:48.694715 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:48.694708    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-9bec5c\"/\"agc-cred-7acbf0effedd\""
Oct 30 07:37:48.694887 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:48.694858    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9bec5c\"/\"openshift-service-ca.crt\""
Oct 30 07:37:48.695002 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:48.694981    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-9bec5c\"/\"payload-processing-external-model-e2e--314725d2-dockercfg-2wsrt\""
Oct 30 07:37:48.836172 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:48.836139    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd2ed4b7-a88b-4f06-9673-3859fcef03d6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-6855vnp8l\" (UID: \"cd2ed4b7-a88b-4f06-9673-3859fcef03d6\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6855vnp8l"
Oct 30 07:37:48.836338 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:48.836281    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z97k\" (UniqueName: \"kubernetes.io/projected/cd2ed4b7-a88b-4f06-9673-3859fcef03d6-kube-api-access-9z97k\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-6855vnp8l\" (UID: \"cd2ed4b7-a88b-4f06-9673-3859fcef03d6\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6855vnp8l"
Oct 30 07:37:48.836384 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:48.836335    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cd2ed4b7-a88b-4f06-9673-3859fcef03d6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-6855vnp8l\" (UID: \"cd2ed4b7-a88b-4f06-9673-3859fcef03d6\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6855vnp8l"
Oct 30 07:37:48.836460 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:48.836437    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cd2ed4b7-a88b-4f06-9673-3859fcef03d6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-6855vnp8l\" (UID: \"cd2ed4b7-a88b-4f06-9673-3859fcef03d6\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6855vnp8l"
Oct 30 07:37:48.944869 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:48.939722    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd2ed4b7-a88b-4f06-9673-3859fcef03d6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-6855vnp8l\" (UID: \"cd2ed4b7-a88b-4f06-9673-3859fcef03d6\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6855vnp8l"
Oct 30 07:37:48.944869 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:48.939899    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z97k\" (UniqueName: \"kubernetes.io/projected/cd2ed4b7-a88b-4f06-9673-3859fcef03d6-kube-api-access-9z97k\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-6855vnp8l\" (UID: \"cd2ed4b7-a88b-4f06-9673-3859fcef03d6\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6855vnp8l"
Oct 30 07:37:48.944869 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:48.939974    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cd2ed4b7-a88b-4f06-9673-3859fcef03d6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-6855vnp8l\" (UID: \"cd2ed4b7-a88b-4f06-9673-3859fcef03d6\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6855vnp8l"
Oct 30 07:37:48.944869 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:48.940044    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cd2ed4b7-a88b-4f06-9673-3859fcef03d6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-6855vnp8l\" (UID: \"cd2ed4b7-a88b-4f06-9673-3859fcef03d6\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6855vnp8l"
Oct 30 07:37:48.944869 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:48.941843    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd2ed4b7-a88b-4f06-9673-3859fcef03d6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-6855vnp8l\" (UID: \"cd2ed4b7-a88b-4f06-9673-3859fcef03d6\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6855vnp8l"
Oct 30 07:37:48.944869 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:48.942611    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cd2ed4b7-a88b-4f06-9673-3859fcef03d6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-6855vnp8l\" (UID: \"cd2ed4b7-a88b-4f06-9673-3859fcef03d6\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6855vnp8l"
Oct 30 07:37:48.955838 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:48.955424    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cd2ed4b7-a88b-4f06-9673-3859fcef03d6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-6855vnp8l\" (UID: \"cd2ed4b7-a88b-4f06-9673-3859fcef03d6\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6855vnp8l"
Oct 30 07:37:48.987826 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:48.975008    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z97k\" (UniqueName: \"kubernetes.io/projected/cd2ed4b7-a88b-4f06-9673-3859fcef03d6-kube-api-access-9z97k\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-6855vnp8l\" (UID: \"cd2ed4b7-a88b-4f06-9673-3859fcef03d6\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6855vnp8l"
Oct 30 07:37:48.987826 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:48.982042    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6855vnp8l"
Oct 30 07:37:49.368573 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:37:49.368543    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd2ed4b7_a88b_4f06_9673_3859fcef03d6.slice/crio-2fb44639ddb9836e207c29baac829a8a1c996521c557e00cf43e4f83988fed23 WatchSource:0}: Error finding container 2fb44639ddb9836e207c29baac829a8a1c996521c557e00cf43e4f83988fed23: Status 404 returned error can't find the container with id 2fb44639ddb9836e207c29baac829a8a1c996521c557e00cf43e4f83988fed23
Oct 30 07:37:49.369966 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:49.369933    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6855vnp8l"]
Oct 30 07:37:50.144181 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:50.144147    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6855vnp8l" event={"ID":"cd2ed4b7-a88b-4f06-9673-3859fcef03d6","Type":"ContainerStarted","Data":"a8135f7e9681e9610b84c8d57f30ed66cbb5335ea43d8564ba07dbf1cd23c41d"}
Oct 30 07:37:50.144341 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:50.144203    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6855vnp8l" event={"ID":"cd2ed4b7-a88b-4f06-9673-3859fcef03d6","Type":"ContainerStarted","Data":"2fb44639ddb9836e207c29baac829a8a1c996521c557e00cf43e4f83988fed23"}
Oct 30 07:37:50.144431 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:50.144364    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6855vnp8l"
Oct 30 07:37:50.191273 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:50.191229    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6855vnp8l" podStartSLOduration=2.191214755 podStartE2EDuration="2.191214755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:37:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:37:50.189088523 +0000 UTC m=+2428.798495493" watchObservedRunningTime="2026-10-30 07:37:50.191214755 +0000 UTC m=+2428.800621726"
Oct 30 07:37:51.507185 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:51.507152    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8e7948/payload-processing-external-model-e2e-extmod-a-8e7948-5dd8xq2fh"]
Oct 30 07:37:52.802572 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:52.802540    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-786dq29q8"]
Oct 30 07:37:52.805609 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:52.805588    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-786dq29q8"
Oct 30 07:37:54.785434 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:54.785396    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz"]
Oct 30 07:37:54.785958 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:54.785587    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz" podUID="ac672938-faf4-44e5-98ca-ef87de6e3437" containerName="istio-proxy" containerID="cri-o://a564e6bd67a47cf2c049350c73939efe44422062ba3d2789c423e06a55320524" gracePeriod=30
Oct 30 07:37:56.508088 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:56.508048    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-8e7948/payload-processing-external-model-e2e-extmod-a-8e7948-5dd8xq2fh" podUID="b457056c-a22b-480a-8ea4-3dcd9a2b4dae" containerName="payload-processing" containerID="cri-o://b73e70a1786e7d52c53fd5f1e9a9925badbe4656ae18a90ad709c38d3a6ea77a" gracePeriod=25
Oct 30 07:37:56.787157 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:56.787131    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8e7948/payload-processing-external-model-e2e-extmod-a-8e7948-5dd8xq2fh"
Oct 30 07:37:56.869682 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:56.869644    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b457056c-a22b-480a-8ea4-3dcd9a2b4dae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b457056c-a22b-480a-8ea4-3dcd9a2b4dae-plugins-config-volume\") pod \"b457056c-a22b-480a-8ea4-3dcd9a2b4dae\" (UID: \"b457056c-a22b-480a-8ea4-3dcd9a2b4dae\") "
Oct 30 07:37:56.869862 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:56.869742    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b457056c-a22b-480a-8ea4-3dcd9a2b4dae-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b457056c-a22b-480a-8ea4-3dcd9a2b4dae-provider-credentials\") pod \"b457056c-a22b-480a-8ea4-3dcd9a2b4dae\" (UID: \"b457056c-a22b-480a-8ea4-3dcd9a2b4dae\") "
Oct 30 07:37:56.869912 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:56.869860    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b457056c-a22b-480a-8ea4-3dcd9a2b4dae-kube-api-access-blk86\" (UniqueName: \"kubernetes.io/projected/b457056c-a22b-480a-8ea4-3dcd9a2b4dae-kube-api-access-blk86\") pod \"b457056c-a22b-480a-8ea4-3dcd9a2b4dae\" (UID: \"b457056c-a22b-480a-8ea4-3dcd9a2b4dae\") "
Oct 30 07:37:56.869993 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:56.869980    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b457056c-a22b-480a-8ea4-3dcd9a2b4dae-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b457056c-a22b-480a-8ea4-3dcd9a2b4dae-routing-overlay-volume\") pod \"b457056c-a22b-480a-8ea4-3dcd9a2b4dae\" (UID: \"b457056c-a22b-480a-8ea4-3dcd9a2b4dae\") "
Oct 30 07:37:56.870626 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:56.870600    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b457056c-a22b-480a-8ea4-3dcd9a2b4dae-routing-overlay-volume" pod "b457056c-a22b-480a-8ea4-3dcd9a2b4dae" (UID: "b457056c-a22b-480a-8ea4-3dcd9a2b4dae"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:37:56.870935 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:56.870908    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b457056c-a22b-480a-8ea4-3dcd9a2b4dae-plugins-config-volume" pod "b457056c-a22b-480a-8ea4-3dcd9a2b4dae" (UID: "b457056c-a22b-480a-8ea4-3dcd9a2b4dae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:37:56.873867 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:56.873798    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b457056c-a22b-480a-8ea4-3dcd9a2b4dae-provider-credentials" pod "b457056c-a22b-480a-8ea4-3dcd9a2b4dae" (UID: "b457056c-a22b-480a-8ea4-3dcd9a2b4dae"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:37:56.874003 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:56.873898    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b457056c-a22b-480a-8ea4-3dcd9a2b4dae-kube-api-access-blk86" pod "b457056c-a22b-480a-8ea4-3dcd9a2b4dae" (UID: "b457056c-a22b-480a-8ea4-3dcd9a2b4dae"). InnerVolumeSpecName "kube-api-access-blk86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:37:56.971381 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:56.971345    6224 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b457056c-a22b-480a-8ea4-3dcd9a2b4dae-routing-overlay-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:37:56.971381 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:56.971376    6224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b457056c-a22b-480a-8ea4-3dcd9a2b4dae-plugins-config-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:37:56.971381 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:56.971385    6224 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b457056c-a22b-480a-8ea4-3dcd9a2b4dae-provider-credentials\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:37:56.971625 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:56.971396    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blk86\" (UniqueName: \"kubernetes.io/projected/b457056c-a22b-480a-8ea4-3dcd9a2b4dae-kube-api-access-blk86\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:37:57.196456 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:57.196416    6224 generic.go:356] "Generic (PLEG): container finished" podID="b457056c-a22b-480a-8ea4-3dcd9a2b4dae" containerID="b73e70a1786e7d52c53fd5f1e9a9925badbe4656ae18a90ad709c38d3a6ea77a" exitCode=0
Oct 30 07:37:57.196621 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:57.196460    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8e7948/payload-processing-external-model-e2e-extmod-a-8e7948-5dd8xq2fh"
Oct 30 07:37:57.196621 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:57.196496    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8e7948/payload-processing-external-model-e2e-extmod-a-8e7948-5dd8xq2fh" event={"ID":"b457056c-a22b-480a-8ea4-3dcd9a2b4dae","Type":"ContainerDied","Data":"b73e70a1786e7d52c53fd5f1e9a9925badbe4656ae18a90ad709c38d3a6ea77a"}
Oct 30 07:37:57.196621 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:57.196553    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8e7948/payload-processing-external-model-e2e-extmod-a-8e7948-5dd8xq2fh" event={"ID":"b457056c-a22b-480a-8ea4-3dcd9a2b4dae","Type":"ContainerDied","Data":"8282a43a291e4a028e1e64ab063b3d0a7b588e6c3810ca48867ea3194866cb61"}
Oct 30 07:37:57.196621 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:57.196578    6224 scope.go:122] "RemoveContainer" containerID="b73e70a1786e7d52c53fd5f1e9a9925badbe4656ae18a90ad709c38d3a6ea77a"
Oct 30 07:37:57.207624 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:57.207596    6224 scope.go:122] "RemoveContainer" containerID="b73e70a1786e7d52c53fd5f1e9a9925badbe4656ae18a90ad709c38d3a6ea77a"
Oct 30 07:37:57.207976 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:37:57.207950    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b73e70a1786e7d52c53fd5f1e9a9925badbe4656ae18a90ad709c38d3a6ea77a\": container with ID starting with b73e70a1786e7d52c53fd5f1e9a9925badbe4656ae18a90ad709c38d3a6ea77a not found: ID does not exist" containerID="b73e70a1786e7d52c53fd5f1e9a9925badbe4656ae18a90ad709c38d3a6ea77a"
Oct 30 07:37:57.208082 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:57.207983    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b73e70a1786e7d52c53fd5f1e9a9925badbe4656ae18a90ad709c38d3a6ea77a"} err="failed to get container status \"b73e70a1786e7d52c53fd5f1e9a9925badbe4656ae18a90ad709c38d3a6ea77a\": rpc error: code = NotFound desc = could not find container \"b73e70a1786e7d52c53fd5f1e9a9925badbe4656ae18a90ad709c38d3a6ea77a\": container with ID starting with b73e70a1786e7d52c53fd5f1e9a9925badbe4656ae18a90ad709c38d3a6ea77a not found: ID does not exist"
Oct 30 07:37:57.255576 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:57.255543    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8e7948/payload-processing-external-model-e2e-extmod-a-8e7948-5dd8xq2fh"]
Oct 30 07:37:57.259759 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:57.259736    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-8e7948/payload-processing-external-model-e2e-extmod-a-8e7948-5dd8xq2fh"]
Oct 30 07:37:57.504549 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:57.504468    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b457056c-a22b-480a-8ea4-3dcd9a2b4dae" path="/var/lib/kubelet/pods/b457056c-a22b-480a-8ea4-3dcd9a2b4dae/volumes"
Oct 30 07:37:57.803282 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:57.803243    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-786dq29q8" podUID="fafd3773-189e-4e9f-b4fa-c6bababa9a83" containerName="payload-processing" containerID="cri-o://b87f9821901f630d0d17ea7651b6733e0534d064c60014725b83a383b1cc1bc7" gracePeriod=25
Oct 30 07:37:58.084628 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:58.084596    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-786dq29q8"
Oct 30 07:37:58.187955 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:58.187920    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fafd3773-189e-4e9f-b4fa-c6bababa9a83-kube-api-access-9xgqb\" (UniqueName: \"kubernetes.io/projected/fafd3773-189e-4e9f-b4fa-c6bababa9a83-kube-api-access-9xgqb\") pod \"fafd3773-189e-4e9f-b4fa-c6bababa9a83\" (UID: \"fafd3773-189e-4e9f-b4fa-c6bababa9a83\") "
Oct 30 07:37:58.188129 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:58.188068    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fafd3773-189e-4e9f-b4fa-c6bababa9a83-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fafd3773-189e-4e9f-b4fa-c6bababa9a83-plugins-config-volume\") pod \"fafd3773-189e-4e9f-b4fa-c6bababa9a83\" (UID: \"fafd3773-189e-4e9f-b4fa-c6bababa9a83\") "
Oct 30 07:37:58.188255 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:58.188241    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fafd3773-189e-4e9f-b4fa-c6bababa9a83-provider-credentials\" (UniqueName: \"kubernetes.io/projected/fafd3773-189e-4e9f-b4fa-c6bababa9a83-provider-credentials\") pod \"fafd3773-189e-4e9f-b4fa-c6bababa9a83\" (UID: \"fafd3773-189e-4e9f-b4fa-c6bababa9a83\") "
Oct 30 07:37:58.188327 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:58.188269    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fafd3773-189e-4e9f-b4fa-c6bababa9a83-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fafd3773-189e-4e9f-b4fa-c6bababa9a83-routing-overlay-volume\") pod \"fafd3773-189e-4e9f-b4fa-c6bababa9a83\" (UID: \"fafd3773-189e-4e9f-b4fa-c6bababa9a83\") "
Oct 30 07:37:58.188580 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:58.188543    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fafd3773-189e-4e9f-b4fa-c6bababa9a83-plugins-config-volume" pod "fafd3773-189e-4e9f-b4fa-c6bababa9a83" (UID: "fafd3773-189e-4e9f-b4fa-c6bababa9a83"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:37:58.188774 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:58.188749    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fafd3773-189e-4e9f-b4fa-c6bababa9a83-routing-overlay-volume" pod "fafd3773-189e-4e9f-b4fa-c6bababa9a83" (UID: "fafd3773-189e-4e9f-b4fa-c6bababa9a83"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:37:58.188929 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:58.188910    6224 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fafd3773-189e-4e9f-b4fa-c6bababa9a83-routing-overlay-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:37:58.189007 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:58.188930    6224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fafd3773-189e-4e9f-b4fa-c6bababa9a83-plugins-config-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:37:58.190325 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:58.190303    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fafd3773-189e-4e9f-b4fa-c6bababa9a83-kube-api-access-9xgqb" pod "fafd3773-189e-4e9f-b4fa-c6bababa9a83" (UID: "fafd3773-189e-4e9f-b4fa-c6bababa9a83"). InnerVolumeSpecName "kube-api-access-9xgqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:37:58.190427 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:58.190373    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fafd3773-189e-4e9f-b4fa-c6bababa9a83-provider-credentials" pod "fafd3773-189e-4e9f-b4fa-c6bababa9a83" (UID: "fafd3773-189e-4e9f-b4fa-c6bababa9a83"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:37:58.206238 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:58.206210    6224 generic.go:356] "Generic (PLEG): container finished" podID="fafd3773-189e-4e9f-b4fa-c6bababa9a83" containerID="b87f9821901f630d0d17ea7651b6733e0534d064c60014725b83a383b1cc1bc7" exitCode=0
Oct 30 07:37:58.206341 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:58.206256    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-786dq29q8"
Oct 30 07:37:58.206341 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:58.206292    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-786dq29q8" event={"ID":"fafd3773-189e-4e9f-b4fa-c6bababa9a83","Type":"ContainerDied","Data":"b87f9821901f630d0d17ea7651b6733e0534d064c60014725b83a383b1cc1bc7"}
Oct 30 07:37:58.206474 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:58.206350    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-786dq29q8" event={"ID":"fafd3773-189e-4e9f-b4fa-c6bababa9a83","Type":"ContainerDied","Data":"e1c483f93dfb22b33e91c5cc9435ff3b150256dd30e2fcb62fb556ab19d1e29f"}
Oct 30 07:37:58.206474 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:58.206372    6224 scope.go:122] "RemoveContainer" containerID="b87f9821901f630d0d17ea7651b6733e0534d064c60014725b83a383b1cc1bc7"
Oct 30 07:37:58.219426 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:58.219402    6224 scope.go:122] "RemoveContainer" containerID="b87f9821901f630d0d17ea7651b6733e0534d064c60014725b83a383b1cc1bc7"
Oct 30 07:37:58.219702 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:37:58.219683    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b87f9821901f630d0d17ea7651b6733e0534d064c60014725b83a383b1cc1bc7\": container with ID starting with b87f9821901f630d0d17ea7651b6733e0534d064c60014725b83a383b1cc1bc7 not found: ID does not exist" containerID="b87f9821901f630d0d17ea7651b6733e0534d064c60014725b83a383b1cc1bc7"
Oct 30 07:37:58.219746 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:58.219709    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b87f9821901f630d0d17ea7651b6733e0534d064c60014725b83a383b1cc1bc7"} err="failed to get container status \"b87f9821901f630d0d17ea7651b6733e0534d064c60014725b83a383b1cc1bc7\": rpc error: code = NotFound desc = could not find container \"b87f9821901f630d0d17ea7651b6733e0534d064c60014725b83a383b1cc1bc7\": container with ID starting with b87f9821901f630d0d17ea7651b6733e0534d064c60014725b83a383b1cc1bc7 not found: ID does not exist"
Oct 30 07:37:58.267842 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:58.267799    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-786dq29q8"]
Oct 30 07:37:58.273766 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:58.273739    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-f1f700/payload-processing-external-model-e2e-extmod-b-f1f700-786dq29q8"]
Oct 30 07:37:58.290007 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:58.289984    6224 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fafd3773-189e-4e9f-b4fa-c6bababa9a83-provider-credentials\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:37:58.290007 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:58.290006    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xgqb\" (UniqueName: \"kubernetes.io/projected/fafd3773-189e-4e9f-b4fa-c6bababa9a83-kube-api-access-9xgqb\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:37:59.504285 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:37:59.504252    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fafd3773-189e-4e9f-b4fa-c6bababa9a83" path="/var/lib/kubelet/pods/fafd3773-189e-4e9f-b4fa-c6bababa9a83/volumes"
Oct 30 07:38:00.065599 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:00.065544    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz"
Oct 30 07:38:00.215063 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:00.215028    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac672938-faf4-44e5-98ca-ef87de6e3437-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac672938-faf4-44e5-98ca-ef87de6e3437-workload-certs\") pod \"ac672938-faf4-44e5-98ca-ef87de6e3437\" (UID: \"ac672938-faf4-44e5-98ca-ef87de6e3437\") "
Oct 30 07:38:00.215224 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:00.215113    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac672938-faf4-44e5-98ca-ef87de6e3437-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac672938-faf4-44e5-98ca-ef87de6e3437-workload-socket\") pod \"ac672938-faf4-44e5-98ca-ef87de6e3437\" (UID: \"ac672938-faf4-44e5-98ca-ef87de6e3437\") "
Oct 30 07:38:00.215224 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:00.215159    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac672938-faf4-44e5-98ca-ef87de6e3437-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac672938-faf4-44e5-98ca-ef87de6e3437-istiod-ca-cert\") pod \"ac672938-faf4-44e5-98ca-ef87de6e3437\" (UID: \"ac672938-faf4-44e5-98ca-ef87de6e3437\") "
Oct 30 07:38:00.215302 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:00.215232    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac672938-faf4-44e5-98ca-ef87de6e3437-istio-token\" (UniqueName: \"kubernetes.io/projected/ac672938-faf4-44e5-98ca-ef87de6e3437-istio-token\") pod \"ac672938-faf4-44e5-98ca-ef87de6e3437\" (UID: \"ac672938-faf4-44e5-98ca-ef87de6e3437\") "
Oct 30 07:38:00.215302 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:00.215251    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac672938-faf4-44e5-98ca-ef87de6e3437-kube-api-access-kb75n\" (UniqueName: \"kubernetes.io/projected/ac672938-faf4-44e5-98ca-ef87de6e3437-kube-api-access-kb75n\") pod \"ac672938-faf4-44e5-98ca-ef87de6e3437\" (UID: \"ac672938-faf4-44e5-98ca-ef87de6e3437\") "
Oct 30 07:38:00.215302 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:00.215278    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac672938-faf4-44e5-98ca-ef87de6e3437-workload-certs" pod "ac672938-faf4-44e5-98ca-ef87de6e3437" (UID: "ac672938-faf4-44e5-98ca-ef87de6e3437"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:38:00.215469 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:00.215283    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac672938-faf4-44e5-98ca-ef87de6e3437-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac672938-faf4-44e5-98ca-ef87de6e3437-credential-socket\") pod \"ac672938-faf4-44e5-98ca-ef87de6e3437\" (UID: \"ac672938-faf4-44e5-98ca-ef87de6e3437\") "
Oct 30 07:38:00.215469 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:00.215414    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac672938-faf4-44e5-98ca-ef87de6e3437-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac672938-faf4-44e5-98ca-ef87de6e3437-istio-data\") pod \"ac672938-faf4-44e5-98ca-ef87de6e3437\" (UID: \"ac672938-faf4-44e5-98ca-ef87de6e3437\") "
Oct 30 07:38:00.215469 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:00.215452    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac672938-faf4-44e5-98ca-ef87de6e3437-workload-socket" pod "ac672938-faf4-44e5-98ca-ef87de6e3437" (UID: "ac672938-faf4-44e5-98ca-ef87de6e3437"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:38:00.215615 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:00.215518    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac672938-faf4-44e5-98ca-ef87de6e3437-credential-socket" pod "ac672938-faf4-44e5-98ca-ef87de6e3437" (UID: "ac672938-faf4-44e5-98ca-ef87de6e3437"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:38:00.215615 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:00.215550    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac672938-faf4-44e5-98ca-ef87de6e3437-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac672938-faf4-44e5-98ca-ef87de6e3437-istio-podinfo\") pod \"ac672938-faf4-44e5-98ca-ef87de6e3437\" (UID: \"ac672938-faf4-44e5-98ca-ef87de6e3437\") "
Oct 30 07:38:00.215615 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:00.215600    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac672938-faf4-44e5-98ca-ef87de6e3437-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac672938-faf4-44e5-98ca-ef87de6e3437-istio-envoy\") pod \"ac672938-faf4-44e5-98ca-ef87de6e3437\" (UID: \"ac672938-faf4-44e5-98ca-ef87de6e3437\") "
Oct 30 07:38:00.215758 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:00.215664    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac672938-faf4-44e5-98ca-ef87de6e3437-istio-data" pod "ac672938-faf4-44e5-98ca-ef87de6e3437" (UID: "ac672938-faf4-44e5-98ca-ef87de6e3437"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:38:00.215758 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:00.215584    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac672938-faf4-44e5-98ca-ef87de6e3437-istiod-ca-cert" pod "ac672938-faf4-44e5-98ca-ef87de6e3437" (UID: "ac672938-faf4-44e5-98ca-ef87de6e3437"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:38:00.216257 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:00.216221    6224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac672938-faf4-44e5-98ca-ef87de6e3437-istio-data\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:38:00.216257 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:00.216249    6224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac672938-faf4-44e5-98ca-ef87de6e3437-workload-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:38:00.216458 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:00.216265    6224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac672938-faf4-44e5-98ca-ef87de6e3437-workload-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:38:00.216458 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:00.216279    6224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac672938-faf4-44e5-98ca-ef87de6e3437-istiod-ca-cert\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:38:00.216458 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:00.216294    6224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac672938-faf4-44e5-98ca-ef87de6e3437-credential-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:38:00.217661 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:00.217632    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac672938-faf4-44e5-98ca-ef87de6e3437-istio-token" pod "ac672938-faf4-44e5-98ca-ef87de6e3437" (UID: "ac672938-faf4-44e5-98ca-ef87de6e3437"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:38:00.217784 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:00.217690    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac672938-faf4-44e5-98ca-ef87de6e3437-istio-podinfo" pod "ac672938-faf4-44e5-98ca-ef87de6e3437" (UID: "ac672938-faf4-44e5-98ca-ef87de6e3437"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:38:00.217784 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:00.217706    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac672938-faf4-44e5-98ca-ef87de6e3437-kube-api-access-kb75n" pod "ac672938-faf4-44e5-98ca-ef87de6e3437" (UID: "ac672938-faf4-44e5-98ca-ef87de6e3437"). InnerVolumeSpecName "kube-api-access-kb75n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:38:00.218033 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:00.218011    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac672938-faf4-44e5-98ca-ef87de6e3437-istio-envoy" pod "ac672938-faf4-44e5-98ca-ef87de6e3437" (UID: "ac672938-faf4-44e5-98ca-ef87de6e3437"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:38:00.225322 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:00.225293    6224 generic.go:356] "Generic (PLEG): container finished" podID="ac672938-faf4-44e5-98ca-ef87de6e3437" containerID="a564e6bd67a47cf2c049350c73939efe44422062ba3d2789c423e06a55320524" exitCode=0
Oct 30 07:38:00.225430 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:00.225343    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz"
Oct 30 07:38:00.225430 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:00.225373    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz" event={"ID":"ac672938-faf4-44e5-98ca-ef87de6e3437","Type":"ContainerDied","Data":"a564e6bd67a47cf2c049350c73939efe44422062ba3d2789c423e06a55320524"}
Oct 30 07:38:00.225430 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:00.225427    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz" event={"ID":"ac672938-faf4-44e5-98ca-ef87de6e3437","Type":"ContainerDied","Data":"348e35c907071667cf9297001a361e5e15463c6c107b7a92413692453350074f"}
Oct 30 07:38:00.225605 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:00.225450    6224 scope.go:122] "RemoveContainer" containerID="a564e6bd67a47cf2c049350c73939efe44422062ba3d2789c423e06a55320524"
Oct 30 07:38:00.252376 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:00.252357    6224 scope.go:122] "RemoveContainer" containerID="a564e6bd67a47cf2c049350c73939efe44422062ba3d2789c423e06a55320524"
Oct 30 07:38:00.252744 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:38:00.252721    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a564e6bd67a47cf2c049350c73939efe44422062ba3d2789c423e06a55320524\": container with ID starting with a564e6bd67a47cf2c049350c73939efe44422062ba3d2789c423e06a55320524 not found: ID does not exist" containerID="a564e6bd67a47cf2c049350c73939efe44422062ba3d2789c423e06a55320524"
Oct 30 07:38:00.252826 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:00.252752    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a564e6bd67a47cf2c049350c73939efe44422062ba3d2789c423e06a55320524"} err="failed to get container status \"a564e6bd67a47cf2c049350c73939efe44422062ba3d2789c423e06a55320524\": rpc error: code = NotFound desc = could not find container \"a564e6bd67a47cf2c049350c73939efe44422062ba3d2789c423e06a55320524\": container with ID starting with a564e6bd67a47cf2c049350c73939efe44422062ba3d2789c423e06a55320524 not found: ID does not exist"
Oct 30 07:38:00.295505 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:00.295477    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz"]
Oct 30 07:38:00.299832 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:00.299793    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-7hclz"]
Oct 30 07:38:00.317396 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:00.317340    6224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac672938-faf4-44e5-98ca-ef87de6e3437-istio-podinfo\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:38:00.317396 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:00.317363    6224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac672938-faf4-44e5-98ca-ef87de6e3437-istio-envoy\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:38:00.317396 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:00.317374    6224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac672938-faf4-44e5-98ca-ef87de6e3437-istio-token\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:38:00.317396 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:00.317383    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kb75n\" (UniqueName: \"kubernetes.io/projected/ac672938-faf4-44e5-98ca-ef87de6e3437-kube-api-access-kb75n\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:38:01.155112 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:01.155082    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6855vnp8l"
Oct 30 07:38:01.505910 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:01.505801    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac672938-faf4-44e5-98ca-ef87de6e3437" path="/var/lib/kubelet/pods/ac672938-faf4-44e5-98ca-ef87de6e3437/volumes"
Oct 30 07:38:01.858652 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:38:01.858614    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:38:04.713899 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:04.713818    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:38:06.299230 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:06.299199    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6855vnp8l"]
Oct 30 07:38:11.300045 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:11.300001    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6855vnp8l" podUID="cd2ed4b7-a88b-4f06-9673-3859fcef03d6" containerName="payload-processing" containerID="cri-o://a8135f7e9681e9610b84c8d57f30ed66cbb5335ea43d8564ba07dbf1cd23c41d" gracePeriod=25
Oct 30 07:38:11.594975 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:11.594949    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6855vnp8l"
Oct 30 07:38:11.663966 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:11.663936    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd2ed4b7-a88b-4f06-9673-3859fcef03d6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd2ed4b7-a88b-4f06-9673-3859fcef03d6-plugins-config-volume\") pod \"cd2ed4b7-a88b-4f06-9673-3859fcef03d6\" (UID: \"cd2ed4b7-a88b-4f06-9673-3859fcef03d6\") "
Oct 30 07:38:11.664257 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:11.664239    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd2ed4b7-a88b-4f06-9673-3859fcef03d6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cd2ed4b7-a88b-4f06-9673-3859fcef03d6-routing-overlay-volume\") pod \"cd2ed4b7-a88b-4f06-9673-3859fcef03d6\" (UID: \"cd2ed4b7-a88b-4f06-9673-3859fcef03d6\") "
Oct 30 07:38:11.664376 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:11.664363    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd2ed4b7-a88b-4f06-9673-3859fcef03d6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/cd2ed4b7-a88b-4f06-9673-3859fcef03d6-provider-credentials\") pod \"cd2ed4b7-a88b-4f06-9673-3859fcef03d6\" (UID: \"cd2ed4b7-a88b-4f06-9673-3859fcef03d6\") "
Oct 30 07:38:11.664473 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:11.664461    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd2ed4b7-a88b-4f06-9673-3859fcef03d6-kube-api-access-9z97k\" (UniqueName: \"kubernetes.io/projected/cd2ed4b7-a88b-4f06-9673-3859fcef03d6-kube-api-access-9z97k\") pod \"cd2ed4b7-a88b-4f06-9673-3859fcef03d6\" (UID: \"cd2ed4b7-a88b-4f06-9673-3859fcef03d6\") "
Oct 30 07:38:11.664538 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:11.664422    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd2ed4b7-a88b-4f06-9673-3859fcef03d6-plugins-config-volume" pod "cd2ed4b7-a88b-4f06-9673-3859fcef03d6" (UID: "cd2ed4b7-a88b-4f06-9673-3859fcef03d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:38:11.664599 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:11.664574    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd2ed4b7-a88b-4f06-9673-3859fcef03d6-routing-overlay-volume" pod "cd2ed4b7-a88b-4f06-9673-3859fcef03d6" (UID: "cd2ed4b7-a88b-4f06-9673-3859fcef03d6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:38:11.664905 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:11.664886    6224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd2ed4b7-a88b-4f06-9673-3859fcef03d6-plugins-config-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:38:11.664984 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:11.664908    6224 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cd2ed4b7-a88b-4f06-9673-3859fcef03d6-routing-overlay-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:38:11.666854 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:11.666818    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd2ed4b7-a88b-4f06-9673-3859fcef03d6-provider-credentials" pod "cd2ed4b7-a88b-4f06-9673-3859fcef03d6" (UID: "cd2ed4b7-a88b-4f06-9673-3859fcef03d6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:38:11.667144 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:11.667117    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd2ed4b7-a88b-4f06-9673-3859fcef03d6-kube-api-access-9z97k" pod "cd2ed4b7-a88b-4f06-9673-3859fcef03d6" (UID: "cd2ed4b7-a88b-4f06-9673-3859fcef03d6"). InnerVolumeSpecName "kube-api-access-9z97k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:38:11.765612 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:11.765586    6224 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cd2ed4b7-a88b-4f06-9673-3859fcef03d6-provider-credentials\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:38:11.765612 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:11.765614    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9z97k\" (UniqueName: \"kubernetes.io/projected/cd2ed4b7-a88b-4f06-9673-3859fcef03d6-kube-api-access-9z97k\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:38:12.318768 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:12.318731    6224 generic.go:356] "Generic (PLEG): container finished" podID="cd2ed4b7-a88b-4f06-9673-3859fcef03d6" containerID="a8135f7e9681e9610b84c8d57f30ed66cbb5335ea43d8564ba07dbf1cd23c41d" exitCode=0
Oct 30 07:38:12.319261 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:12.318783    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6855vnp8l"
Oct 30 07:38:12.319261 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:12.318825    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6855vnp8l" event={"ID":"cd2ed4b7-a88b-4f06-9673-3859fcef03d6","Type":"ContainerDied","Data":"a8135f7e9681e9610b84c8d57f30ed66cbb5335ea43d8564ba07dbf1cd23c41d"}
Oct 30 07:38:12.319261 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:12.318879    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6855vnp8l" event={"ID":"cd2ed4b7-a88b-4f06-9673-3859fcef03d6","Type":"ContainerDied","Data":"2fb44639ddb9836e207c29baac829a8a1c996521c557e00cf43e4f83988fed23"}
Oct 30 07:38:12.319261 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:12.318907    6224 scope.go:122] "RemoveContainer" containerID="a8135f7e9681e9610b84c8d57f30ed66cbb5335ea43d8564ba07dbf1cd23c41d"
Oct 30 07:38:12.329213 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:12.329191    6224 scope.go:122] "RemoveContainer" containerID="a8135f7e9681e9610b84c8d57f30ed66cbb5335ea43d8564ba07dbf1cd23c41d"
Oct 30 07:38:12.329474 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:38:12.329454    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8135f7e9681e9610b84c8d57f30ed66cbb5335ea43d8564ba07dbf1cd23c41d\": container with ID starting with a8135f7e9681e9610b84c8d57f30ed66cbb5335ea43d8564ba07dbf1cd23c41d not found: ID does not exist" containerID="a8135f7e9681e9610b84c8d57f30ed66cbb5335ea43d8564ba07dbf1cd23c41d"
Oct 30 07:38:12.329527 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:12.329482    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8135f7e9681e9610b84c8d57f30ed66cbb5335ea43d8564ba07dbf1cd23c41d"} err="failed to get container status \"a8135f7e9681e9610b84c8d57f30ed66cbb5335ea43d8564ba07dbf1cd23c41d\": rpc error: code = NotFound desc = could not find container \"a8135f7e9681e9610b84c8d57f30ed66cbb5335ea43d8564ba07dbf1cd23c41d\": container with ID starting with a8135f7e9681e9610b84c8d57f30ed66cbb5335ea43d8564ba07dbf1cd23c41d not found: ID does not exist"
Oct 30 07:38:12.377379 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:12.377355    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6855vnp8l"]
Oct 30 07:38:12.391218 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:12.391194    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6855vnp8l"]
Oct 30 07:38:13.505123 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:13.505078    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd2ed4b7-a88b-4f06-9673-3859fcef03d6" path="/var/lib/kubelet/pods/cd2ed4b7-a88b-4f06-9673-3859fcef03d6/volumes"
Oct 30 07:38:20.987475 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:20.987445    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-6b49ljgld"]
Oct 30 07:38:20.988100 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:20.988085    6224 cpu_manager.go:414] "containerMap: removing container" podUID="b457056c-a22b-480a-8ea4-3dcd9a2b4dae" containerName="payload-processing"
Oct 30 07:38:20.988143 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:20.988101    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b457056c-a22b-480a-8ea4-3dcd9a2b4dae" containerName="payload-processing"
Oct 30 07:38:20.988143 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:20.988118    6224 cpu_manager.go:414] "containerMap: removing container" podUID="cd2ed4b7-a88b-4f06-9673-3859fcef03d6" containerName="payload-processing"
Oct 30 07:38:20.988143 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:20.988123    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd2ed4b7-a88b-4f06-9673-3859fcef03d6" containerName="payload-processing"
Oct 30 07:38:20.988143 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:20.988134    6224 cpu_manager.go:414] "containerMap: removing container" podUID="ac672938-faf4-44e5-98ca-ef87de6e3437" containerName="istio-proxy"
Oct 30 07:38:20.988143 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:20.988140    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac672938-faf4-44e5-98ca-ef87de6e3437" containerName="istio-proxy"
Oct 30 07:38:20.988306 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:20.988159    6224 cpu_manager.go:414] "containerMap: removing container" podUID="fafd3773-189e-4e9f-b4fa-c6bababa9a83" containerName="payload-processing"
Oct 30 07:38:20.988306 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:20.988165    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fafd3773-189e-4e9f-b4fa-c6bababa9a83" containerName="payload-processing"
Oct 30 07:38:20.988306 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:20.988266    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="b457056c-a22b-480a-8ea4-3dcd9a2b4dae" containerName="payload-processing"
Oct 30 07:38:20.988306 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:20.988274    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd2ed4b7-a88b-4f06-9673-3859fcef03d6" containerName="payload-processing"
Oct 30 07:38:20.988306 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:20.988283    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac672938-faf4-44e5-98ca-ef87de6e3437" containerName="istio-proxy"
Oct 30 07:38:20.988306 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:20.988292    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="fafd3773-189e-4e9f-b4fa-c6bababa9a83" containerName="payload-processing"
Oct 30 07:38:20.992013 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:20.991990    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-6b49ljgld"
Oct 30 07:38:21.010732 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:21.010709    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-6b49ljgld"]
Oct 30 07:38:21.021403 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:21.021367    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1f34aa\"/\"agc-cred-dd6c04d9ae3b\""
Oct 30 07:38:21.021403 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:21.021373    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1f34aa\"/\"agc-cred-652ba7d1390f\""
Oct 30 07:38:21.021403 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:21.021376    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1f34aa\"/\"payload-processing-external-model-plugins-e2e-extmod-b-1f34aa\""
Oct 30 07:38:21.024430 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:21.024408    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1f34aa\"/\"openshift-service-ca.crt\""
Oct 30 07:38:21.024552 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:21.024496    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1f34aa\"/\"payload-processing-external-model-e2e--e56a4edc-dockercfg-s85b7\""
Oct 30 07:38:21.024552 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:21.024534    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1f34aa\"/\"routing-overlay\""
Oct 30 07:38:21.024673 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:21.024603    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1f34aa\"/\"kube-root-ca.crt\""
Oct 30 07:38:21.076023 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:21.075992    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/de4a9e38-8d9a-4f17-bb63-dc29ef3fe3da-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-6b49ljgld\" (UID: \"de4a9e38-8d9a-4f17-bb63-dc29ef3fe3da\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-6b49ljgld"
Oct 30 07:38:21.076146 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:21.076110    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/de4a9e38-8d9a-4f17-bb63-dc29ef3fe3da-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-6b49ljgld\" (UID: \"de4a9e38-8d9a-4f17-bb63-dc29ef3fe3da\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-6b49ljgld"
Oct 30 07:38:21.076190 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:21.076166    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s9mc\" (UniqueName: \"kubernetes.io/projected/de4a9e38-8d9a-4f17-bb63-dc29ef3fe3da-kube-api-access-7s9mc\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-6b49ljgld\" (UID: \"de4a9e38-8d9a-4f17-bb63-dc29ef3fe3da\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-6b49ljgld"
Oct 30 07:38:21.076329 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:21.076309    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de4a9e38-8d9a-4f17-bb63-dc29ef3fe3da-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-6b49ljgld\" (UID: \"de4a9e38-8d9a-4f17-bb63-dc29ef3fe3da\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-6b49ljgld"
Oct 30 07:38:21.177520 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:21.177486    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/de4a9e38-8d9a-4f17-bb63-dc29ef3fe3da-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-6b49ljgld\" (UID: \"de4a9e38-8d9a-4f17-bb63-dc29ef3fe3da\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-6b49ljgld"
Oct 30 07:38:21.177648 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:21.177536    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s9mc\" (UniqueName: \"kubernetes.io/projected/de4a9e38-8d9a-4f17-bb63-dc29ef3fe3da-kube-api-access-7s9mc\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-6b49ljgld\" (UID: \"de4a9e38-8d9a-4f17-bb63-dc29ef3fe3da\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-6b49ljgld"
Oct 30 07:38:21.177648 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:21.177603    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de4a9e38-8d9a-4f17-bb63-dc29ef3fe3da-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-6b49ljgld\" (UID: \"de4a9e38-8d9a-4f17-bb63-dc29ef3fe3da\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-6b49ljgld"
Oct 30 07:38:21.177735 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:21.177670    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/de4a9e38-8d9a-4f17-bb63-dc29ef3fe3da-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-6b49ljgld\" (UID: \"de4a9e38-8d9a-4f17-bb63-dc29ef3fe3da\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-6b49ljgld"
Oct 30 07:38:21.178171 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:21.178145    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/de4a9e38-8d9a-4f17-bb63-dc29ef3fe3da-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-6b49ljgld\" (UID: \"de4a9e38-8d9a-4f17-bb63-dc29ef3fe3da\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-6b49ljgld"
Oct 30 07:38:21.178319 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:21.178294    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de4a9e38-8d9a-4f17-bb63-dc29ef3fe3da-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-6b49ljgld\" (UID: \"de4a9e38-8d9a-4f17-bb63-dc29ef3fe3da\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-6b49ljgld"
Oct 30 07:38:21.180156 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:21.180136    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/de4a9e38-8d9a-4f17-bb63-dc29ef3fe3da-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-6b49ljgld\" (UID: \"de4a9e38-8d9a-4f17-bb63-dc29ef3fe3da\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-6b49ljgld"
Oct 30 07:38:21.219660 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:21.219623    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s9mc\" (UniqueName: \"kubernetes.io/projected/de4a9e38-8d9a-4f17-bb63-dc29ef3fe3da-kube-api-access-7s9mc\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-6b49ljgld\" (UID: \"de4a9e38-8d9a-4f17-bb63-dc29ef3fe3da\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-6b49ljgld"
Oct 30 07:38:21.305458 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:21.305426    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-6b49ljgld"
Oct 30 07:38:21.622580 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:21.622548    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-6b49ljgld"]
Oct 30 07:38:21.622712 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:38:21.622576    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde4a9e38_8d9a_4f17_bb63_dc29ef3fe3da.slice/crio-98df27d9b36ae606a440bc27b86bbbedb0d9af299e8ffdf95f3f9bcce2abb608 WatchSource:0}: Error finding container 98df27d9b36ae606a440bc27b86bbbedb0d9af299e8ffdf95f3f9bcce2abb608: Status 404 returned error can't find the container with id 98df27d9b36ae606a440bc27b86bbbedb0d9af299e8ffdf95f3f9bcce2abb608
Oct 30 07:38:22.406348 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:22.406311    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-6b49ljgld" event={"ID":"de4a9e38-8d9a-4f17-bb63-dc29ef3fe3da","Type":"ContainerStarted","Data":"4622f4002d8c4f01e3cbaa16b509b3151b197054178f79355c610e6853e09836"}
Oct 30 07:38:22.406795 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:22.406368    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-6b49ljgld" event={"ID":"de4a9e38-8d9a-4f17-bb63-dc29ef3fe3da","Type":"ContainerStarted","Data":"98df27d9b36ae606a440bc27b86bbbedb0d9af299e8ffdf95f3f9bcce2abb608"}
Oct 30 07:38:22.406795 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:22.406393    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-6b49ljgld"
Oct 30 07:38:22.453741 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:22.453695    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-6b49ljgld" podStartSLOduration=2.453682537 podStartE2EDuration="2.453682537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:38:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:38:22.451974002 +0000 UTC m=+2461.061380974" watchObservedRunningTime="2026-10-30 07:38:22.453682537 +0000 UTC m=+2461.063089507"
Oct 30 07:38:27.746185 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:27.746149    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-586dxf8bw"]
Oct 30 07:38:31.741536 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:31.741503    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f97kjsn"]
Oct 30 07:38:31.747311 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:31.747292    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f97kjsn"
Oct 30 07:38:31.768616 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:31.768593    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f97kjsn"]
Oct 30 07:38:31.779685 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:31.779663    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-0f5081\"/\"agc-cred-56d4535fe8f4\""
Oct 30 07:38:31.779796 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:31.779744    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-0f5081\"/\"payload-processing-external-model-e2e--8bb4c319-dockercfg-w92sm\""
Oct 30 07:38:31.860842 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:38:31.860782    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:38:31.955689 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:31.955643    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/71bd479f-89f2-4c1a-8215-d5e488a875dc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0f5081-65f97kjsn\" (UID: \"71bd479f-89f2-4c1a-8215-d5e488a875dc\") " pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f97kjsn"
Oct 30 07:38:31.955933 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:31.955715    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/71bd479f-89f2-4c1a-8215-d5e488a875dc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0f5081-65f97kjsn\" (UID: \"71bd479f-89f2-4c1a-8215-d5e488a875dc\") " pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f97kjsn"
Oct 30 07:38:31.955933 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:31.955742    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqqbz\" (UniqueName: \"kubernetes.io/projected/71bd479f-89f2-4c1a-8215-d5e488a875dc-kube-api-access-mqqbz\") pod \"payload-processing-external-model-e2e-extmod-a-0f5081-65f97kjsn\" (UID: \"71bd479f-89f2-4c1a-8215-d5e488a875dc\") " pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f97kjsn"
Oct 30 07:38:31.956062 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:31.955962    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71bd479f-89f2-4c1a-8215-d5e488a875dc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0f5081-65f97kjsn\" (UID: \"71bd479f-89f2-4c1a-8215-d5e488a875dc\") " pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f97kjsn"
Oct 30 07:38:32.067596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:32.057094    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/71bd479f-89f2-4c1a-8215-d5e488a875dc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0f5081-65f97kjsn\" (UID: \"71bd479f-89f2-4c1a-8215-d5e488a875dc\") " pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f97kjsn"
Oct 30 07:38:32.067596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:32.057173    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/71bd479f-89f2-4c1a-8215-d5e488a875dc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0f5081-65f97kjsn\" (UID: \"71bd479f-89f2-4c1a-8215-d5e488a875dc\") " pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f97kjsn"
Oct 30 07:38:32.067596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:32.057204    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqqbz\" (UniqueName: \"kubernetes.io/projected/71bd479f-89f2-4c1a-8215-d5e488a875dc-kube-api-access-mqqbz\") pod \"payload-processing-external-model-e2e-extmod-a-0f5081-65f97kjsn\" (UID: \"71bd479f-89f2-4c1a-8215-d5e488a875dc\") " pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f97kjsn"
Oct 30 07:38:32.067596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:32.057408    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71bd479f-89f2-4c1a-8215-d5e488a875dc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0f5081-65f97kjsn\" (UID: \"71bd479f-89f2-4c1a-8215-d5e488a875dc\") " pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f97kjsn"
Oct 30 07:38:32.067596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:32.058542    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71bd479f-89f2-4c1a-8215-d5e488a875dc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0f5081-65f97kjsn\" (UID: \"71bd479f-89f2-4c1a-8215-d5e488a875dc\") " pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f97kjsn"
Oct 30 07:38:32.067596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:32.059573    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/71bd479f-89f2-4c1a-8215-d5e488a875dc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0f5081-65f97kjsn\" (UID: \"71bd479f-89f2-4c1a-8215-d5e488a875dc\") " pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f97kjsn"
Oct 30 07:38:32.069589 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:32.069538    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/71bd479f-89f2-4c1a-8215-d5e488a875dc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0f5081-65f97kjsn\" (UID: \"71bd479f-89f2-4c1a-8215-d5e488a875dc\") " pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f97kjsn"
Oct 30 07:38:32.086876 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:32.086678    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqqbz\" (UniqueName: \"kubernetes.io/projected/71bd479f-89f2-4c1a-8215-d5e488a875dc-kube-api-access-mqqbz\") pod \"payload-processing-external-model-e2e-extmod-a-0f5081-65f97kjsn\" (UID: \"71bd479f-89f2-4c1a-8215-d5e488a875dc\") " pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f97kjsn"
Oct 30 07:38:32.361127 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:32.361041    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f97kjsn"
Oct 30 07:38:32.666181 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:32.666151    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f97kjsn"]
Oct 30 07:38:32.746719 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:32.746643    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-586dxf8bw" podUID="b88b16e6-cb7c-492a-be7c-a028acb1bf19" containerName="payload-processing" containerID="cri-o://d30c091386108faa662544aea5670ecc03e0f91bb68796ce1880329912bab484" gracePeriod=25
Oct 30 07:38:33.093543 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:33.090486    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-586dxf8bw"
Oct 30 07:38:33.175556 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:33.175474    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b88b16e6-cb7c-492a-be7c-a028acb1bf19-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b88b16e6-cb7c-492a-be7c-a028acb1bf19-plugins-config-volume\") pod \"b88b16e6-cb7c-492a-be7c-a028acb1bf19\" (UID: \"b88b16e6-cb7c-492a-be7c-a028acb1bf19\") "
Oct 30 07:38:33.175777 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:33.175750    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b88b16e6-cb7c-492a-be7c-a028acb1bf19-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b88b16e6-cb7c-492a-be7c-a028acb1bf19-provider-credentials\") pod \"b88b16e6-cb7c-492a-be7c-a028acb1bf19\" (UID: \"b88b16e6-cb7c-492a-be7c-a028acb1bf19\") "
Oct 30 07:38:33.175868 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:33.175798    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b88b16e6-cb7c-492a-be7c-a028acb1bf19-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b88b16e6-cb7c-492a-be7c-a028acb1bf19-routing-overlay-volume\") pod \"b88b16e6-cb7c-492a-be7c-a028acb1bf19\" (UID: \"b88b16e6-cb7c-492a-be7c-a028acb1bf19\") "
Oct 30 07:38:33.175976 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:33.175957    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b88b16e6-cb7c-492a-be7c-a028acb1bf19-kube-api-access-v9l4d\" (UniqueName: \"kubernetes.io/projected/b88b16e6-cb7c-492a-be7c-a028acb1bf19-kube-api-access-v9l4d\") pod \"b88b16e6-cb7c-492a-be7c-a028acb1bf19\" (UID: \"b88b16e6-cb7c-492a-be7c-a028acb1bf19\") "
Oct 30 07:38:33.176073 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:33.176048    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b88b16e6-cb7c-492a-be7c-a028acb1bf19-plugins-config-volume" pod "b88b16e6-cb7c-492a-be7c-a028acb1bf19" (UID: "b88b16e6-cb7c-492a-be7c-a028acb1bf19"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:38:33.176559 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:33.176534    6224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b88b16e6-cb7c-492a-be7c-a028acb1bf19-plugins-config-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:38:33.176681 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:33.176597    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b88b16e6-cb7c-492a-be7c-a028acb1bf19-routing-overlay-volume" pod "b88b16e6-cb7c-492a-be7c-a028acb1bf19" (UID: "b88b16e6-cb7c-492a-be7c-a028acb1bf19"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:38:33.183729 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:33.183704    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b88b16e6-cb7c-492a-be7c-a028acb1bf19-provider-credentials" pod "b88b16e6-cb7c-492a-be7c-a028acb1bf19" (UID: "b88b16e6-cb7c-492a-be7c-a028acb1bf19"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:38:33.184269 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:33.184248    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b88b16e6-cb7c-492a-be7c-a028acb1bf19-kube-api-access-v9l4d" pod "b88b16e6-cb7c-492a-be7c-a028acb1bf19" (UID: "b88b16e6-cb7c-492a-be7c-a028acb1bf19"). InnerVolumeSpecName "kube-api-access-v9l4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:38:33.277450 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:33.277412    6224 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b88b16e6-cb7c-492a-be7c-a028acb1bf19-provider-credentials\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:38:33.277450 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:33.277445    6224 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b88b16e6-cb7c-492a-be7c-a028acb1bf19-routing-overlay-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:38:33.277658 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:33.277461    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9l4d\" (UniqueName: \"kubernetes.io/projected/b88b16e6-cb7c-492a-be7c-a028acb1bf19-kube-api-access-v9l4d\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:38:33.432760 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:33.432665    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-6b49ljgld"
Oct 30 07:38:33.511987 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:33.511945    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f97kjsn" event={"ID":"71bd479f-89f2-4c1a-8215-d5e488a875dc","Type":"ContainerStarted","Data":"1520070e7cafde221922636e6de598ff1ce160183da634910dd154e9a7f30176"}
Oct 30 07:38:33.512177 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:33.512009    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f97kjsn" event={"ID":"71bd479f-89f2-4c1a-8215-d5e488a875dc","Type":"ContainerStarted","Data":"78d71afce89ea97624cc4662ec11f811bdd1c149901d73cbe607009cd92179e5"}
Oct 30 07:38:33.512177 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:33.512157    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f97kjsn"
Oct 30 07:38:33.513907 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:33.513879    6224 generic.go:356] "Generic (PLEG): container finished" podID="b88b16e6-cb7c-492a-be7c-a028acb1bf19" containerID="d30c091386108faa662544aea5670ecc03e0f91bb68796ce1880329912bab484" exitCode=0
Oct 30 07:38:33.514041 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:33.513927    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-586dxf8bw" event={"ID":"b88b16e6-cb7c-492a-be7c-a028acb1bf19","Type":"ContainerDied","Data":"d30c091386108faa662544aea5670ecc03e0f91bb68796ce1880329912bab484"}
Oct 30 07:38:33.514041 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:33.513938    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-586dxf8bw"
Oct 30 07:38:33.514041 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:33.513969    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-586dxf8bw" event={"ID":"b88b16e6-cb7c-492a-be7c-a028acb1bf19","Type":"ContainerDied","Data":"9fc9523cc8e9b41a5499fb5713327388bd0d1cbb31acaa6861841b0a1c31dc6e"}
Oct 30 07:38:33.514041 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:33.513998    6224 scope.go:122] "RemoveContainer" containerID="d30c091386108faa662544aea5670ecc03e0f91bb68796ce1880329912bab484"
Oct 30 07:38:33.529166 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:33.529139    6224 scope.go:122] "RemoveContainer" containerID="d30c091386108faa662544aea5670ecc03e0f91bb68796ce1880329912bab484"
Oct 30 07:38:33.529488 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:38:33.529458    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d30c091386108faa662544aea5670ecc03e0f91bb68796ce1880329912bab484\": container with ID starting with d30c091386108faa662544aea5670ecc03e0f91bb68796ce1880329912bab484 not found: ID does not exist" containerID="d30c091386108faa662544aea5670ecc03e0f91bb68796ce1880329912bab484"
Oct 30 07:38:33.529615 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:33.529499    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d30c091386108faa662544aea5670ecc03e0f91bb68796ce1880329912bab484"} err="failed to get container status \"d30c091386108faa662544aea5670ecc03e0f91bb68796ce1880329912bab484\": rpc error: code = NotFound desc = could not find container \"d30c091386108faa662544aea5670ecc03e0f91bb68796ce1880329912bab484\": container with ID starting with d30c091386108faa662544aea5670ecc03e0f91bb68796ce1880329912bab484 not found: ID does not exist"
Oct 30 07:38:33.563208 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:33.563165    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f97kjsn" podStartSLOduration=2.563152859 podStartE2EDuration="2.563152859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:38:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:38:33.561126799 +0000 UTC m=+2472.170533761" watchObservedRunningTime="2026-10-30 07:38:33.563152859 +0000 UTC m=+2472.172559868"
Oct 30 07:38:33.581838 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:38:33.580655    6224 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-0f5081/agc-cred-56d4535fe8f4: secret "agc-cred-56d4535fe8f4" not found
Oct 30 07:38:33.581838 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:38:33.580684    6224 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f97kjsn: secret "agc-cred-56d4535fe8f4" not found
Oct 30 07:38:33.581838 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:38:33.580774    6224 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71bd479f-89f2-4c1a-8215-d5e488a875dc-provider-credentials podName:71bd479f-89f2-4c1a-8215-d5e488a875dc nodeName:}" failed. No retries permitted until 2026-10-30 07:38:34.080754043 +0000 UTC m=+2472.690161014 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/71bd479f-89f2-4c1a-8215-d5e488a875dc-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-0f5081-65f97kjsn" (UID: "71bd479f-89f2-4c1a-8215-d5e488a875dc") : secret "agc-cred-56d4535fe8f4" not found
Oct 30 07:38:33.604708 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:33.604679    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-586dxf8bw"]
Oct 30 07:38:33.612763 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:33.612738    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-586dxf8bw"]
Oct 30 07:38:34.087047 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:38:34.087016    6224 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-0f5081/agc-cred-56d4535fe8f4: secret "agc-cred-56d4535fe8f4" not found
Oct 30 07:38:34.087047 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:38:34.087040    6224 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f97kjsn: secret "agc-cred-56d4535fe8f4" not found
Oct 30 07:38:34.087472 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:38:34.087112    6224 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71bd479f-89f2-4c1a-8215-d5e488a875dc-provider-credentials podName:71bd479f-89f2-4c1a-8215-d5e488a875dc nodeName:}" failed. No retries permitted until 2026-10-30 07:38:35.087098326 +0000 UTC m=+2473.696505296 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/71bd479f-89f2-4c1a-8215-d5e488a875dc-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-0f5081-65f97kjsn" (UID: "71bd479f-89f2-4c1a-8215-d5e488a875dc") : secret "agc-cred-56d4535fe8f4" not found
Oct 30 07:38:34.786530 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:34.786492    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-dbc4f8mdw"]
Oct 30 07:38:34.787246 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:34.787224    6224 cpu_manager.go:414] "containerMap: removing container" podUID="b88b16e6-cb7c-492a-be7c-a028acb1bf19" containerName="payload-processing"
Oct 30 07:38:34.787246 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:34.787245    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b88b16e6-cb7c-492a-be7c-a028acb1bf19" containerName="payload-processing"
Oct 30 07:38:34.787428 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:34.787415    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="b88b16e6-cb7c-492a-be7c-a028acb1bf19" containerName="payload-processing"
Oct 30 07:38:34.791751 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:34.791731    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-dbc4f8mdw"
Oct 30 07:38:34.806888 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:34.806862    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/308e35e5-8933-4967-a5aa-0bd40e1ac1ac-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-dbc4f8mdw\" (UID: \"308e35e5-8933-4967-a5aa-0bd40e1ac1ac\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-dbc4f8mdw"
Oct 30 07:38:34.807021 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:34.807000    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/308e35e5-8933-4967-a5aa-0bd40e1ac1ac-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-dbc4f8mdw\" (UID: \"308e35e5-8933-4967-a5aa-0bd40e1ac1ac\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-dbc4f8mdw"
Oct 30 07:38:34.807182 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:34.807161    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g98vl\" (UniqueName: \"kubernetes.io/projected/308e35e5-8933-4967-a5aa-0bd40e1ac1ac-kube-api-access-g98vl\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-dbc4f8mdw\" (UID: \"308e35e5-8933-4967-a5aa-0bd40e1ac1ac\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-dbc4f8mdw"
Oct 30 07:38:34.807301 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:34.807261    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/308e35e5-8933-4967-a5aa-0bd40e1ac1ac-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-dbc4f8mdw\" (UID: \"308e35e5-8933-4967-a5aa-0bd40e1ac1ac\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-dbc4f8mdw"
Oct 30 07:38:34.810041 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:34.810021    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-dbc4f8mdw"]
Oct 30 07:38:34.908201 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:34.908171    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g98vl\" (UniqueName: \"kubernetes.io/projected/308e35e5-8933-4967-a5aa-0bd40e1ac1ac-kube-api-access-g98vl\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-dbc4f8mdw\" (UID: \"308e35e5-8933-4967-a5aa-0bd40e1ac1ac\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-dbc4f8mdw"
Oct 30 07:38:34.908363 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:34.908216    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/308e35e5-8933-4967-a5aa-0bd40e1ac1ac-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-dbc4f8mdw\" (UID: \"308e35e5-8933-4967-a5aa-0bd40e1ac1ac\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-dbc4f8mdw"
Oct 30 07:38:34.908363 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:34.908292    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/308e35e5-8933-4967-a5aa-0bd40e1ac1ac-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-dbc4f8mdw\" (UID: \"308e35e5-8933-4967-a5aa-0bd40e1ac1ac\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-dbc4f8mdw"
Oct 30 07:38:34.908363 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:34.908357    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/308e35e5-8933-4967-a5aa-0bd40e1ac1ac-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-dbc4f8mdw\" (UID: \"308e35e5-8933-4967-a5aa-0bd40e1ac1ac\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-dbc4f8mdw"
Oct 30 07:38:34.909076 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:34.909048    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/308e35e5-8933-4967-a5aa-0bd40e1ac1ac-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-dbc4f8mdw\" (UID: \"308e35e5-8933-4967-a5aa-0bd40e1ac1ac\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-dbc4f8mdw"
Oct 30 07:38:34.909076 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:34.909066    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/308e35e5-8933-4967-a5aa-0bd40e1ac1ac-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-dbc4f8mdw\" (UID: \"308e35e5-8933-4967-a5aa-0bd40e1ac1ac\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-dbc4f8mdw"
Oct 30 07:38:34.910761 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:34.910743    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/308e35e5-8933-4967-a5aa-0bd40e1ac1ac-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-dbc4f8mdw\" (UID: \"308e35e5-8933-4967-a5aa-0bd40e1ac1ac\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-dbc4f8mdw"
Oct 30 07:38:34.933251 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:34.933231    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g98vl\" (UniqueName: \"kubernetes.io/projected/308e35e5-8933-4967-a5aa-0bd40e1ac1ac-kube-api-access-g98vl\") pod \"payload-processing-external-model-e2e-extmod-b-1f34aa-dbc4f8mdw\" (UID: \"308e35e5-8933-4967-a5aa-0bd40e1ac1ac\") " pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-dbc4f8mdw"
Oct 30 07:38:35.105549 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:35.105474    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-dbc4f8mdw"
Oct 30 07:38:35.111244 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:38:35.111219    6224 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-0f5081/agc-cred-56d4535fe8f4: secret "agc-cred-56d4535fe8f4" not found
Oct 30 07:38:35.111244 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:38:35.111238    6224 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f97kjsn: secret "agc-cred-56d4535fe8f4" not found
Oct 30 07:38:35.111425 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:38:35.111287    6224 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71bd479f-89f2-4c1a-8215-d5e488a875dc-provider-credentials podName:71bd479f-89f2-4c1a-8215-d5e488a875dc nodeName:}" failed. No retries permitted until 2026-10-30 07:38:37.111273745 +0000 UTC m=+2475.720680693 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/71bd479f-89f2-4c1a-8215-d5e488a875dc-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-0f5081-65f97kjsn" (UID: "71bd479f-89f2-4c1a-8215-d5e488a875dc") : secret "agc-cred-56d4535fe8f4" not found
Oct 30 07:38:35.409634 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:38:35.409606    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod308e35e5_8933_4967_a5aa_0bd40e1ac1ac.slice/crio-522e33d53d7bd21735e76122b981184adac119b9f526d447966de9f748eb9042 WatchSource:0}: Error finding container 522e33d53d7bd21735e76122b981184adac119b9f526d447966de9f748eb9042: Status 404 returned error can't find the container with id 522e33d53d7bd21735e76122b981184adac119b9f526d447966de9f748eb9042
Oct 30 07:38:35.412213 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:35.412193    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-dbc4f8mdw"]
Oct 30 07:38:35.503731 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:35.503705    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b88b16e6-cb7c-492a-be7c-a028acb1bf19" path="/var/lib/kubelet/pods/b88b16e6-cb7c-492a-be7c-a028acb1bf19/volumes"
Oct 30 07:38:35.533312 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:35.533280    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-dbc4f8mdw" event={"ID":"308e35e5-8933-4967-a5aa-0bd40e1ac1ac","Type":"ContainerStarted","Data":"768d2619f2d3e617835b4a4a4874895fc180d3dc58b679da538e311c025b121c"}
Oct 30 07:38:35.533422 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:35.533336    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-dbc4f8mdw" event={"ID":"308e35e5-8933-4967-a5aa-0bd40e1ac1ac","Type":"ContainerStarted","Data":"522e33d53d7bd21735e76122b981184adac119b9f526d447966de9f748eb9042"}
Oct 30 07:38:35.533422 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:35.533375    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-dbc4f8mdw"
Oct 30 07:38:35.581524 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:35.581481    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-dbc4f8mdw" podStartSLOduration=1.581467913 podStartE2EDuration="1.581467913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:38:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:38:35.580051304 +0000 UTC m=+2474.189458271" watchObservedRunningTime="2026-10-30 07:38:35.581467913 +0000 UTC m=+2474.190874883"
Oct 30 07:38:37.139119 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:38:37.139085    6224 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-0f5081/agc-cred-56d4535fe8f4: secret "agc-cred-56d4535fe8f4" not found
Oct 30 07:38:37.139119 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:38:37.139112    6224 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f97kjsn: secret "agc-cred-56d4535fe8f4" not found
Oct 30 07:38:37.139550 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:38:37.139168    6224 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71bd479f-89f2-4c1a-8215-d5e488a875dc-provider-credentials podName:71bd479f-89f2-4c1a-8215-d5e488a875dc nodeName:}" failed. No retries permitted until 2026-10-30 07:38:41.139151471 +0000 UTC m=+2479.748558434 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/71bd479f-89f2-4c1a-8215-d5e488a875dc-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-0f5081-65f97kjsn" (UID: "71bd479f-89f2-4c1a-8215-d5e488a875dc") : secret "agc-cred-56d4535fe8f4" not found
Oct 30 07:38:37.908195 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:37.908160    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-9d6bcf8f-srvqq"]
Oct 30 07:38:37.912953 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:37.912934    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-9d6bcf8f-srvqq"
Oct 30 07:38:37.931896 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:37.931869    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-9d6bcf8f-srvqq"]
Oct 30 07:38:37.942426 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:37.942390    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-fb42c6f07a14\""
Oct 30 07:38:37.942597 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:37.942562    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 30 07:38:37.942597 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:37.942581    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-lwdqb\""
Oct 30 07:38:37.942731 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:37.942696    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 30 07:38:37.944005 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:37.943989    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 30 07:38:37.944104 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:37.944011    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 30 07:38:37.947758 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:37.947738    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d95a630d-751d-4171-8a10-331cc3382c23-provider-credentials\") pod \"payload-processing-external-model-9d6bcf8f-srvqq\" (UID: \"d95a630d-751d-4171-8a10-331cc3382c23\") " pod="models-as-a-service/payload-processing-external-model-9d6bcf8f-srvqq"
Oct 30 07:38:37.947830 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:37.947770    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zz5s\" (UniqueName: \"kubernetes.io/projected/d95a630d-751d-4171-8a10-331cc3382c23-kube-api-access-9zz5s\") pod \"payload-processing-external-model-9d6bcf8f-srvqq\" (UID: \"d95a630d-751d-4171-8a10-331cc3382c23\") " pod="models-as-a-service/payload-processing-external-model-9d6bcf8f-srvqq"
Oct 30 07:38:37.947868 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:37.947798    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d95a630d-751d-4171-8a10-331cc3382c23-routing-overlay-volume\") pod \"payload-processing-external-model-9d6bcf8f-srvqq\" (UID: \"d95a630d-751d-4171-8a10-331cc3382c23\") " pod="models-as-a-service/payload-processing-external-model-9d6bcf8f-srvqq"
Oct 30 07:38:37.948005 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:37.947989    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d95a630d-751d-4171-8a10-331cc3382c23-plugins-config-volume\") pod \"payload-processing-external-model-9d6bcf8f-srvqq\" (UID: \"d95a630d-751d-4171-8a10-331cc3382c23\") " pod="models-as-a-service/payload-processing-external-model-9d6bcf8f-srvqq"
Oct 30 07:38:38.049924 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:38.049884    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d95a630d-751d-4171-8a10-331cc3382c23-provider-credentials\") pod \"payload-processing-external-model-9d6bcf8f-srvqq\" (UID: \"d95a630d-751d-4171-8a10-331cc3382c23\") " pod="models-as-a-service/payload-processing-external-model-9d6bcf8f-srvqq"
Oct 30 07:38:38.050105 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:38.049939    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zz5s\" (UniqueName: \"kubernetes.io/projected/d95a630d-751d-4171-8a10-331cc3382c23-kube-api-access-9zz5s\") pod \"payload-processing-external-model-9d6bcf8f-srvqq\" (UID: \"d95a630d-751d-4171-8a10-331cc3382c23\") " pod="models-as-a-service/payload-processing-external-model-9d6bcf8f-srvqq"
Oct 30 07:38:38.050105 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:38.049980    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d95a630d-751d-4171-8a10-331cc3382c23-routing-overlay-volume\") pod \"payload-processing-external-model-9d6bcf8f-srvqq\" (UID: \"d95a630d-751d-4171-8a10-331cc3382c23\") " pod="models-as-a-service/payload-processing-external-model-9d6bcf8f-srvqq"
Oct 30 07:38:38.050240 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:38.050106    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d95a630d-751d-4171-8a10-331cc3382c23-plugins-config-volume\") pod \"payload-processing-external-model-9d6bcf8f-srvqq\" (UID: \"d95a630d-751d-4171-8a10-331cc3382c23\") " pod="models-as-a-service/payload-processing-external-model-9d6bcf8f-srvqq"
Oct 30 07:38:38.050887 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:38.050862    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d95a630d-751d-4171-8a10-331cc3382c23-routing-overlay-volume\") pod \"payload-processing-external-model-9d6bcf8f-srvqq\" (UID: \"d95a630d-751d-4171-8a10-331cc3382c23\") " pod="models-as-a-service/payload-processing-external-model-9d6bcf8f-srvqq"
Oct 30 07:38:38.051251 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:38.051212    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d95a630d-751d-4171-8a10-331cc3382c23-plugins-config-volume\") pod \"payload-processing-external-model-9d6bcf8f-srvqq\" (UID: \"d95a630d-751d-4171-8a10-331cc3382c23\") " pod="models-as-a-service/payload-processing-external-model-9d6bcf8f-srvqq"
Oct 30 07:38:38.054404 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:38.054378    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d95a630d-751d-4171-8a10-331cc3382c23-provider-credentials\") pod \"payload-processing-external-model-9d6bcf8f-srvqq\" (UID: \"d95a630d-751d-4171-8a10-331cc3382c23\") " pod="models-as-a-service/payload-processing-external-model-9d6bcf8f-srvqq"
Oct 30 07:38:38.075252 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:38.075221    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zz5s\" (UniqueName: \"kubernetes.io/projected/d95a630d-751d-4171-8a10-331cc3382c23-kube-api-access-9zz5s\") pod \"payload-processing-external-model-9d6bcf8f-srvqq\" (UID: \"d95a630d-751d-4171-8a10-331cc3382c23\") " pod="models-as-a-service/payload-processing-external-model-9d6bcf8f-srvqq"
Oct 30 07:38:38.233135 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:38.230793    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-9d6bcf8f-srvqq"
Oct 30 07:38:38.540536 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:38:38.540500    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd95a630d_751d_4171_8a10_331cc3382c23.slice/crio-ae871faacf6d319105c98e816b4872d4b4f05c01ce8d28f104b9252d7511426f WatchSource:0}: Error finding container ae871faacf6d319105c98e816b4872d4b4f05c01ce8d28f104b9252d7511426f: Status 404 returned error can't find the container with id ae871faacf6d319105c98e816b4872d4b4f05c01ce8d28f104b9252d7511426f
Oct 30 07:38:38.542026 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:38.541998    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-9d6bcf8f-srvqq"]
Oct 30 07:38:38.558564 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:38.558537    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-9d6bcf8f-srvqq" event={"ID":"d95a630d-751d-4171-8a10-331cc3382c23","Type":"ContainerStarted","Data":"ae871faacf6d319105c98e816b4872d4b4f05c01ce8d28f104b9252d7511426f"}
Oct 30 07:38:39.567005 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:39.566966    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-9d6bcf8f-srvqq" event={"ID":"d95a630d-751d-4171-8a10-331cc3382c23","Type":"ContainerStarted","Data":"14f63fab7cb70b85991ac04916137260570034dab9a503b1969734eeed998470"}
Oct 30 07:38:39.567401 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:39.567092    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-9d6bcf8f-srvqq"
Oct 30 07:38:39.615401 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:39.615353    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-9d6bcf8f-srvqq" podStartSLOduration=2.615337839 podStartE2EDuration="2.615337839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:38:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:38:39.612884211 +0000 UTC m=+2478.222291181" watchObservedRunningTime="2026-10-30 07:38:39.615337839 +0000 UTC m=+2478.224744798"
Oct 30 07:38:41.196608 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:38:41.196569    6224 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-0f5081/agc-cred-56d4535fe8f4: secret "agc-cred-56d4535fe8f4" not found
Oct 30 07:38:41.196608 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:38:41.196605    6224 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f97kjsn: secret "agc-cred-56d4535fe8f4" not found
Oct 30 07:38:41.197079 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:38:41.196670    6224 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71bd479f-89f2-4c1a-8215-d5e488a875dc-provider-credentials podName:71bd479f-89f2-4c1a-8215-d5e488a875dc nodeName:}" failed. No retries permitted until 2026-10-30 07:38:49.196652101 +0000 UTC m=+2487.806059050 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/71bd479f-89f2-4c1a-8215-d5e488a875dc-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-0f5081-65f97kjsn" (UID: "71bd479f-89f2-4c1a-8215-d5e488a875dc") : secret "agc-cred-56d4535fe8f4" not found
Oct 30 07:38:44.529600 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:44.529569    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f97kjsn"
Oct 30 07:38:45.726407 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:45.726368    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-6b49ljgld"]
Oct 30 07:38:45.737956 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:45.737925    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-dbc4f8mdw"]
Oct 30 07:38:45.741639 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:45.741618    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-dbc4f8mdw"
Oct 30 07:38:48.673879 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:48.673847    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f97kjsn"]
Oct 30 07:38:49.232381 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:38:49.232334    6224 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-0f5081/agc-cred-56d4535fe8f4: secret "agc-cred-56d4535fe8f4" not found
Oct 30 07:38:49.232381 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:38:49.232373    6224 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f97kjsn: secret "agc-cred-56d4535fe8f4" not found
Oct 30 07:38:49.232607 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:38:49.232432    6224 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71bd479f-89f2-4c1a-8215-d5e488a875dc-provider-credentials podName:71bd479f-89f2-4c1a-8215-d5e488a875dc nodeName:}" failed. No retries permitted until 2026-10-30 07:39:05.232418402 +0000 UTC m=+2503.841825363 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/71bd479f-89f2-4c1a-8215-d5e488a875dc-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-0f5081-65f97kjsn" (UID: "71bd479f-89f2-4c1a-8215-d5e488a875dc") : secret "agc-cred-56d4535fe8f4" not found
Oct 30 07:38:50.577970 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:50.577941    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-9d6bcf8f-srvqq"
Oct 30 07:38:50.727533 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:50.727491    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-6b49ljgld" podUID="de4a9e38-8d9a-4f17-bb63-dc29ef3fe3da" containerName="payload-processing" containerID="cri-o://4622f4002d8c4f01e3cbaa16b509b3151b197054178f79355c610e6853e09836" gracePeriod=25
Oct 30 07:38:50.739139 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:50.739099    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-dbc4f8mdw" podUID="308e35e5-8933-4967-a5aa-0bd40e1ac1ac" containerName="payload-processing" containerID="cri-o://768d2619f2d3e617835b4a4a4874895fc180d3dc58b679da538e311c025b121c" gracePeriod=25
Oct 30 07:38:51.038483 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:51.038455    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-6b49ljgld"
Oct 30 07:38:51.057911 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:51.057888    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-dbc4f8mdw"
Oct 30 07:38:51.058303 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:51.058289    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de4a9e38-8d9a-4f17-bb63-dc29ef3fe3da-provider-credentials\" (UniqueName: \"kubernetes.io/projected/de4a9e38-8d9a-4f17-bb63-dc29ef3fe3da-provider-credentials\") pod \"de4a9e38-8d9a-4f17-bb63-dc29ef3fe3da\" (UID: \"de4a9e38-8d9a-4f17-bb63-dc29ef3fe3da\") "
Oct 30 07:38:51.058511 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:51.058486    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de4a9e38-8d9a-4f17-bb63-dc29ef3fe3da-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/de4a9e38-8d9a-4f17-bb63-dc29ef3fe3da-routing-overlay-volume\") pod \"de4a9e38-8d9a-4f17-bb63-dc29ef3fe3da\" (UID: \"de4a9e38-8d9a-4f17-bb63-dc29ef3fe3da\") "
Oct 30 07:38:51.058593 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:51.058527    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de4a9e38-8d9a-4f17-bb63-dc29ef3fe3da-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de4a9e38-8d9a-4f17-bb63-dc29ef3fe3da-plugins-config-volume\") pod \"de4a9e38-8d9a-4f17-bb63-dc29ef3fe3da\" (UID: \"de4a9e38-8d9a-4f17-bb63-dc29ef3fe3da\") "
Oct 30 07:38:51.058593 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:51.058576    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de4a9e38-8d9a-4f17-bb63-dc29ef3fe3da-kube-api-access-7s9mc\" (UniqueName: \"kubernetes.io/projected/de4a9e38-8d9a-4f17-bb63-dc29ef3fe3da-kube-api-access-7s9mc\") pod \"de4a9e38-8d9a-4f17-bb63-dc29ef3fe3da\" (UID: \"de4a9e38-8d9a-4f17-bb63-dc29ef3fe3da\") "
Oct 30 07:38:51.058908 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:51.058864    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de4a9e38-8d9a-4f17-bb63-dc29ef3fe3da-routing-overlay-volume" pod "de4a9e38-8d9a-4f17-bb63-dc29ef3fe3da" (UID: "de4a9e38-8d9a-4f17-bb63-dc29ef3fe3da"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:38:51.059035 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:51.059011    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de4a9e38-8d9a-4f17-bb63-dc29ef3fe3da-plugins-config-volume" pod "de4a9e38-8d9a-4f17-bb63-dc29ef3fe3da" (UID: "de4a9e38-8d9a-4f17-bb63-dc29ef3fe3da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:38:51.059245 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:51.059227    6224 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/de4a9e38-8d9a-4f17-bb63-dc29ef3fe3da-routing-overlay-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:38:51.059291 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:51.059254    6224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de4a9e38-8d9a-4f17-bb63-dc29ef3fe3da-plugins-config-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:38:51.060366 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:51.060343    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de4a9e38-8d9a-4f17-bb63-dc29ef3fe3da-provider-credentials" pod "de4a9e38-8d9a-4f17-bb63-dc29ef3fe3da" (UID: "de4a9e38-8d9a-4f17-bb63-dc29ef3fe3da"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:38:51.060497 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:51.060480    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de4a9e38-8d9a-4f17-bb63-dc29ef3fe3da-kube-api-access-7s9mc" pod "de4a9e38-8d9a-4f17-bb63-dc29ef3fe3da" (UID: "de4a9e38-8d9a-4f17-bb63-dc29ef3fe3da"). InnerVolumeSpecName "kube-api-access-7s9mc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:38:51.159801 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:51.159726    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/308e35e5-8933-4967-a5aa-0bd40e1ac1ac-provider-credentials\" (UniqueName: \"kubernetes.io/projected/308e35e5-8933-4967-a5aa-0bd40e1ac1ac-provider-credentials\") pod \"308e35e5-8933-4967-a5aa-0bd40e1ac1ac\" (UID: \"308e35e5-8933-4967-a5aa-0bd40e1ac1ac\") "
Oct 30 07:38:51.159981 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:51.159915    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/308e35e5-8933-4967-a5aa-0bd40e1ac1ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/308e35e5-8933-4967-a5aa-0bd40e1ac1ac-plugins-config-volume\") pod \"308e35e5-8933-4967-a5aa-0bd40e1ac1ac\" (UID: \"308e35e5-8933-4967-a5aa-0bd40e1ac1ac\") "
Oct 30 07:38:51.159981 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:51.159948    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/308e35e5-8933-4967-a5aa-0bd40e1ac1ac-kube-api-access-g98vl\" (UniqueName: \"kubernetes.io/projected/308e35e5-8933-4967-a5aa-0bd40e1ac1ac-kube-api-access-g98vl\") pod \"308e35e5-8933-4967-a5aa-0bd40e1ac1ac\" (UID: \"308e35e5-8933-4967-a5aa-0bd40e1ac1ac\") "
Oct 30 07:38:51.159981 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:51.159975    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/308e35e5-8933-4967-a5aa-0bd40e1ac1ac-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/308e35e5-8933-4967-a5aa-0bd40e1ac1ac-routing-overlay-volume\") pod \"308e35e5-8933-4967-a5aa-0bd40e1ac1ac\" (UID: \"308e35e5-8933-4967-a5aa-0bd40e1ac1ac\") "
Oct 30 07:38:51.160395 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:51.160367    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/308e35e5-8933-4967-a5aa-0bd40e1ac1ac-plugins-config-volume" pod "308e35e5-8933-4967-a5aa-0bd40e1ac1ac" (UID: "308e35e5-8933-4967-a5aa-0bd40e1ac1ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:38:51.160395 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:51.160384    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/308e35e5-8933-4967-a5aa-0bd40e1ac1ac-routing-overlay-volume" pod "308e35e5-8933-4967-a5aa-0bd40e1ac1ac" (UID: "308e35e5-8933-4967-a5aa-0bd40e1ac1ac"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:38:51.160599 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:51.160525    6224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/308e35e5-8933-4967-a5aa-0bd40e1ac1ac-plugins-config-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:38:51.160599 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:51.160539    6224 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/308e35e5-8933-4967-a5aa-0bd40e1ac1ac-routing-overlay-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:38:51.160599 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:51.160553    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s9mc\" (UniqueName: \"kubernetes.io/projected/de4a9e38-8d9a-4f17-bb63-dc29ef3fe3da-kube-api-access-7s9mc\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:38:51.160599 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:51.160568    6224 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/de4a9e38-8d9a-4f17-bb63-dc29ef3fe3da-provider-credentials\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:38:51.162452 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:51.162429    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/308e35e5-8933-4967-a5aa-0bd40e1ac1ac-provider-credentials" pod "308e35e5-8933-4967-a5aa-0bd40e1ac1ac" (UID: "308e35e5-8933-4967-a5aa-0bd40e1ac1ac"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:38:51.162569 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:51.162460    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/308e35e5-8933-4967-a5aa-0bd40e1ac1ac-kube-api-access-g98vl" pod "308e35e5-8933-4967-a5aa-0bd40e1ac1ac" (UID: "308e35e5-8933-4967-a5aa-0bd40e1ac1ac"). InnerVolumeSpecName "kube-api-access-g98vl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:38:51.261728 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:51.261699    6224 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/308e35e5-8933-4967-a5aa-0bd40e1ac1ac-provider-credentials\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:38:51.261728 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:51.261723    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g98vl\" (UniqueName: \"kubernetes.io/projected/308e35e5-8933-4967-a5aa-0bd40e1ac1ac-kube-api-access-g98vl\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:38:51.669308 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:51.669277    6224 generic.go:356] "Generic (PLEG): container finished" podID="308e35e5-8933-4967-a5aa-0bd40e1ac1ac" containerID="768d2619f2d3e617835b4a4a4874895fc180d3dc58b679da538e311c025b121c" exitCode=0
Oct 30 07:38:51.669702 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:51.669338    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-dbc4f8mdw"
Oct 30 07:38:51.669702 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:51.669350    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-dbc4f8mdw" event={"ID":"308e35e5-8933-4967-a5aa-0bd40e1ac1ac","Type":"ContainerDied","Data":"768d2619f2d3e617835b4a4a4874895fc180d3dc58b679da538e311c025b121c"}
Oct 30 07:38:51.669702 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:51.669405    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-dbc4f8mdw" event={"ID":"308e35e5-8933-4967-a5aa-0bd40e1ac1ac","Type":"ContainerDied","Data":"522e33d53d7bd21735e76122b981184adac119b9f526d447966de9f748eb9042"}
Oct 30 07:38:51.669702 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:51.669432    6224 scope.go:122] "RemoveContainer" containerID="768d2619f2d3e617835b4a4a4874895fc180d3dc58b679da538e311c025b121c"
Oct 30 07:38:51.670918 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:51.670898    6224 generic.go:356] "Generic (PLEG): container finished" podID="de4a9e38-8d9a-4f17-bb63-dc29ef3fe3da" containerID="4622f4002d8c4f01e3cbaa16b509b3151b197054178f79355c610e6853e09836" exitCode=0
Oct 30 07:38:51.671034 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:51.670955    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-6b49ljgld"
Oct 30 07:38:51.671034 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:51.670986    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-6b49ljgld" event={"ID":"de4a9e38-8d9a-4f17-bb63-dc29ef3fe3da","Type":"ContainerDied","Data":"4622f4002d8c4f01e3cbaa16b509b3151b197054178f79355c610e6853e09836"}
Oct 30 07:38:51.671034 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:51.671029    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-6b49ljgld" event={"ID":"de4a9e38-8d9a-4f17-bb63-dc29ef3fe3da","Type":"ContainerDied","Data":"98df27d9b36ae606a440bc27b86bbbedb0d9af299e8ffdf95f3f9bcce2abb608"}
Oct 30 07:38:51.679592 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:51.679576    6224 scope.go:122] "RemoveContainer" containerID="768d2619f2d3e617835b4a4a4874895fc180d3dc58b679da538e311c025b121c"
Oct 30 07:38:51.679898 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:38:51.679876    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"768d2619f2d3e617835b4a4a4874895fc180d3dc58b679da538e311c025b121c\": container with ID starting with 768d2619f2d3e617835b4a4a4874895fc180d3dc58b679da538e311c025b121c not found: ID does not exist" containerID="768d2619f2d3e617835b4a4a4874895fc180d3dc58b679da538e311c025b121c"
Oct 30 07:38:51.679964 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:51.679914    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"768d2619f2d3e617835b4a4a4874895fc180d3dc58b679da538e311c025b121c"} err="failed to get container status \"768d2619f2d3e617835b4a4a4874895fc180d3dc58b679da538e311c025b121c\": rpc error: code = NotFound desc = could not find container \"768d2619f2d3e617835b4a4a4874895fc180d3dc58b679da538e311c025b121c\": container with ID starting with 768d2619f2d3e617835b4a4a4874895fc180d3dc58b679da538e311c025b121c not found: ID does not exist"
Oct 30 07:38:51.679964 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:51.679931    6224 scope.go:122] "RemoveContainer" containerID="4622f4002d8c4f01e3cbaa16b509b3151b197054178f79355c610e6853e09836"
Oct 30 07:38:51.688956 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:51.688940    6224 scope.go:122] "RemoveContainer" containerID="4622f4002d8c4f01e3cbaa16b509b3151b197054178f79355c610e6853e09836"
Oct 30 07:38:51.689206 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:38:51.689187    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4622f4002d8c4f01e3cbaa16b509b3151b197054178f79355c610e6853e09836\": container with ID starting with 4622f4002d8c4f01e3cbaa16b509b3151b197054178f79355c610e6853e09836 not found: ID does not exist" containerID="4622f4002d8c4f01e3cbaa16b509b3151b197054178f79355c610e6853e09836"
Oct 30 07:38:51.689258 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:51.689217    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4622f4002d8c4f01e3cbaa16b509b3151b197054178f79355c610e6853e09836"} err="failed to get container status \"4622f4002d8c4f01e3cbaa16b509b3151b197054178f79355c610e6853e09836\": rpc error: code = NotFound desc = could not find container \"4622f4002d8c4f01e3cbaa16b509b3151b197054178f79355c610e6853e09836\": container with ID starting with 4622f4002d8c4f01e3cbaa16b509b3151b197054178f79355c610e6853e09836 not found: ID does not exist"
Oct 30 07:38:51.727630 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:51.727611    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-6b49ljgld"]
Oct 30 07:38:51.741268 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:51.741245    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-6b49ljgld"]
Oct 30 07:38:51.781156 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:51.781136    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-dbc4f8mdw"]
Oct 30 07:38:51.784152 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:51.784134    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-1f34aa/payload-processing-external-model-e2e-extmod-b-1f34aa-dbc4f8mdw"]
Oct 30 07:38:53.504293 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:53.504261    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="308e35e5-8933-4967-a5aa-0bd40e1ac1ac" path="/var/lib/kubelet/pods/308e35e5-8933-4967-a5aa-0bd40e1ac1ac/volumes"
Oct 30 07:38:53.504710 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:53.504693    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de4a9e38-8d9a-4f17-bb63-dc29ef3fe3da" path="/var/lib/kubelet/pods/de4a9e38-8d9a-4f17-bb63-dc29ef3fe3da/volumes"
Oct 30 07:38:53.674485 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:53.674449    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f97kjsn" podUID="71bd479f-89f2-4c1a-8215-d5e488a875dc" containerName="payload-processing" containerID="cri-o://1520070e7cafde221922636e6de598ff1ce160183da634910dd154e9a7f30176" gracePeriod=25
Oct 30 07:38:53.978328 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:53.978295    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f97kjsn"
Oct 30 07:38:54.103075 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:54.102989    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71bd479f-89f2-4c1a-8215-d5e488a875dc-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/71bd479f-89f2-4c1a-8215-d5e488a875dc-routing-overlay-volume\") pod \"71bd479f-89f2-4c1a-8215-d5e488a875dc\" (UID: \"71bd479f-89f2-4c1a-8215-d5e488a875dc\") "
Oct 30 07:38:54.103075 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:54.103070    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71bd479f-89f2-4c1a-8215-d5e488a875dc-kube-api-access-mqqbz\" (UniqueName: \"kubernetes.io/projected/71bd479f-89f2-4c1a-8215-d5e488a875dc-kube-api-access-mqqbz\") pod \"71bd479f-89f2-4c1a-8215-d5e488a875dc\" (UID: \"71bd479f-89f2-4c1a-8215-d5e488a875dc\") "
Oct 30 07:38:54.103299 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:54.103145    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71bd479f-89f2-4c1a-8215-d5e488a875dc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71bd479f-89f2-4c1a-8215-d5e488a875dc-plugins-config-volume\") pod \"71bd479f-89f2-4c1a-8215-d5e488a875dc\" (UID: \"71bd479f-89f2-4c1a-8215-d5e488a875dc\") "
Oct 30 07:38:54.103389 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:54.103364    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71bd479f-89f2-4c1a-8215-d5e488a875dc-provider-credentials\" (UniqueName: \"kubernetes.io/projected/71bd479f-89f2-4c1a-8215-d5e488a875dc-provider-credentials\") pod \"71bd479f-89f2-4c1a-8215-d5e488a875dc\" (UID: \"71bd479f-89f2-4c1a-8215-d5e488a875dc\") "
Oct 30 07:38:54.103524 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:54.103426    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71bd479f-89f2-4c1a-8215-d5e488a875dc-routing-overlay-volume" pod "71bd479f-89f2-4c1a-8215-d5e488a875dc" (UID: "71bd479f-89f2-4c1a-8215-d5e488a875dc"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:38:54.103582 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:54.103527    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71bd479f-89f2-4c1a-8215-d5e488a875dc-plugins-config-volume" pod "71bd479f-89f2-4c1a-8215-d5e488a875dc" (UID: "71bd479f-89f2-4c1a-8215-d5e488a875dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:38:54.103944 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:54.103922    6224 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/71bd479f-89f2-4c1a-8215-d5e488a875dc-routing-overlay-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:38:54.104038 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:54.103949    6224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71bd479f-89f2-4c1a-8215-d5e488a875dc-plugins-config-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:38:54.105427 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:54.105402    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71bd479f-89f2-4c1a-8215-d5e488a875dc-kube-api-access-mqqbz" pod "71bd479f-89f2-4c1a-8215-d5e488a875dc" (UID: "71bd479f-89f2-4c1a-8215-d5e488a875dc"). InnerVolumeSpecName "kube-api-access-mqqbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:38:54.105427 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:54.105418    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71bd479f-89f2-4c1a-8215-d5e488a875dc-provider-credentials" pod "71bd479f-89f2-4c1a-8215-d5e488a875dc" (UID: "71bd479f-89f2-4c1a-8215-d5e488a875dc"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:38:54.205107 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:54.205077    6224 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/71bd479f-89f2-4c1a-8215-d5e488a875dc-provider-credentials\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:38:54.205107 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:54.205103    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqqbz\" (UniqueName: \"kubernetes.io/projected/71bd479f-89f2-4c1a-8215-d5e488a875dc-kube-api-access-mqqbz\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:38:54.695327 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:54.695288    6224 generic.go:356] "Generic (PLEG): container finished" podID="71bd479f-89f2-4c1a-8215-d5e488a875dc" containerID="1520070e7cafde221922636e6de598ff1ce160183da634910dd154e9a7f30176" exitCode=0
Oct 30 07:38:54.695736 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:54.695349    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f97kjsn"
Oct 30 07:38:54.695736 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:54.695370    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f97kjsn" event={"ID":"71bd479f-89f2-4c1a-8215-d5e488a875dc","Type":"ContainerDied","Data":"1520070e7cafde221922636e6de598ff1ce160183da634910dd154e9a7f30176"}
Oct 30 07:38:54.695736 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:54.695455    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f97kjsn" event={"ID":"71bd479f-89f2-4c1a-8215-d5e488a875dc","Type":"ContainerDied","Data":"78d71afce89ea97624cc4662ec11f811bdd1c149901d73cbe607009cd92179e5"}
Oct 30 07:38:54.695736 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:54.695478    6224 scope.go:122] "RemoveContainer" containerID="1520070e7cafde221922636e6de598ff1ce160183da634910dd154e9a7f30176"
Oct 30 07:38:54.707096 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:54.707074    6224 scope.go:122] "RemoveContainer" containerID="1520070e7cafde221922636e6de598ff1ce160183da634910dd154e9a7f30176"
Oct 30 07:38:54.707384 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:38:54.707359    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1520070e7cafde221922636e6de598ff1ce160183da634910dd154e9a7f30176\": container with ID starting with 1520070e7cafde221922636e6de598ff1ce160183da634910dd154e9a7f30176 not found: ID does not exist" containerID="1520070e7cafde221922636e6de598ff1ce160183da634910dd154e9a7f30176"
Oct 30 07:38:54.707432 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:54.707392    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1520070e7cafde221922636e6de598ff1ce160183da634910dd154e9a7f30176"} err="failed to get container status \"1520070e7cafde221922636e6de598ff1ce160183da634910dd154e9a7f30176\": rpc error: code = NotFound desc = could not find container \"1520070e7cafde221922636e6de598ff1ce160183da634910dd154e9a7f30176\": container with ID starting with 1520070e7cafde221922636e6de598ff1ce160183da634910dd154e9a7f30176 not found: ID does not exist"
Oct 30 07:38:54.755780 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:54.755734    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f97kjsn"]
Oct 30 07:38:54.769539 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:54.769509    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-0f5081/payload-processing-external-model-e2e-extmod-a-0f5081-65f97kjsn"]
Oct 30 07:38:55.504550 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:38:55.504520    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71bd479f-89f2-4c1a-8215-d5e488a875dc" path="/var/lib/kubelet/pods/71bd479f-89f2-4c1a-8215-d5e488a875dc/volumes"
Oct 30 07:39:00.666728 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:00.666690    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-8e7948/payload-processing-external-model-e2e-extmod-a-8e7948-6d8dbfhct"]
Oct 30 07:39:00.667346 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:00.667331    6224 cpu_manager.go:414] "containerMap: removing container" podUID="71bd479f-89f2-4c1a-8215-d5e488a875dc" containerName="payload-processing"
Oct 30 07:39:00.667394 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:00.667347    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71bd479f-89f2-4c1a-8215-d5e488a875dc" containerName="payload-processing"
Oct 30 07:39:00.667394 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:00.667362    6224 cpu_manager.go:414] "containerMap: removing container" podUID="308e35e5-8933-4967-a5aa-0bd40e1ac1ac" containerName="payload-processing"
Oct 30 07:39:00.667394 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:00.667368    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="308e35e5-8933-4967-a5aa-0bd40e1ac1ac" containerName="payload-processing"
Oct 30 07:39:00.667394 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:00.667380    6224 cpu_manager.go:414] "containerMap: removing container" podUID="de4a9e38-8d9a-4f17-bb63-dc29ef3fe3da" containerName="payload-processing"
Oct 30 07:39:00.667394 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:00.667385    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de4a9e38-8d9a-4f17-bb63-dc29ef3fe3da" containerName="payload-processing"
Oct 30 07:39:00.667544 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:00.667514    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="71bd479f-89f2-4c1a-8215-d5e488a875dc" containerName="payload-processing"
Oct 30 07:39:00.667544 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:00.667524    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="308e35e5-8933-4967-a5aa-0bd40e1ac1ac" containerName="payload-processing"
Oct 30 07:39:00.667544 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:00.667532    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="de4a9e38-8d9a-4f17-bb63-dc29ef3fe3da" containerName="payload-processing"
Oct 30 07:39:00.672325 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:00.672299    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8e7948/payload-processing-external-model-e2e-extmod-a-8e7948-6d8dbfhct"
Oct 30 07:39:00.706211 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:00.704263    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8e7948\"/\"agc-cred-f002320cf03f\""
Oct 30 07:39:00.706211 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:00.704376    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8e7948\"/\"payload-processing-external-model-plugins-e2e-extmod-a-8e7948\""
Oct 30 07:39:00.706211 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:00.704462    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8e7948\"/\"kube-root-ca.crt\""
Oct 30 07:39:00.706211 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:00.704558    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8e7948\"/\"routing-overlay\""
Oct 30 07:39:00.706211 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:00.704623    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8e7948\"/\"openshift-service-ca.crt\""
Oct 30 07:39:00.706211 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:00.704949    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8e7948\"/\"payload-processing-external-model-e2e--8346c882-dockercfg-9njlq\""
Oct 30 07:39:00.711799 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:00.711768    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8e7948/payload-processing-external-model-e2e-extmod-a-8e7948-6d8dbfhct"]
Oct 30 07:39:00.816842 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:00.810613    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/01fea4c7-b874-49e8-a8d5-c0b9b50413fb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8e7948-6d8dbfhct\" (UID: \"01fea4c7-b874-49e8-a8d5-c0b9b50413fb\") " pod="ai-tenant-e2e-extmod-a-8e7948/payload-processing-external-model-e2e-extmod-a-8e7948-6d8dbfhct"
Oct 30 07:39:00.825785 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:00.825744    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqd9b\" (UniqueName: \"kubernetes.io/projected/01fea4c7-b874-49e8-a8d5-c0b9b50413fb-kube-api-access-sqd9b\") pod \"payload-processing-external-model-e2e-extmod-a-8e7948-6d8dbfhct\" (UID: \"01fea4c7-b874-49e8-a8d5-c0b9b50413fb\") " pod="ai-tenant-e2e-extmod-a-8e7948/payload-processing-external-model-e2e-extmod-a-8e7948-6d8dbfhct"
Oct 30 07:39:00.828881 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:00.826126    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/01fea4c7-b874-49e8-a8d5-c0b9b50413fb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8e7948-6d8dbfhct\" (UID: \"01fea4c7-b874-49e8-a8d5-c0b9b50413fb\") " pod="ai-tenant-e2e-extmod-a-8e7948/payload-processing-external-model-e2e-extmod-a-8e7948-6d8dbfhct"
Oct 30 07:39:00.833831 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:00.829243    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01fea4c7-b874-49e8-a8d5-c0b9b50413fb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8e7948-6d8dbfhct\" (UID: \"01fea4c7-b874-49e8-a8d5-c0b9b50413fb\") " pod="ai-tenant-e2e-extmod-a-8e7948/payload-processing-external-model-e2e-extmod-a-8e7948-6d8dbfhct"
Oct 30 07:39:00.935927 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:00.935839    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqd9b\" (UniqueName: \"kubernetes.io/projected/01fea4c7-b874-49e8-a8d5-c0b9b50413fb-kube-api-access-sqd9b\") pod \"payload-processing-external-model-e2e-extmod-a-8e7948-6d8dbfhct\" (UID: \"01fea4c7-b874-49e8-a8d5-c0b9b50413fb\") " pod="ai-tenant-e2e-extmod-a-8e7948/payload-processing-external-model-e2e-extmod-a-8e7948-6d8dbfhct"
Oct 30 07:39:00.935927 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:00.935876    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/01fea4c7-b874-49e8-a8d5-c0b9b50413fb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8e7948-6d8dbfhct\" (UID: \"01fea4c7-b874-49e8-a8d5-c0b9b50413fb\") " pod="ai-tenant-e2e-extmod-a-8e7948/payload-processing-external-model-e2e-extmod-a-8e7948-6d8dbfhct"
Oct 30 07:39:00.936170 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:00.935936    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01fea4c7-b874-49e8-a8d5-c0b9b50413fb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8e7948-6d8dbfhct\" (UID: \"01fea4c7-b874-49e8-a8d5-c0b9b50413fb\") " pod="ai-tenant-e2e-extmod-a-8e7948/payload-processing-external-model-e2e-extmod-a-8e7948-6d8dbfhct"
Oct 30 07:39:00.936170 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:00.935997    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/01fea4c7-b874-49e8-a8d5-c0b9b50413fb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8e7948-6d8dbfhct\" (UID: \"01fea4c7-b874-49e8-a8d5-c0b9b50413fb\") " pod="ai-tenant-e2e-extmod-a-8e7948/payload-processing-external-model-e2e-extmod-a-8e7948-6d8dbfhct"
Oct 30 07:39:00.936762 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:00.936737    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/01fea4c7-b874-49e8-a8d5-c0b9b50413fb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8e7948-6d8dbfhct\" (UID: \"01fea4c7-b874-49e8-a8d5-c0b9b50413fb\") " pod="ai-tenant-e2e-extmod-a-8e7948/payload-processing-external-model-e2e-extmod-a-8e7948-6d8dbfhct"
Oct 30 07:39:00.936762 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:00.936753    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01fea4c7-b874-49e8-a8d5-c0b9b50413fb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8e7948-6d8dbfhct\" (UID: \"01fea4c7-b874-49e8-a8d5-c0b9b50413fb\") " pod="ai-tenant-e2e-extmod-a-8e7948/payload-processing-external-model-e2e-extmod-a-8e7948-6d8dbfhct"
Oct 30 07:39:00.938535 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:00.938503    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/01fea4c7-b874-49e8-a8d5-c0b9b50413fb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8e7948-6d8dbfhct\" (UID: \"01fea4c7-b874-49e8-a8d5-c0b9b50413fb\") " pod="ai-tenant-e2e-extmod-a-8e7948/payload-processing-external-model-e2e-extmod-a-8e7948-6d8dbfhct"
Oct 30 07:39:00.979365 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:00.979336    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqd9b\" (UniqueName: \"kubernetes.io/projected/01fea4c7-b874-49e8-a8d5-c0b9b50413fb-kube-api-access-sqd9b\") pod \"payload-processing-external-model-e2e-extmod-a-8e7948-6d8dbfhct\" (UID: \"01fea4c7-b874-49e8-a8d5-c0b9b50413fb\") " pod="ai-tenant-e2e-extmod-a-8e7948/payload-processing-external-model-e2e-extmod-a-8e7948-6d8dbfhct"
Oct 30 07:39:00.988238 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:00.988212    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8e7948/payload-processing-external-model-e2e-extmod-a-8e7948-6d8dbfhct"
Oct 30 07:39:01.293517 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:39:01.293486    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01fea4c7_b874_49e8_a8d5_c0b9b50413fb.slice/crio-a64a5d4c120e1bf6d09652a10895d98d745641f0906b5c73a3c6919b641d41df WatchSource:0}: Error finding container a64a5d4c120e1bf6d09652a10895d98d745641f0906b5c73a3c6919b641d41df: Status 404 returned error can't find the container with id a64a5d4c120e1bf6d09652a10895d98d745641f0906b5c73a3c6919b641d41df
Oct 30 07:39:01.295549 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:01.295515    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8e7948/payload-processing-external-model-e2e-extmod-a-8e7948-6d8dbfhct"]
Oct 30 07:39:01.751548 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:01.751455    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8e7948/payload-processing-external-model-e2e-extmod-a-8e7948-6d8dbfhct" event={"ID":"01fea4c7-b874-49e8-a8d5-c0b9b50413fb","Type":"ContainerStarted","Data":"6a6497dd3113e468ee4bddac46f13225b86e65344a1e3dac15b292d5b7c6b6aa"}
Oct 30 07:39:01.751548 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:01.751530    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8e7948/payload-processing-external-model-e2e-extmod-a-8e7948-6d8dbfhct" event={"ID":"01fea4c7-b874-49e8-a8d5-c0b9b50413fb","Type":"ContainerStarted","Data":"a64a5d4c120e1bf6d09652a10895d98d745641f0906b5c73a3c6919b641d41df"}
Oct 30 07:39:01.752083 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:01.751677    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-8e7948/payload-processing-external-model-e2e-extmod-a-8e7948-6d8dbfhct"
Oct 30 07:39:01.800109 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:01.800058    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-8e7948/payload-processing-external-model-e2e-extmod-a-8e7948-6d8dbfhct" podStartSLOduration=1.800043442 podStartE2EDuration="1.800043442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:39:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:39:01.796503523 +0000 UTC m=+2500.405910495" watchObservedRunningTime="2026-10-30 07:39:01.800043442 +0000 UTC m=+2500.409450465"
Oct 30 07:39:01.858953 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:39:01.858904    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:39:12.762556 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:12.762526    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-8e7948/payload-processing-external-model-e2e-extmod-a-8e7948-6d8dbfhct"
Oct 30 07:39:15.868768 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:15.868734    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:39:31.884910 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:39:31.884729    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:39:31.891244 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:31.891215    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-9d6bcf8f-srvqq"]
Oct 30 07:39:36.891928 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:36.891846    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-9d6bcf8f-srvqq" podUID="d95a630d-751d-4171-8a10-331cc3382c23" containerName="payload-processing" containerID="cri-o://14f63fab7cb70b85991ac04916137260570034dab9a503b1969734eeed998470" gracePeriod=25
Oct 30 07:39:37.032273 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:37.032245    6224 generic.go:356] "Generic (PLEG): container finished" podID="d95a630d-751d-4171-8a10-331cc3382c23" containerID="14f63fab7cb70b85991ac04916137260570034dab9a503b1969734eeed998470" exitCode=0
Oct 30 07:39:37.032499 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:37.032325    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-9d6bcf8f-srvqq" event={"ID":"d95a630d-751d-4171-8a10-331cc3382c23","Type":"ContainerDied","Data":"14f63fab7cb70b85991ac04916137260570034dab9a503b1969734eeed998470"}
Oct 30 07:39:37.166480 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:37.166448    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-9d6bcf8f-srvqq"
Oct 30 07:39:37.215069 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:37.215042    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d95a630d-751d-4171-8a10-331cc3382c23-kube-api-access-9zz5s\" (UniqueName: \"kubernetes.io/projected/d95a630d-751d-4171-8a10-331cc3382c23-kube-api-access-9zz5s\") pod \"d95a630d-751d-4171-8a10-331cc3382c23\" (UID: \"d95a630d-751d-4171-8a10-331cc3382c23\") "
Oct 30 07:39:37.215317 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:37.215289    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d95a630d-751d-4171-8a10-331cc3382c23-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d95a630d-751d-4171-8a10-331cc3382c23-plugins-config-volume\") pod \"d95a630d-751d-4171-8a10-331cc3382c23\" (UID: \"d95a630d-751d-4171-8a10-331cc3382c23\") "
Oct 30 07:39:37.215495 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:37.215448    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d95a630d-751d-4171-8a10-331cc3382c23-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d95a630d-751d-4171-8a10-331cc3382c23-provider-credentials\") pod \"d95a630d-751d-4171-8a10-331cc3382c23\" (UID: \"d95a630d-751d-4171-8a10-331cc3382c23\") "
Oct 30 07:39:37.215495 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:37.215480    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d95a630d-751d-4171-8a10-331cc3382c23-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d95a630d-751d-4171-8a10-331cc3382c23-routing-overlay-volume\") pod \"d95a630d-751d-4171-8a10-331cc3382c23\" (UID: \"d95a630d-751d-4171-8a10-331cc3382c23\") "
Oct 30 07:39:37.215944 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:37.215915    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d95a630d-751d-4171-8a10-331cc3382c23-plugins-config-volume" pod "d95a630d-751d-4171-8a10-331cc3382c23" (UID: "d95a630d-751d-4171-8a10-331cc3382c23"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:39:37.216310 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:37.216288    6224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d95a630d-751d-4171-8a10-331cc3382c23-plugins-config-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:39:37.217738 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:37.217713    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d95a630d-751d-4171-8a10-331cc3382c23-kube-api-access-9zz5s" pod "d95a630d-751d-4171-8a10-331cc3382c23" (UID: "d95a630d-751d-4171-8a10-331cc3382c23"). InnerVolumeSpecName "kube-api-access-9zz5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:39:37.218007 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:37.217982    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d95a630d-751d-4171-8a10-331cc3382c23-routing-overlay-volume" pod "d95a630d-751d-4171-8a10-331cc3382c23" (UID: "d95a630d-751d-4171-8a10-331cc3382c23"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:39:37.218337 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:37.218318    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d95a630d-751d-4171-8a10-331cc3382c23-provider-credentials" pod "d95a630d-751d-4171-8a10-331cc3382c23" (UID: "d95a630d-751d-4171-8a10-331cc3382c23"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:39:37.317919 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:37.317891    6224 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d95a630d-751d-4171-8a10-331cc3382c23-provider-credentials\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:39:37.317919 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:37.317917    6224 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d95a630d-751d-4171-8a10-331cc3382c23-routing-overlay-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:39:37.317919 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:37.317927    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zz5s\" (UniqueName: \"kubernetes.io/projected/d95a630d-751d-4171-8a10-331cc3382c23-kube-api-access-9zz5s\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:39:38.041727 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:38.041692    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-9d6bcf8f-srvqq"
Oct 30 07:39:38.042210 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:38.041694    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-9d6bcf8f-srvqq" event={"ID":"d95a630d-751d-4171-8a10-331cc3382c23","Type":"ContainerDied","Data":"ae871faacf6d319105c98e816b4872d4b4f05c01ce8d28f104b9252d7511426f"}
Oct 30 07:39:38.042210 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:38.041846    6224 scope.go:122] "RemoveContainer" containerID="14f63fab7cb70b85991ac04916137260570034dab9a503b1969734eeed998470"
Oct 30 07:39:38.102949 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:38.102922    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-9d6bcf8f-srvqq"]
Oct 30 07:39:38.117043 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:38.117020    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-9d6bcf8f-srvqq"]
Oct 30 07:39:39.503960 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:39.503928    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d95a630d-751d-4171-8a10-331cc3382c23" path="/var/lib/kubelet/pods/d95a630d-751d-4171-8a10-331cc3382c23/volumes"
Oct 30 07:39:42.714390 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:42.714356    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-675d78f79c-4vvll"]
Oct 30 07:39:42.715009 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:42.714994    6224 cpu_manager.go:414] "containerMap: removing container" podUID="d95a630d-751d-4171-8a10-331cc3382c23" containerName="payload-processing"
Oct 30 07:39:42.715071 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:42.715010    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d95a630d-751d-4171-8a10-331cc3382c23" containerName="payload-processing"
Oct 30 07:39:42.715158 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:42.715148    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="d95a630d-751d-4171-8a10-331cc3382c23" containerName="payload-processing"
Oct 30 07:39:42.719913 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:42.719894    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-675d78f79c-4vvll"
Oct 30 07:39:42.739794 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:42.739773    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-a3d385576abf\""
Oct 30 07:39:42.754014 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:42.753978    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-dtg2k\""
Oct 30 07:39:42.757196 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:42.757163    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 30 07:39:42.757312 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:42.757276    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 30 07:39:42.757385 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:42.757339    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 30 07:39:42.757444 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:42.757400    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 30 07:39:42.762903 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:42.762879    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-675d78f79c-4vvll"]
Oct 30 07:39:42.791161 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:42.791137    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd4fe93a-47b4-4d4a-ba3b-eabf92e83b5f-plugins-config-volume\") pod \"payload-processing-external-model-675d78f79c-4vvll\" (UID: \"cd4fe93a-47b4-4d4a-ba3b-eabf92e83b5f\") " pod="models-as-a-service/payload-processing-external-model-675d78f79c-4vvll"
Oct 30 07:39:42.791256 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:42.791242    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cd4fe93a-47b4-4d4a-ba3b-eabf92e83b5f-routing-overlay-volume\") pod \"payload-processing-external-model-675d78f79c-4vvll\" (UID: \"cd4fe93a-47b4-4d4a-ba3b-eabf92e83b5f\") " pod="models-as-a-service/payload-processing-external-model-675d78f79c-4vvll"
Oct 30 07:39:42.791301 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:42.791288    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cd4fe93a-47b4-4d4a-ba3b-eabf92e83b5f-provider-credentials\") pod \"payload-processing-external-model-675d78f79c-4vvll\" (UID: \"cd4fe93a-47b4-4d4a-ba3b-eabf92e83b5f\") " pod="models-as-a-service/payload-processing-external-model-675d78f79c-4vvll"
Oct 30 07:39:42.791455 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:42.791439    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spzk7\" (UniqueName: \"kubernetes.io/projected/cd4fe93a-47b4-4d4a-ba3b-eabf92e83b5f-kube-api-access-spzk7\") pod \"payload-processing-external-model-675d78f79c-4vvll\" (UID: \"cd4fe93a-47b4-4d4a-ba3b-eabf92e83b5f\") " pod="models-as-a-service/payload-processing-external-model-675d78f79c-4vvll"
Oct 30 07:39:42.893269 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:42.893229    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spzk7\" (UniqueName: \"kubernetes.io/projected/cd4fe93a-47b4-4d4a-ba3b-eabf92e83b5f-kube-api-access-spzk7\") pod \"payload-processing-external-model-675d78f79c-4vvll\" (UID: \"cd4fe93a-47b4-4d4a-ba3b-eabf92e83b5f\") " pod="models-as-a-service/payload-processing-external-model-675d78f79c-4vvll"
Oct 30 07:39:42.893611 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:42.893590    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd4fe93a-47b4-4d4a-ba3b-eabf92e83b5f-plugins-config-volume\") pod \"payload-processing-external-model-675d78f79c-4vvll\" (UID: \"cd4fe93a-47b4-4d4a-ba3b-eabf92e83b5f\") " pod="models-as-a-service/payload-processing-external-model-675d78f79c-4vvll"
Oct 30 07:39:42.893903 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:42.893882    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cd4fe93a-47b4-4d4a-ba3b-eabf92e83b5f-routing-overlay-volume\") pod \"payload-processing-external-model-675d78f79c-4vvll\" (UID: \"cd4fe93a-47b4-4d4a-ba3b-eabf92e83b5f\") " pod="models-as-a-service/payload-processing-external-model-675d78f79c-4vvll"
Oct 30 07:39:42.894055 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:42.894039    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cd4fe93a-47b4-4d4a-ba3b-eabf92e83b5f-provider-credentials\") pod \"payload-processing-external-model-675d78f79c-4vvll\" (UID: \"cd4fe93a-47b4-4d4a-ba3b-eabf92e83b5f\") " pod="models-as-a-service/payload-processing-external-model-675d78f79c-4vvll"
Oct 30 07:39:42.895601 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:42.895578    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd4fe93a-47b4-4d4a-ba3b-eabf92e83b5f-plugins-config-volume\") pod \"payload-processing-external-model-675d78f79c-4vvll\" (UID: \"cd4fe93a-47b4-4d4a-ba3b-eabf92e83b5f\") " pod="models-as-a-service/payload-processing-external-model-675d78f79c-4vvll"
Oct 30 07:39:42.896855 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:42.896796    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cd4fe93a-47b4-4d4a-ba3b-eabf92e83b5f-routing-overlay-volume\") pod \"payload-processing-external-model-675d78f79c-4vvll\" (UID: \"cd4fe93a-47b4-4d4a-ba3b-eabf92e83b5f\") " pod="models-as-a-service/payload-processing-external-model-675d78f79c-4vvll"
Oct 30 07:39:42.899652 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:42.899620    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cd4fe93a-47b4-4d4a-ba3b-eabf92e83b5f-provider-credentials\") pod \"payload-processing-external-model-675d78f79c-4vvll\" (UID: \"cd4fe93a-47b4-4d4a-ba3b-eabf92e83b5f\") " pod="models-as-a-service/payload-processing-external-model-675d78f79c-4vvll"
Oct 30 07:39:42.925851 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:42.925531    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spzk7\" (UniqueName: \"kubernetes.io/projected/cd4fe93a-47b4-4d4a-ba3b-eabf92e83b5f-kube-api-access-spzk7\") pod \"payload-processing-external-model-675d78f79c-4vvll\" (UID: \"cd4fe93a-47b4-4d4a-ba3b-eabf92e83b5f\") " pod="models-as-a-service/payload-processing-external-model-675d78f79c-4vvll"
Oct 30 07:39:43.033868 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:43.033838    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-675d78f79c-4vvll"
Oct 30 07:39:43.346442 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:43.346357    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gshg6"]
Oct 30 07:39:43.352119 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:43.352098    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gshg6"
Oct 30 07:39:43.353906 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:43.353882    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-675d78f79c-4vvll"]
Oct 30 07:39:43.356427 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:39:43.356399    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd4fe93a_47b4_4d4a_ba3b_eabf92e83b5f.slice/crio-c230ea4d5855761a7321afae624163391a7cd94d467eec9e4b0e666d4d3cc5a2 WatchSource:0}: Error finding container c230ea4d5855761a7321afae624163391a7cd94d467eec9e4b0e666d4d3cc5a2: Status 404 returned error can't find the container with id c230ea4d5855761a7321afae624163391a7cd94d467eec9e4b0e666d4d3cc5a2
Oct 30 07:39:43.378136 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:43.377930    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gshg6"]
Oct 30 07:39:43.408865 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:43.406420    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqcxx\" (UniqueName: \"kubernetes.io/projected/02fa3182-1c4a-418e-882b-08db6f71a0a7-kube-api-access-zqcxx\") pod \"maas-default-gateway-openshift-default-6dcf757899-gshg6\" (UID: \"02fa3182-1c4a-418e-882b-08db6f71a0a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gshg6"
Oct 30 07:39:43.408865 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:43.406485    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02fa3182-1c4a-418e-882b-08db6f71a0a7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-gshg6\" (UID: \"02fa3182-1c4a-418e-882b-08db6f71a0a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gshg6"
Oct 30 07:39:43.408865 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:43.406632    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02fa3182-1c4a-418e-882b-08db6f71a0a7-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-gshg6\" (UID: \"02fa3182-1c4a-418e-882b-08db6f71a0a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gshg6"
Oct 30 07:39:43.408865 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:43.406749    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02fa3182-1c4a-418e-882b-08db6f71a0a7-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-gshg6\" (UID: \"02fa3182-1c4a-418e-882b-08db6f71a0a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gshg6"
Oct 30 07:39:43.408865 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:43.406830    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02fa3182-1c4a-418e-882b-08db6f71a0a7-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-gshg6\" (UID: \"02fa3182-1c4a-418e-882b-08db6f71a0a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gshg6"
Oct 30 07:39:43.408865 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:43.406881    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02fa3182-1c4a-418e-882b-08db6f71a0a7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-gshg6\" (UID: \"02fa3182-1c4a-418e-882b-08db6f71a0a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gshg6"
Oct 30 07:39:43.408865 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:43.407738    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02fa3182-1c4a-418e-882b-08db6f71a0a7-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-gshg6\" (UID: \"02fa3182-1c4a-418e-882b-08db6f71a0a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gshg6"
Oct 30 07:39:43.408865 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:43.407792    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02fa3182-1c4a-418e-882b-08db6f71a0a7-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-gshg6\" (UID: \"02fa3182-1c4a-418e-882b-08db6f71a0a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gshg6"
Oct 30 07:39:43.408865 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:43.407853    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02fa3182-1c4a-418e-882b-08db6f71a0a7-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-gshg6\" (UID: \"02fa3182-1c4a-418e-882b-08db6f71a0a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gshg6"
Oct 30 07:39:43.509240 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:43.509204    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02fa3182-1c4a-418e-882b-08db6f71a0a7-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-gshg6\" (UID: \"02fa3182-1c4a-418e-882b-08db6f71a0a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gshg6"
Oct 30 07:39:43.509395 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:43.509295    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02fa3182-1c4a-418e-882b-08db6f71a0a7-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-gshg6\" (UID: \"02fa3182-1c4a-418e-882b-08db6f71a0a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gshg6"
Oct 30 07:39:43.509395 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:43.509340    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02fa3182-1c4a-418e-882b-08db6f71a0a7-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-gshg6\" (UID: \"02fa3182-1c4a-418e-882b-08db6f71a0a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gshg6"
Oct 30 07:39:43.509395 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:43.509372    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02fa3182-1c4a-418e-882b-08db6f71a0a7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-gshg6\" (UID: \"02fa3182-1c4a-418e-882b-08db6f71a0a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gshg6"
Oct 30 07:39:43.509584 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:43.509412    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02fa3182-1c4a-418e-882b-08db6f71a0a7-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-gshg6\" (UID: \"02fa3182-1c4a-418e-882b-08db6f71a0a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gshg6"
Oct 30 07:39:43.509584 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:43.509449    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02fa3182-1c4a-418e-882b-08db6f71a0a7-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-gshg6\" (UID: \"02fa3182-1c4a-418e-882b-08db6f71a0a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gshg6"
Oct 30 07:39:43.509584 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:43.509487    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02fa3182-1c4a-418e-882b-08db6f71a0a7-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-gshg6\" (UID: \"02fa3182-1c4a-418e-882b-08db6f71a0a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gshg6"
Oct 30 07:39:43.509584 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:43.509554    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqcxx\" (UniqueName: \"kubernetes.io/projected/02fa3182-1c4a-418e-882b-08db6f71a0a7-kube-api-access-zqcxx\") pod \"maas-default-gateway-openshift-default-6dcf757899-gshg6\" (UID: \"02fa3182-1c4a-418e-882b-08db6f71a0a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gshg6"
Oct 30 07:39:43.509792 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:43.509593    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02fa3182-1c4a-418e-882b-08db6f71a0a7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-gshg6\" (UID: \"02fa3182-1c4a-418e-882b-08db6f71a0a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gshg6"
Oct 30 07:39:43.509792 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:43.509675    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02fa3182-1c4a-418e-882b-08db6f71a0a7-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-gshg6\" (UID: \"02fa3182-1c4a-418e-882b-08db6f71a0a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gshg6"
Oct 30 07:39:43.509792 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:43.509711    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02fa3182-1c4a-418e-882b-08db6f71a0a7-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-gshg6\" (UID: \"02fa3182-1c4a-418e-882b-08db6f71a0a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gshg6"
Oct 30 07:39:43.509792 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:43.509739    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02fa3182-1c4a-418e-882b-08db6f71a0a7-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-gshg6\" (UID: \"02fa3182-1c4a-418e-882b-08db6f71a0a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gshg6"
Oct 30 07:39:43.510225 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:43.510195    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02fa3182-1c4a-418e-882b-08db6f71a0a7-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-gshg6\" (UID: \"02fa3182-1c4a-418e-882b-08db6f71a0a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gshg6"
Oct 30 07:39:43.510313 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:43.510290    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02fa3182-1c4a-418e-882b-08db6f71a0a7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-gshg6\" (UID: \"02fa3182-1c4a-418e-882b-08db6f71a0a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gshg6"
Oct 30 07:39:43.511491 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:43.511469    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02fa3182-1c4a-418e-882b-08db6f71a0a7-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-gshg6\" (UID: \"02fa3182-1c4a-418e-882b-08db6f71a0a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gshg6"
Oct 30 07:39:43.511717 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:43.511701    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02fa3182-1c4a-418e-882b-08db6f71a0a7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-gshg6\" (UID: \"02fa3182-1c4a-418e-882b-08db6f71a0a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gshg6"
Oct 30 07:39:43.534523 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:43.534494    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02fa3182-1c4a-418e-882b-08db6f71a0a7-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-gshg6\" (UID: \"02fa3182-1c4a-418e-882b-08db6f71a0a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gshg6"
Oct 30 07:39:43.534734 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:43.534713    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqcxx\" (UniqueName: \"kubernetes.io/projected/02fa3182-1c4a-418e-882b-08db6f71a0a7-kube-api-access-zqcxx\") pod \"maas-default-gateway-openshift-default-6dcf757899-gshg6\" (UID: \"02fa3182-1c4a-418e-882b-08db6f71a0a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gshg6"
Oct 30 07:39:43.721052 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:43.721014    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gshg6"
Oct 30 07:39:44.038268 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:39:44.038241    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02fa3182_1c4a_418e_882b_08db6f71a0a7.slice/crio-909d86539fe4f91aeb59ca7b0e0029075eea245cb1851db390cbd83e825c2705 WatchSource:0}: Error finding container 909d86539fe4f91aeb59ca7b0e0029075eea245cb1851db390cbd83e825c2705: Status 404 returned error can't find the container with id 909d86539fe4f91aeb59ca7b0e0029075eea245cb1851db390cbd83e825c2705
Oct 30 07:39:44.039236 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:44.038977    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gshg6"]
Oct 30 07:39:44.041198 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:44.041168    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:39:44.041309 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:44.041244    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:39:44.041309 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:44.041290    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:39:44.084916 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:44.084882    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gshg6" event={"ID":"02fa3182-1c4a-418e-882b-08db6f71a0a7","Type":"ContainerStarted","Data":"909d86539fe4f91aeb59ca7b0e0029075eea245cb1851db390cbd83e825c2705"}
Oct 30 07:39:44.086834 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:44.086777    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-675d78f79c-4vvll" event={"ID":"cd4fe93a-47b4-4d4a-ba3b-eabf92e83b5f","Type":"ContainerStarted","Data":"07766c4eea52bc13597a4dd12763427578327c146682e27dff5c3d2132176f12"}
Oct 30 07:39:44.086921 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:44.086856    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-675d78f79c-4vvll"
Oct 30 07:39:44.086921 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:44.086880    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-675d78f79c-4vvll" event={"ID":"cd4fe93a-47b4-4d4a-ba3b-eabf92e83b5f","Type":"ContainerStarted","Data":"c230ea4d5855761a7321afae624163391a7cd94d467eec9e4b0e666d4d3cc5a2"}
Oct 30 07:39:44.131748 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:44.131704    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-675d78f79c-4vvll" podStartSLOduration=2.131690997 podStartE2EDuration="2.131690997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:39:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:39:44.131480741 +0000 UTC m=+2542.740887713" watchObservedRunningTime="2026-10-30 07:39:44.131690997 +0000 UTC m=+2542.741097961"
Oct 30 07:39:45.096704 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:45.096661    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gshg6" event={"ID":"02fa3182-1c4a-418e-882b-08db6f71a0a7","Type":"ContainerStarted","Data":"64497b29ea7254f96b9c5b866e4ba23ff66249e289eb73b8e56c235f49b4b153"}
Oct 30 07:39:45.159857 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:45.159778    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gshg6" podStartSLOduration=2.159762939 podStartE2EDuration="2.159762939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:39:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:39:45.159595946 +0000 UTC m=+2543.769002920" watchObservedRunningTime="2026-10-30 07:39:45.159762939 +0000 UTC m=+2543.769169908"
Oct 30 07:39:45.721736 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:45.721701    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gshg6"
Oct 30 07:39:45.723014 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:45.722982    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-gshg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Oct 30 07:39:45.723143 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:45.723040    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gshg6" podUID="02fa3182-1c4a-418e-882b-08db6f71a0a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Oct 30 07:39:46.721559 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:46.721522    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-gshg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Oct 30 07:39:46.721999 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:46.721576    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gshg6" podUID="02fa3182-1c4a-418e-882b-08db6f71a0a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Oct 30 07:39:47.722100 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:47.722064    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-gshg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Oct 30 07:39:47.722576 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:47.722125    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gshg6" podUID="02fa3182-1c4a-418e-882b-08db6f71a0a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Oct 30 07:39:47.932854 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:47.932677    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-675d78f79c-4vvll"]
Oct 30 07:39:48.721929 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:48.721888    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-gshg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Oct 30 07:39:48.722118 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:48.721947    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gshg6" podUID="02fa3182-1c4a-418e-882b-08db6f71a0a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Oct 30 07:39:49.721900 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:49.721869    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-gshg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Oct 30 07:39:49.722086 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:49.721918    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gshg6" podUID="02fa3182-1c4a-418e-882b-08db6f71a0a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Oct 30 07:39:50.721515 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:50.721477    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-gshg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Oct 30 07:39:50.721933 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:50.721541    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gshg6" podUID="02fa3182-1c4a-418e-882b-08db6f71a0a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Oct 30 07:39:51.721533 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:51.721502    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-gshg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Oct 30 07:39:51.722086 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:51.721551    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gshg6" podUID="02fa3182-1c4a-418e-882b-08db6f71a0a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Oct 30 07:39:52.933642 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:52.933594    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-675d78f79c-4vvll" podUID="cd4fe93a-47b4-4d4a-ba3b-eabf92e83b5f" containerName="payload-processing" containerID="cri-o://07766c4eea52bc13597a4dd12763427578327c146682e27dff5c3d2132176f12" gracePeriod=25
Oct 30 07:39:52.945769 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:52.945743    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gshg6"
Oct 30 07:39:52.946316 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:52.946288    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gshg6"
Oct 30 07:39:52.947569 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:52.947263    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gshg6"
Oct 30 07:39:53.029834 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:39:53.029619    6224 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd4fe93a_47b4_4d4a_ba3b_eabf92e83b5f.slice/crio-conmon-07766c4eea52bc13597a4dd12763427578327c146682e27dff5c3d2132176f12.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 07:39:53.158951 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:53.158914    6224 generic.go:356] "Generic (PLEG): container finished" podID="cd4fe93a-47b4-4d4a-ba3b-eabf92e83b5f" containerID="07766c4eea52bc13597a4dd12763427578327c146682e27dff5c3d2132176f12" exitCode=0
Oct 30 07:39:53.159108 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:53.158989    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-675d78f79c-4vvll" event={"ID":"cd4fe93a-47b4-4d4a-ba3b-eabf92e83b5f","Type":"ContainerDied","Data":"07766c4eea52bc13597a4dd12763427578327c146682e27dff5c3d2132176f12"}
Oct 30 07:39:53.334738 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:53.334323    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-675d78f79c-4vvll"
Oct 30 07:39:53.510750 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:53.510706    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd4fe93a-47b4-4d4a-ba3b-eabf92e83b5f-kube-api-access-spzk7\" (UniqueName: \"kubernetes.io/projected/cd4fe93a-47b4-4d4a-ba3b-eabf92e83b5f-kube-api-access-spzk7\") pod \"cd4fe93a-47b4-4d4a-ba3b-eabf92e83b5f\" (UID: \"cd4fe93a-47b4-4d4a-ba3b-eabf92e83b5f\") "
Oct 30 07:39:53.510977 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:53.510862    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd4fe93a-47b4-4d4a-ba3b-eabf92e83b5f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cd4fe93a-47b4-4d4a-ba3b-eabf92e83b5f-routing-overlay-volume\") pod \"cd4fe93a-47b4-4d4a-ba3b-eabf92e83b5f\" (UID: \"cd4fe93a-47b4-4d4a-ba3b-eabf92e83b5f\") "
Oct 30 07:39:53.510977 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:53.510899    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd4fe93a-47b4-4d4a-ba3b-eabf92e83b5f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/cd4fe93a-47b4-4d4a-ba3b-eabf92e83b5f-provider-credentials\") pod \"cd4fe93a-47b4-4d4a-ba3b-eabf92e83b5f\" (UID: \"cd4fe93a-47b4-4d4a-ba3b-eabf92e83b5f\") "
Oct 30 07:39:53.511120 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:53.510986    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd4fe93a-47b4-4d4a-ba3b-eabf92e83b5f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd4fe93a-47b4-4d4a-ba3b-eabf92e83b5f-plugins-config-volume\") pod \"cd4fe93a-47b4-4d4a-ba3b-eabf92e83b5f\" (UID: \"cd4fe93a-47b4-4d4a-ba3b-eabf92e83b5f\") "
Oct 30 07:39:53.511955 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:53.511925    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd4fe93a-47b4-4d4a-ba3b-eabf92e83b5f-plugins-config-volume" pod "cd4fe93a-47b4-4d4a-ba3b-eabf92e83b5f" (UID: "cd4fe93a-47b4-4d4a-ba3b-eabf92e83b5f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:39:53.512560 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:53.512530    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd4fe93a-47b4-4d4a-ba3b-eabf92e83b5f-routing-overlay-volume" pod "cd4fe93a-47b4-4d4a-ba3b-eabf92e83b5f" (UID: "cd4fe93a-47b4-4d4a-ba3b-eabf92e83b5f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:39:53.514741 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:53.514708    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd4fe93a-47b4-4d4a-ba3b-eabf92e83b5f-kube-api-access-spzk7" pod "cd4fe93a-47b4-4d4a-ba3b-eabf92e83b5f" (UID: "cd4fe93a-47b4-4d4a-ba3b-eabf92e83b5f"). InnerVolumeSpecName "kube-api-access-spzk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:39:53.515594 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:53.515568    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd4fe93a-47b4-4d4a-ba3b-eabf92e83b5f-provider-credentials" pod "cd4fe93a-47b4-4d4a-ba3b-eabf92e83b5f" (UID: "cd4fe93a-47b4-4d4a-ba3b-eabf92e83b5f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:39:53.613301 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:53.613197    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spzk7\" (UniqueName: \"kubernetes.io/projected/cd4fe93a-47b4-4d4a-ba3b-eabf92e83b5f-kube-api-access-spzk7\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:39:53.613301 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:53.613239    6224 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cd4fe93a-47b4-4d4a-ba3b-eabf92e83b5f-routing-overlay-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:39:53.613301 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:53.613256    6224 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cd4fe93a-47b4-4d4a-ba3b-eabf92e83b5f-provider-credentials\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:39:53.613301 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:53.613274    6224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd4fe93a-47b4-4d4a-ba3b-eabf92e83b5f-plugins-config-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:39:54.169336 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:54.169301    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-675d78f79c-4vvll"
Oct 30 07:39:54.169878 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:54.169309    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-675d78f79c-4vvll" event={"ID":"cd4fe93a-47b4-4d4a-ba3b-eabf92e83b5f","Type":"ContainerDied","Data":"c230ea4d5855761a7321afae624163391a7cd94d467eec9e4b0e666d4d3cc5a2"}
Oct 30 07:39:54.169878 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:54.169441    6224 scope.go:122] "RemoveContainer" containerID="07766c4eea52bc13597a4dd12763427578327c146682e27dff5c3d2132176f12"
Oct 30 07:39:54.231877 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:54.231845    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-675d78f79c-4vvll"]
Oct 30 07:39:54.244113 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:54.244079    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-675d78f79c-4vvll"]
Oct 30 07:39:55.504778 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:55.504744    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd4fe93a-47b4-4d4a-ba3b-eabf92e83b5f" path="/var/lib/kubelet/pods/cd4fe93a-47b4-4d4a-ba3b-eabf92e83b5f/volumes"
Oct 30 07:39:58.388505 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:58.388465    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dtptn"]
Oct 30 07:39:58.390898 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:58.389329    6224 cpu_manager.go:414] "containerMap: removing container" podUID="cd4fe93a-47b4-4d4a-ba3b-eabf92e83b5f" containerName="payload-processing"
Oct 30 07:39:58.390898 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:58.389346    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd4fe93a-47b4-4d4a-ba3b-eabf92e83b5f" containerName="payload-processing"
Oct 30 07:39:58.390898 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:58.389447    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd4fe93a-47b4-4d4a-ba3b-eabf92e83b5f" containerName="payload-processing"
Oct 30 07:39:58.393447 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:58.393426    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dtptn"
Oct 30 07:39:58.419735 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:58.419702    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dtptn"]
Oct 30 07:39:58.473846 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:58.473787    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a0e028c-e9eb-4260-bfa3-37104c26af0e-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-dtptn\" (UID: \"0a0e028c-e9eb-4260-bfa3-37104c26af0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dtptn"
Oct 30 07:39:58.474090 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:58.474062    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a0e028c-e9eb-4260-bfa3-37104c26af0e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-dtptn\" (UID: \"0a0e028c-e9eb-4260-bfa3-37104c26af0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dtptn"
Oct 30 07:39:58.474248 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:58.474118    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a0e028c-e9eb-4260-bfa3-37104c26af0e-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-dtptn\" (UID: \"0a0e028c-e9eb-4260-bfa3-37104c26af0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dtptn"
Oct 30 07:39:58.474248 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:58.474207    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a0e028c-e9eb-4260-bfa3-37104c26af0e-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-dtptn\" (UID: \"0a0e028c-e9eb-4260-bfa3-37104c26af0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dtptn"
Oct 30 07:39:58.474370 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:58.474272    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a0e028c-e9eb-4260-bfa3-37104c26af0e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-dtptn\" (UID: \"0a0e028c-e9eb-4260-bfa3-37104c26af0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dtptn"
Oct 30 07:39:58.474370 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:58.474305    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a0e028c-e9eb-4260-bfa3-37104c26af0e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-dtptn\" (UID: \"0a0e028c-e9eb-4260-bfa3-37104c26af0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dtptn"
Oct 30 07:39:58.474972 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:58.474911    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a0e028c-e9eb-4260-bfa3-37104c26af0e-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-dtptn\" (UID: \"0a0e028c-e9eb-4260-bfa3-37104c26af0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dtptn"
Oct 30 07:39:58.475101 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:58.475056    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bhmd\" (UniqueName: \"kubernetes.io/projected/0a0e028c-e9eb-4260-bfa3-37104c26af0e-kube-api-access-2bhmd\") pod \"maas-default-gateway-openshift-default-6dcf757899-dtptn\" (UID: \"0a0e028c-e9eb-4260-bfa3-37104c26af0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dtptn"
Oct 30 07:39:58.475187 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:58.475120    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a0e028c-e9eb-4260-bfa3-37104c26af0e-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-dtptn\" (UID: \"0a0e028c-e9eb-4260-bfa3-37104c26af0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dtptn"
Oct 30 07:39:58.576064 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:58.576025    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a0e028c-e9eb-4260-bfa3-37104c26af0e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-dtptn\" (UID: \"0a0e028c-e9eb-4260-bfa3-37104c26af0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dtptn"
Oct 30 07:39:58.576226 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:58.576072    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a0e028c-e9eb-4260-bfa3-37104c26af0e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-dtptn\" (UID: \"0a0e028c-e9eb-4260-bfa3-37104c26af0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dtptn"
Oct 30 07:39:58.576226 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:58.576111    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a0e028c-e9eb-4260-bfa3-37104c26af0e-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-dtptn\" (UID: \"0a0e028c-e9eb-4260-bfa3-37104c26af0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dtptn"
Oct 30 07:39:58.576226 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:58.576177    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bhmd\" (UniqueName: \"kubernetes.io/projected/0a0e028c-e9eb-4260-bfa3-37104c26af0e-kube-api-access-2bhmd\") pod \"maas-default-gateway-openshift-default-6dcf757899-dtptn\" (UID: \"0a0e028c-e9eb-4260-bfa3-37104c26af0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dtptn"
Oct 30 07:39:58.576226 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:58.576224    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a0e028c-e9eb-4260-bfa3-37104c26af0e-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-dtptn\" (UID: \"0a0e028c-e9eb-4260-bfa3-37104c26af0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dtptn"
Oct 30 07:39:58.576429 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:58.576248    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a0e028c-e9eb-4260-bfa3-37104c26af0e-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-dtptn\" (UID: \"0a0e028c-e9eb-4260-bfa3-37104c26af0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dtptn"
Oct 30 07:39:58.576429 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:58.576321    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a0e028c-e9eb-4260-bfa3-37104c26af0e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-dtptn\" (UID: \"0a0e028c-e9eb-4260-bfa3-37104c26af0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dtptn"
Oct 30 07:39:58.576429 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:58.576373    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a0e028c-e9eb-4260-bfa3-37104c26af0e-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-dtptn\" (UID: \"0a0e028c-e9eb-4260-bfa3-37104c26af0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dtptn"
Oct 30 07:39:58.576429 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:58.576409    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a0e028c-e9eb-4260-bfa3-37104c26af0e-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-dtptn\" (UID: \"0a0e028c-e9eb-4260-bfa3-37104c26af0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dtptn"
Oct 30 07:39:58.576648 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:58.576604    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a0e028c-e9eb-4260-bfa3-37104c26af0e-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-dtptn\" (UID: \"0a0e028c-e9eb-4260-bfa3-37104c26af0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dtptn"
Oct 30 07:39:58.576711 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:58.576663    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a0e028c-e9eb-4260-bfa3-37104c26af0e-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-dtptn\" (UID: \"0a0e028c-e9eb-4260-bfa3-37104c26af0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dtptn"
Oct 30 07:39:58.576793 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:58.576768    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a0e028c-e9eb-4260-bfa3-37104c26af0e-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-dtptn\" (UID: \"0a0e028c-e9eb-4260-bfa3-37104c26af0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dtptn"
Oct 30 07:39:58.576948 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:58.576923    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a0e028c-e9eb-4260-bfa3-37104c26af0e-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-dtptn\" (UID: \"0a0e028c-e9eb-4260-bfa3-37104c26af0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dtptn"
Oct 30 07:39:58.577143 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:58.577117    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a0e028c-e9eb-4260-bfa3-37104c26af0e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-dtptn\" (UID: \"0a0e028c-e9eb-4260-bfa3-37104c26af0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dtptn"
Oct 30 07:39:58.578494 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:58.578468    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a0e028c-e9eb-4260-bfa3-37104c26af0e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-dtptn\" (UID: \"0a0e028c-e9eb-4260-bfa3-37104c26af0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dtptn"
Oct 30 07:39:58.578634 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:58.578610    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a0e028c-e9eb-4260-bfa3-37104c26af0e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-dtptn\" (UID: \"0a0e028c-e9eb-4260-bfa3-37104c26af0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dtptn"
Oct 30 07:39:58.601688 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:58.601660    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a0e028c-e9eb-4260-bfa3-37104c26af0e-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-dtptn\" (UID: \"0a0e028c-e9eb-4260-bfa3-37104c26af0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dtptn"
Oct 30 07:39:58.601869 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:58.601825    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bhmd\" (UniqueName: \"kubernetes.io/projected/0a0e028c-e9eb-4260-bfa3-37104c26af0e-kube-api-access-2bhmd\") pod \"maas-default-gateway-openshift-default-6dcf757899-dtptn\" (UID: \"0a0e028c-e9eb-4260-bfa3-37104c26af0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dtptn"
Oct 30 07:39:58.708267 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:58.708192    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dtptn"
Oct 30 07:39:59.025404 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:39:59.025368    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a0e028c_e9eb_4260_bfa3_37104c26af0e.slice/crio-59a6482eb088c36fc7a30dddad74708643ca1448ea2bfcd4a4a5f314edb81d6a WatchSource:0}: Error finding container 59a6482eb088c36fc7a30dddad74708643ca1448ea2bfcd4a4a5f314edb81d6a: Status 404 returned error can't find the container with id 59a6482eb088c36fc7a30dddad74708643ca1448ea2bfcd4a4a5f314edb81d6a
Oct 30 07:39:59.026879 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:59.026848    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dtptn"]
Oct 30 07:39:59.028445 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:59.028415    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:39:59.028511 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:59.028476    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:39:59.028511 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:59.028504    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:39:59.215895 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:59.215845    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dtptn" event={"ID":"0a0e028c-e9eb-4260-bfa3-37104c26af0e","Type":"ContainerStarted","Data":"38419b1742aa3fdf56b3aca550444cdbdcfa1a9ebf7069bf5ffecde45b004f61"}
Oct 30 07:39:59.216077 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:59.215925    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dtptn" event={"ID":"0a0e028c-e9eb-4260-bfa3-37104c26af0e","Type":"ContainerStarted","Data":"59a6482eb088c36fc7a30dddad74708643ca1448ea2bfcd4a4a5f314edb81d6a"}
Oct 30 07:39:59.267061 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:59.266999    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dtptn" podStartSLOduration=1.266981536 podStartE2EDuration="1.266981536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:39:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:39:59.264148514 +0000 UTC m=+2557.873555485" watchObservedRunningTime="2026-10-30 07:39:59.266981536 +0000 UTC m=+2557.876388507"
Oct 30 07:39:59.708432 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:39:59.708400    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dtptn"
Oct 30 07:40:00.720919 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:00.720056    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-dtptn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Oct 30 07:40:00.720919 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:00.720114    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dtptn" podUID="0a0e028c-e9eb-4260-bfa3-37104c26af0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Oct 30 07:40:01.708534 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:01.708494    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-dtptn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Oct 30 07:40:01.708723 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:01.708559    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dtptn" podUID="0a0e028c-e9eb-4260-bfa3-37104c26af0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Oct 30 07:40:01.861038 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:40:01.860996    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:40:02.391140 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:02.391106    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-866cjr7gk"]
Oct 30 07:40:02.396851 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:02.396798    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-866cjr7gk"
Oct 30 07:40:02.406775 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:02.406742    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-866cjr7gk"]
Oct 30 07:40:02.416389 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:02.416365    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e27e0c\"/\"kube-root-ca.crt\""
Oct 30 07:40:02.416647 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:02.416630    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e27e0c\"/\"routing-overlay\""
Oct 30 07:40:02.416953 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:02.416934    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e27e0c\"/\"payload-processing-external-model-e2e--e3e25a9c-dockercfg-bdjwm\""
Oct 30 07:40:02.417162 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:02.417147    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e27e0c\"/\"agc-cred-17bac3af1751\""
Oct 30 07:40:02.428647 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:02.428623    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e27e0c\"/\"openshift-service-ca.crt\""
Oct 30 07:40:02.428788 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:02.428684    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e27e0c\"/\"payload-processing-external-model-plugins-e2e-extmod-b-e27e0c\""
Oct 30 07:40:02.460954 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:02.460924    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkqhn\" (UniqueName: \"kubernetes.io/projected/23d95410-a777-4c72-a377-3a3495fa0a15-kube-api-access-lkqhn\") pod \"payload-processing-external-model-e2e-extmod-b-e27e0c-866cjr7gk\" (UID: \"23d95410-a777-4c72-a377-3a3495fa0a15\") " pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-866cjr7gk"
Oct 30 07:40:02.461138 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:02.461113    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23d95410-a777-4c72-a377-3a3495fa0a15-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e27e0c-866cjr7gk\" (UID: \"23d95410-a777-4c72-a377-3a3495fa0a15\") " pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-866cjr7gk"
Oct 30 07:40:02.461297 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:02.461275    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/23d95410-a777-4c72-a377-3a3495fa0a15-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e27e0c-866cjr7gk\" (UID: \"23d95410-a777-4c72-a377-3a3495fa0a15\") " pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-866cjr7gk"
Oct 30 07:40:02.461357 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:02.461342    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/23d95410-a777-4c72-a377-3a3495fa0a15-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e27e0c-866cjr7gk\" (UID: \"23d95410-a777-4c72-a377-3a3495fa0a15\") " pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-866cjr7gk"
Oct 30 07:40:02.562739 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:02.562707    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23d95410-a777-4c72-a377-3a3495fa0a15-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e27e0c-866cjr7gk\" (UID: \"23d95410-a777-4c72-a377-3a3495fa0a15\") " pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-866cjr7gk"
Oct 30 07:40:02.562992 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:02.562862    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/23d95410-a777-4c72-a377-3a3495fa0a15-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e27e0c-866cjr7gk\" (UID: \"23d95410-a777-4c72-a377-3a3495fa0a15\") " pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-866cjr7gk"
Oct 30 07:40:02.562992 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:02.562920    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/23d95410-a777-4c72-a377-3a3495fa0a15-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e27e0c-866cjr7gk\" (UID: \"23d95410-a777-4c72-a377-3a3495fa0a15\") " pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-866cjr7gk"
Oct 30 07:40:02.563139 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:02.563002    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkqhn\" (UniqueName: \"kubernetes.io/projected/23d95410-a777-4c72-a377-3a3495fa0a15-kube-api-access-lkqhn\") pod \"payload-processing-external-model-e2e-extmod-b-e27e0c-866cjr7gk\" (UID: \"23d95410-a777-4c72-a377-3a3495fa0a15\") " pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-866cjr7gk"
Oct 30 07:40:02.583831 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:02.563674    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23d95410-a777-4c72-a377-3a3495fa0a15-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e27e0c-866cjr7gk\" (UID: \"23d95410-a777-4c72-a377-3a3495fa0a15\") " pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-866cjr7gk"
Oct 30 07:40:02.584918 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:02.584886    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/23d95410-a777-4c72-a377-3a3495fa0a15-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e27e0c-866cjr7gk\" (UID: \"23d95410-a777-4c72-a377-3a3495fa0a15\") " pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-866cjr7gk"
Oct 30 07:40:02.588731 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:02.588697    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/23d95410-a777-4c72-a377-3a3495fa0a15-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e27e0c-866cjr7gk\" (UID: \"23d95410-a777-4c72-a377-3a3495fa0a15\") " pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-866cjr7gk"
Oct 30 07:40:02.590614 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:02.590587    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkqhn\" (UniqueName: \"kubernetes.io/projected/23d95410-a777-4c72-a377-3a3495fa0a15-kube-api-access-lkqhn\") pod \"payload-processing-external-model-e2e-extmod-b-e27e0c-866cjr7gk\" (UID: \"23d95410-a777-4c72-a377-3a3495fa0a15\") " pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-866cjr7gk"
Oct 30 07:40:02.709338 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:02.709250    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-dtptn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Oct 30 07:40:02.709601 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:02.709570    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dtptn" podUID="0a0e028c-e9eb-4260-bfa3-37104c26af0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Oct 30 07:40:02.717079 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:02.717047    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-866cjr7gk"
Oct 30 07:40:03.091611 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:40:03.091503    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23d95410_a777_4c72_a377_3a3495fa0a15.slice/crio-8e5ae3d6df8deb6d110926d04c0eec359f7f5092b6afbdf3ed855abc49d488f4 WatchSource:0}: Error finding container 8e5ae3d6df8deb6d110926d04c0eec359f7f5092b6afbdf3ed855abc49d488f4: Status 404 returned error can't find the container with id 8e5ae3d6df8deb6d110926d04c0eec359f7f5092b6afbdf3ed855abc49d488f4
Oct 30 07:40:03.095568 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:03.095538    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-866cjr7gk"]
Oct 30 07:40:03.306270 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:03.306234    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-866cjr7gk" event={"ID":"23d95410-a777-4c72-a377-3a3495fa0a15","Type":"ContainerStarted","Data":"afab0df3e9b78baf31c86511ea425f7447e6829134f0af12dead17c9363f1ad6"}
Oct 30 07:40:03.306470 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:03.306297    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-866cjr7gk" event={"ID":"23d95410-a777-4c72-a377-3a3495fa0a15","Type":"ContainerStarted","Data":"8e5ae3d6df8deb6d110926d04c0eec359f7f5092b6afbdf3ed855abc49d488f4"}
Oct 30 07:40:03.306947 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:03.306913    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-866cjr7gk"
Oct 30 07:40:03.351885 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:03.351759    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-866cjr7gk" podStartSLOduration=1.351745339 podStartE2EDuration="1.351745339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:40:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:40:03.351578699 +0000 UTC m=+2561.960985669" watchObservedRunningTime="2026-10-30 07:40:03.351745339 +0000 UTC m=+2561.961152305"
Oct 30 07:40:03.709176 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:03.709088    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-dtptn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Oct 30 07:40:03.709176 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:03.709150    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dtptn" podUID="0a0e028c-e9eb-4260-bfa3-37104c26af0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Oct 30 07:40:04.709024 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:04.708982    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-dtptn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Oct 30 07:40:04.709450 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:04.709040    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dtptn" podUID="0a0e028c-e9eb-4260-bfa3-37104c26af0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Oct 30 07:40:05.708923 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:05.708885    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-dtptn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Oct 30 07:40:05.709103 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:05.708941    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dtptn" podUID="0a0e028c-e9eb-4260-bfa3-37104c26af0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Oct 30 07:40:06.708820 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:06.708781    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-dtptn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Oct 30 07:40:06.709000 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:06.708847    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dtptn" podUID="0a0e028c-e9eb-4260-bfa3-37104c26af0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Oct 30 07:40:07.713542 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:07.713513    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dtptn"
Oct 30 07:40:07.714051 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:07.714021    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dtptn"
Oct 30 07:40:07.714680 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:07.714658    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dtptn"
Oct 30 07:40:14.316402 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:14.316375    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-866cjr7gk"
Oct 30 07:40:15.106709 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:15.106671    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:40:18.995831 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:18.995071    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6855h6kll"]
Oct 30 07:40:19.011830 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:19.008455    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6855h6kll"
Oct 30 07:40:19.038414 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:19.034070    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9bec5c\"/\"routing-overlay\""
Oct 30 07:40:19.038414 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:19.034268    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-9bec5c\"/\"payload-processing-external-model-e2e--314725d2-dockercfg-2wsrt\""
Oct 30 07:40:19.038414 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:19.034326    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-9bec5c\"/\"agc-cred-7acbf0effedd\""
Oct 30 07:40:19.054875 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:19.050947    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6855h6kll"]
Oct 30 07:40:19.098732 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:19.067608    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b13977e9-af67-4be1-a07e-87005a6134c8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-6855h6kll\" (UID: \"b13977e9-af67-4be1-a07e-87005a6134c8\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6855h6kll"
Oct 30 07:40:19.098732 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:19.067746    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fz8q\" (UniqueName: \"kubernetes.io/projected/b13977e9-af67-4be1-a07e-87005a6134c8-kube-api-access-5fz8q\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-6855h6kll\" (UID: \"b13977e9-af67-4be1-a07e-87005a6134c8\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6855h6kll"
Oct 30 07:40:19.098732 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:19.067970    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b13977e9-af67-4be1-a07e-87005a6134c8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-6855h6kll\" (UID: \"b13977e9-af67-4be1-a07e-87005a6134c8\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6855h6kll"
Oct 30 07:40:19.098732 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:19.068008    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b13977e9-af67-4be1-a07e-87005a6134c8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-6855h6kll\" (UID: \"b13977e9-af67-4be1-a07e-87005a6134c8\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6855h6kll"
Oct 30 07:40:19.131843 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:19.128109    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9bec5c\"/\"payload-processing-external-model-plugins-e2e-extmod-a-9bec5c\""
Oct 30 07:40:19.131843 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:19.128199    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9bec5c\"/\"openshift-service-ca.crt\""
Oct 30 07:40:19.131843 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:19.128299    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9bec5c\"/\"kube-root-ca.crt\""
Oct 30 07:40:19.172060 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:19.172019    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b13977e9-af67-4be1-a07e-87005a6134c8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-6855h6kll\" (UID: \"b13977e9-af67-4be1-a07e-87005a6134c8\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6855h6kll"
Oct 30 07:40:19.172341 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:19.172315    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b13977e9-af67-4be1-a07e-87005a6134c8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-6855h6kll\" (UID: \"b13977e9-af67-4be1-a07e-87005a6134c8\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6855h6kll"
Oct 30 07:40:19.172512 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:19.172494    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b13977e9-af67-4be1-a07e-87005a6134c8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-6855h6kll\" (UID: \"b13977e9-af67-4be1-a07e-87005a6134c8\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6855h6kll"
Oct 30 07:40:19.172747 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:19.172727    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fz8q\" (UniqueName: \"kubernetes.io/projected/b13977e9-af67-4be1-a07e-87005a6134c8-kube-api-access-5fz8q\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-6855h6kll\" (UID: \"b13977e9-af67-4be1-a07e-87005a6134c8\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6855h6kll"
Oct 30 07:40:19.189839 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:19.177148    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b13977e9-af67-4be1-a07e-87005a6134c8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-6855h6kll\" (UID: \"b13977e9-af67-4be1-a07e-87005a6134c8\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6855h6kll"
Oct 30 07:40:19.189839 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:19.178378    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b13977e9-af67-4be1-a07e-87005a6134c8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-6855h6kll\" (UID: \"b13977e9-af67-4be1-a07e-87005a6134c8\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6855h6kll"
Oct 30 07:40:19.189839 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:19.186798    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b13977e9-af67-4be1-a07e-87005a6134c8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-6855h6kll\" (UID: \"b13977e9-af67-4be1-a07e-87005a6134c8\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6855h6kll"
Oct 30 07:40:19.217893 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:19.217847    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fz8q\" (UniqueName: \"kubernetes.io/projected/b13977e9-af67-4be1-a07e-87005a6134c8-kube-api-access-5fz8q\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-6855h6kll\" (UID: \"b13977e9-af67-4be1-a07e-87005a6134c8\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6855h6kll"
Oct 30 07:40:19.352739 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:19.352703    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6855h6kll"
Oct 30 07:40:19.725373 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:19.724496    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6855h6kll"]
Oct 30 07:40:20.498907 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:20.498871    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6855h6kll"]
Oct 30 07:40:20.576324 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:20.576280    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6855h6kll" event={"ID":"b13977e9-af67-4be1-a07e-87005a6134c8","Type":"ContainerStarted","Data":"91525a71939d011a73bcae849f97b13aad250fa0ba9cf9fa491b46f73bcbc67d"}
Oct 30 07:40:20.576493 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:20.576351    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6855h6kll" event={"ID":"b13977e9-af67-4be1-a07e-87005a6134c8","Type":"ContainerStarted","Data":"d2e9cd237d451b376839395a21e05c85c27da42037948f63eb59ef06eaa27dbd"}
Oct 30 07:40:20.576952 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:20.576929    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6855h6kll"
Oct 30 07:40:20.659036 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:20.658989    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6855h6kll" podStartSLOduration=2.658976156 podStartE2EDuration="2.658976156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:40:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:40:20.656674399 +0000 UTC m=+2579.266081370" watchObservedRunningTime="2026-10-30 07:40:20.658976156 +0000 UTC m=+2579.268383126"
Oct 30 07:40:20.757906 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:40:20.753700    6224 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-9bec5c/agc-cred-7acbf0effedd: secret "agc-cred-7acbf0effedd" not found
Oct 30 07:40:20.758223 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:40:20.758203    6224 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6855h6kll: secret "agc-cred-7acbf0effedd" not found
Oct 30 07:40:20.758456 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:40:20.758443    6224 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b13977e9-af67-4be1-a07e-87005a6134c8-provider-credentials podName:b13977e9-af67-4be1-a07e-87005a6134c8 nodeName:}" failed. No retries permitted until 2026-10-30 07:40:21.258417937 +0000 UTC m=+2579.867824899 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b13977e9-af67-4be1-a07e-87005a6134c8-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-9bec5c-6855h6kll" (UID: "b13977e9-af67-4be1-a07e-87005a6134c8") : secret "agc-cred-7acbf0effedd" not found
Oct 30 07:40:20.761112 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:40:20.758164    6224 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-plugins-e2e-extmod-a-9bec5c: configmap "payload-processing-external-model-plugins-e2e-extmod-a-9bec5c" not found
Oct 30 07:40:20.771675 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:40:20.771656    6224 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b13977e9-af67-4be1-a07e-87005a6134c8-plugins-config-volume podName:b13977e9-af67-4be1-a07e-87005a6134c8 nodeName:}" failed. No retries permitted until 2026-10-30 07:40:21.271633443 +0000 UTC m=+2579.881040415 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/b13977e9-af67-4be1-a07e-87005a6134c8-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-9bec5c-6855h6kll" (UID: "b13977e9-af67-4be1-a07e-87005a6134c8") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-9bec5c" not found
Oct 30 07:40:21.264437 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:40:21.264384    6224 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-9bec5c/agc-cred-7acbf0effedd: secret "agc-cred-7acbf0effedd" not found
Oct 30 07:40:21.264437 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:40:21.264435    6224 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6855h6kll: secret "agc-cred-7acbf0effedd" not found
Oct 30 07:40:21.264755 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:40:21.264512    6224 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b13977e9-af67-4be1-a07e-87005a6134c8-provider-credentials podName:b13977e9-af67-4be1-a07e-87005a6134c8 nodeName:}" failed. No retries permitted until 2026-10-30 07:40:22.264481907 +0000 UTC m=+2580.873888870 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b13977e9-af67-4be1-a07e-87005a6134c8-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-9bec5c-6855h6kll" (UID: "b13977e9-af67-4be1-a07e-87005a6134c8") : secret "agc-cred-7acbf0effedd" not found
Oct 30 07:40:21.365443 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:40:21.365409    6224 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-plugins-e2e-extmod-a-9bec5c: configmap "payload-processing-external-model-plugins-e2e-extmod-a-9bec5c" not found
Oct 30 07:40:21.365648 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:40:21.365482    6224 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b13977e9-af67-4be1-a07e-87005a6134c8-plugins-config-volume podName:b13977e9-af67-4be1-a07e-87005a6134c8 nodeName:}" failed. No retries permitted until 2026-10-30 07:40:22.36546721 +0000 UTC m=+2580.974874182 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/b13977e9-af67-4be1-a07e-87005a6134c8-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-9bec5c-6855h6kll" (UID: "b13977e9-af67-4be1-a07e-87005a6134c8") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-9bec5c" not found
Oct 30 07:40:22.277086 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:40:22.277055    6224 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-9bec5c/agc-cred-7acbf0effedd: secret "agc-cred-7acbf0effedd" not found
Oct 30 07:40:22.277086 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:40:22.277081    6224 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6855h6kll: secret "agc-cred-7acbf0effedd" not found
Oct 30 07:40:22.277535 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:40:22.277143    6224 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b13977e9-af67-4be1-a07e-87005a6134c8-provider-credentials podName:b13977e9-af67-4be1-a07e-87005a6134c8 nodeName:}" failed. No retries permitted until 2026-10-30 07:40:24.27712811 +0000 UTC m=+2582.886535062 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b13977e9-af67-4be1-a07e-87005a6134c8-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-9bec5c-6855h6kll" (UID: "b13977e9-af67-4be1-a07e-87005a6134c8") : secret "agc-cred-7acbf0effedd" not found
Oct 30 07:40:22.378159 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:40:22.378128    6224 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-plugins-e2e-extmod-a-9bec5c: configmap "payload-processing-external-model-plugins-e2e-extmod-a-9bec5c" not found
Oct 30 07:40:22.378333 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:40:22.378201    6224 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b13977e9-af67-4be1-a07e-87005a6134c8-plugins-config-volume podName:b13977e9-af67-4be1-a07e-87005a6134c8 nodeName:}" failed. No retries permitted until 2026-10-30 07:40:24.378185673 +0000 UTC m=+2582.987592628 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/b13977e9-af67-4be1-a07e-87005a6134c8-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-9bec5c-6855h6kll" (UID: "b13977e9-af67-4be1-a07e-87005a6134c8") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-9bec5c" not found
Oct 30 07:40:24.309603 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:40:24.309572    6224 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-9bec5c/agc-cred-7acbf0effedd: secret "agc-cred-7acbf0effedd" not found
Oct 30 07:40:24.309603 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:40:24.309598    6224 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6855h6kll: secret "agc-cred-7acbf0effedd" not found
Oct 30 07:40:24.310022 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:40:24.309655    6224 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b13977e9-af67-4be1-a07e-87005a6134c8-provider-credentials podName:b13977e9-af67-4be1-a07e-87005a6134c8 nodeName:}" failed. No retries permitted until 2026-10-30 07:40:28.309639656 +0000 UTC m=+2586.919046604 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b13977e9-af67-4be1-a07e-87005a6134c8-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-9bec5c-6855h6kll" (UID: "b13977e9-af67-4be1-a07e-87005a6134c8") : secret "agc-cred-7acbf0effedd" not found
Oct 30 07:40:24.410588 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:40:24.410553    6224 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-plugins-e2e-extmod-a-9bec5c: configmap "payload-processing-external-model-plugins-e2e-extmod-a-9bec5c" not found
Oct 30 07:40:24.410780 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:40:24.410655    6224 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b13977e9-af67-4be1-a07e-87005a6134c8-plugins-config-volume podName:b13977e9-af67-4be1-a07e-87005a6134c8 nodeName:}" failed. No retries permitted until 2026-10-30 07:40:28.410640006 +0000 UTC m=+2587.020046955 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/b13977e9-af67-4be1-a07e-87005a6134c8-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-9bec5c-6855h6kll" (UID: "b13977e9-af67-4be1-a07e-87005a6134c8") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-9bec5c" not found
Oct 30 07:40:26.597742 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:26.597700    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6855h6kll" podUID="b13977e9-af67-4be1-a07e-87005a6134c8" containerName="payload-processing" containerID="cri-o://91525a71939d011a73bcae849f97b13aad250fa0ba9cf9fa491b46f73bcbc67d" gracePeriod=25
Oct 30 07:40:26.868942 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:26.868883    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6855h6kll"
Oct 30 07:40:26.943995 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:26.943961    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b13977e9-af67-4be1-a07e-87005a6134c8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b13977e9-af67-4be1-a07e-87005a6134c8-routing-overlay-volume\") pod \"b13977e9-af67-4be1-a07e-87005a6134c8\" (UID: \"b13977e9-af67-4be1-a07e-87005a6134c8\") "
Oct 30 07:40:26.944171 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:26.944062    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b13977e9-af67-4be1-a07e-87005a6134c8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b13977e9-af67-4be1-a07e-87005a6134c8-provider-credentials\") pod \"b13977e9-af67-4be1-a07e-87005a6134c8\" (UID: \"b13977e9-af67-4be1-a07e-87005a6134c8\") "
Oct 30 07:40:26.944246 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:26.944220    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b13977e9-af67-4be1-a07e-87005a6134c8-kube-api-access-5fz8q\" (UniqueName: \"kubernetes.io/projected/b13977e9-af67-4be1-a07e-87005a6134c8-kube-api-access-5fz8q\") pod \"b13977e9-af67-4be1-a07e-87005a6134c8\" (UID: \"b13977e9-af67-4be1-a07e-87005a6134c8\") "
Oct 30 07:40:26.944327 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:26.944309    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b13977e9-af67-4be1-a07e-87005a6134c8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b13977e9-af67-4be1-a07e-87005a6134c8-plugins-config-volume\") pod \"b13977e9-af67-4be1-a07e-87005a6134c8\" (UID: \"b13977e9-af67-4be1-a07e-87005a6134c8\") "
Oct 30 07:40:26.944387 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:26.944365    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b13977e9-af67-4be1-a07e-87005a6134c8-routing-overlay-volume" pod "b13977e9-af67-4be1-a07e-87005a6134c8" (UID: "b13977e9-af67-4be1-a07e-87005a6134c8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:40:26.944789 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:26.944765    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b13977e9-af67-4be1-a07e-87005a6134c8-plugins-config-volume" pod "b13977e9-af67-4be1-a07e-87005a6134c8" (UID: "b13977e9-af67-4be1-a07e-87005a6134c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:40:26.945137 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:26.945105    6224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b13977e9-af67-4be1-a07e-87005a6134c8-plugins-config-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:40:26.945224 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:26.945143    6224 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b13977e9-af67-4be1-a07e-87005a6134c8-routing-overlay-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:40:26.946782 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:26.946755    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b13977e9-af67-4be1-a07e-87005a6134c8-provider-credentials" pod "b13977e9-af67-4be1-a07e-87005a6134c8" (UID: "b13977e9-af67-4be1-a07e-87005a6134c8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:40:26.947010 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:26.946987    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b13977e9-af67-4be1-a07e-87005a6134c8-kube-api-access-5fz8q" pod "b13977e9-af67-4be1-a07e-87005a6134c8" (UID: "b13977e9-af67-4be1-a07e-87005a6134c8"). InnerVolumeSpecName "kube-api-access-5fz8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:40:27.047005 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:27.046977    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fz8q\" (UniqueName: \"kubernetes.io/projected/b13977e9-af67-4be1-a07e-87005a6134c8-kube-api-access-5fz8q\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:40:27.047005 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:27.047002    6224 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b13977e9-af67-4be1-a07e-87005a6134c8-provider-credentials\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:40:27.642399 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:27.642367    6224 generic.go:356] "Generic (PLEG): container finished" podID="b13977e9-af67-4be1-a07e-87005a6134c8" containerID="91525a71939d011a73bcae849f97b13aad250fa0ba9cf9fa491b46f73bcbc67d" exitCode=0
Oct 30 07:40:27.642867 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:27.642438    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6855h6kll" event={"ID":"b13977e9-af67-4be1-a07e-87005a6134c8","Type":"ContainerDied","Data":"91525a71939d011a73bcae849f97b13aad250fa0ba9cf9fa491b46f73bcbc67d"}
Oct 30 07:40:27.642867 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:27.642495    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6855h6kll" event={"ID":"b13977e9-af67-4be1-a07e-87005a6134c8","Type":"ContainerDied","Data":"d2e9cd237d451b376839395a21e05c85c27da42037948f63eb59ef06eaa27dbd"}
Oct 30 07:40:27.642867 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:27.642439    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6855h6kll"
Oct 30 07:40:27.642867 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:27.642515    6224 scope.go:122] "RemoveContainer" containerID="91525a71939d011a73bcae849f97b13aad250fa0ba9cf9fa491b46f73bcbc67d"
Oct 30 07:40:27.652722 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:27.652694    6224 scope.go:122] "RemoveContainer" containerID="91525a71939d011a73bcae849f97b13aad250fa0ba9cf9fa491b46f73bcbc67d"
Oct 30 07:40:27.653036 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:40:27.653005    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91525a71939d011a73bcae849f97b13aad250fa0ba9cf9fa491b46f73bcbc67d\": container with ID starting with 91525a71939d011a73bcae849f97b13aad250fa0ba9cf9fa491b46f73bcbc67d not found: ID does not exist" containerID="91525a71939d011a73bcae849f97b13aad250fa0ba9cf9fa491b46f73bcbc67d"
Oct 30 07:40:27.653147 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:27.653044    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91525a71939d011a73bcae849f97b13aad250fa0ba9cf9fa491b46f73bcbc67d"} err="failed to get container status \"91525a71939d011a73bcae849f97b13aad250fa0ba9cf9fa491b46f73bcbc67d\": rpc error: code = NotFound desc = could not find container \"91525a71939d011a73bcae849f97b13aad250fa0ba9cf9fa491b46f73bcbc67d\": container with ID starting with 91525a71939d011a73bcae849f97b13aad250fa0ba9cf9fa491b46f73bcbc67d not found: ID does not exist"
Oct 30 07:40:27.701969 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:27.701944    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6855h6kll"]
Oct 30 07:40:27.714996 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:27.714955    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-6855h6kll"]
Oct 30 07:40:29.504232 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:29.504195    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b13977e9-af67-4be1-a07e-87005a6134c8" path="/var/lib/kubelet/pods/b13977e9-af67-4be1-a07e-87005a6134c8/volumes"
Oct 30 07:40:31.857773 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:40:31.857731    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:40:35.871649 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:35.871607    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8e7948/payload-processing-external-model-e2e-extmod-a-8e7948-6d8dbfhct"]
Oct 30 07:40:38.426523 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:38.426479    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-79fbc7tg8"]
Oct 30 07:40:38.427192 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:38.427175    6224 cpu_manager.go:414] "containerMap: removing container" podUID="b13977e9-af67-4be1-a07e-87005a6134c8" containerName="payload-processing"
Oct 30 07:40:38.427238 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:38.427194    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b13977e9-af67-4be1-a07e-87005a6134c8" containerName="payload-processing"
Oct 30 07:40:38.427342 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:38.427332    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="b13977e9-af67-4be1-a07e-87005a6134c8" containerName="payload-processing"
Oct 30 07:40:38.432295 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:38.432275    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-79fbc7tg8"
Oct 30 07:40:38.452055 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:38.452022    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-79fbc7tg8"]
Oct 30 07:40:38.461759 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:38.461734    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-9bec5c\"/\"agc-cred-6cd15efdbb3d\""
Oct 30 07:40:38.461759 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:38.461759    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-9bec5c\"/\"payload-processing-external-model-e2e--314725d2-dockercfg-fpjr8\""
Oct 30 07:40:38.461759 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:38.461757    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9bec5c\"/\"openshift-service-ca.crt\""
Oct 30 07:40:38.462067 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:38.461738    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9bec5c\"/\"payload-processing-external-model-plugins-e2e-extmod-a-9bec5c\""
Oct 30 07:40:38.465411 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:38.465387    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9bec5c\"/\"routing-overlay\""
Oct 30 07:40:38.465524 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:38.465408    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9bec5c\"/\"kube-root-ca.crt\""
Oct 30 07:40:38.510829 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:38.510781    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e93f3828-749e-4c9c-95e5-e826ec289fb6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-79fbc7tg8\" (UID: \"e93f3828-749e-4c9c-95e5-e826ec289fb6\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-79fbc7tg8"
Oct 30 07:40:38.511002 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:38.510978    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e93f3828-749e-4c9c-95e5-e826ec289fb6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-79fbc7tg8\" (UID: \"e93f3828-749e-4c9c-95e5-e826ec289fb6\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-79fbc7tg8"
Oct 30 07:40:38.511108 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:38.511086    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn2mb\" (UniqueName: \"kubernetes.io/projected/e93f3828-749e-4c9c-95e5-e826ec289fb6-kube-api-access-bn2mb\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-79fbc7tg8\" (UID: \"e93f3828-749e-4c9c-95e5-e826ec289fb6\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-79fbc7tg8"
Oct 30 07:40:38.511268 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:38.511243    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e93f3828-749e-4c9c-95e5-e826ec289fb6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-79fbc7tg8\" (UID: \"e93f3828-749e-4c9c-95e5-e826ec289fb6\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-79fbc7tg8"
Oct 30 07:40:38.612579 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:38.612535    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e93f3828-749e-4c9c-95e5-e826ec289fb6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-79fbc7tg8\" (UID: \"e93f3828-749e-4c9c-95e5-e826ec289fb6\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-79fbc7tg8"
Oct 30 07:40:38.612755 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:38.612612    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e93f3828-749e-4c9c-95e5-e826ec289fb6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-79fbc7tg8\" (UID: \"e93f3828-749e-4c9c-95e5-e826ec289fb6\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-79fbc7tg8"
Oct 30 07:40:38.612755 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:38.612694    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn2mb\" (UniqueName: \"kubernetes.io/projected/e93f3828-749e-4c9c-95e5-e826ec289fb6-kube-api-access-bn2mb\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-79fbc7tg8\" (UID: \"e93f3828-749e-4c9c-95e5-e826ec289fb6\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-79fbc7tg8"
Oct 30 07:40:38.612921 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:38.612844    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e93f3828-749e-4c9c-95e5-e826ec289fb6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-79fbc7tg8\" (UID: \"e93f3828-749e-4c9c-95e5-e826ec289fb6\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-79fbc7tg8"
Oct 30 07:40:38.613578 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:38.613542    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e93f3828-749e-4c9c-95e5-e826ec289fb6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-79fbc7tg8\" (UID: \"e93f3828-749e-4c9c-95e5-e826ec289fb6\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-79fbc7tg8"
Oct 30 07:40:38.613751 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:38.613724    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e93f3828-749e-4c9c-95e5-e826ec289fb6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-79fbc7tg8\" (UID: \"e93f3828-749e-4c9c-95e5-e826ec289fb6\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-79fbc7tg8"
Oct 30 07:40:38.615302 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:38.615276    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e93f3828-749e-4c9c-95e5-e826ec289fb6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-79fbc7tg8\" (UID: \"e93f3828-749e-4c9c-95e5-e826ec289fb6\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-79fbc7tg8"
Oct 30 07:40:38.641498 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:38.641466    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn2mb\" (UniqueName: \"kubernetes.io/projected/e93f3828-749e-4c9c-95e5-e826ec289fb6-kube-api-access-bn2mb\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-79fbc7tg8\" (UID: \"e93f3828-749e-4c9c-95e5-e826ec289fb6\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-79fbc7tg8"
Oct 30 07:40:38.746270 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:38.746193    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-79fbc7tg8"
Oct 30 07:40:39.104137 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:39.103172    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-79fbc7tg8"]
Oct 30 07:40:39.110567 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:40:39.110362    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode93f3828_749e_4c9c_95e5_e826ec289fb6.slice/crio-620bedb279cec7c8ee777eb66bb6445b5714eef9c128f2cd534a7ed08d3adff9 WatchSource:0}: Error finding container 620bedb279cec7c8ee777eb66bb6445b5714eef9c128f2cd534a7ed08d3adff9: Status 404 returned error can't find the container with id 620bedb279cec7c8ee777eb66bb6445b5714eef9c128f2cd534a7ed08d3adff9
Oct 30 07:40:39.735362 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:39.735323    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-79fbc7tg8" event={"ID":"e93f3828-749e-4c9c-95e5-e826ec289fb6","Type":"ContainerStarted","Data":"0ba88591e215276434c7c8f9f5152000b7917687130c38e08fbbc020a21b346a"}
Oct 30 07:40:39.735846 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:39.735394    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-79fbc7tg8"
Oct 30 07:40:39.735846 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:39.735411    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-79fbc7tg8" event={"ID":"e93f3828-749e-4c9c-95e5-e826ec289fb6","Type":"ContainerStarted","Data":"620bedb279cec7c8ee777eb66bb6445b5714eef9c128f2cd534a7ed08d3adff9"}
Oct 30 07:40:39.782569 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:39.782525    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-79fbc7tg8" podStartSLOduration=1.782511664 podStartE2EDuration="1.782511664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:40:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:40:39.779931216 +0000 UTC m=+2598.389338187" watchObservedRunningTime="2026-10-30 07:40:39.782511664 +0000 UTC m=+2598.391918658"
Oct 30 07:40:40.873311 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:40.873272    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-8e7948/payload-processing-external-model-e2e-extmod-a-8e7948-6d8dbfhct" podUID="01fea4c7-b874-49e8-a8d5-c0b9b50413fb" containerName="payload-processing" containerID="cri-o://6a6497dd3113e468ee4bddac46f13225b86e65344a1e3dac15b292d5b7c6b6aa" gracePeriod=25
Oct 30 07:40:41.223500 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:41.223475    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8e7948/payload-processing-external-model-e2e-extmod-a-8e7948-6d8dbfhct"
Oct 30 07:40:41.257882 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:41.257848    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5b5975486-kzcd4"]
Oct 30 07:40:41.258499 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:41.258486    6224 cpu_manager.go:414] "containerMap: removing container" podUID="01fea4c7-b874-49e8-a8d5-c0b9b50413fb" containerName="payload-processing"
Oct 30 07:40:41.258546 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:41.258500    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01fea4c7-b874-49e8-a8d5-c0b9b50413fb" containerName="payload-processing"
Oct 30 07:40:41.258639 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:41.258628    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="01fea4c7-b874-49e8-a8d5-c0b9b50413fb" containerName="payload-processing"
Oct 30 07:40:41.262052 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:41.262033    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5b5975486-kzcd4"
Oct 30 07:40:41.277493 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:41.277464    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5b5975486-kzcd4"]
Oct 30 07:40:41.278861 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:41.278838    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-c204db61a2ae\""
Oct 30 07:40:41.282957 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:41.282927    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 30 07:40:41.283063 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:41.282968    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-8d2rr\""
Oct 30 07:40:41.283063 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:41.282970    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 30 07:40:41.283063 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:41.283051    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 30 07:40:41.283063 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:41.283056    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 30 07:40:41.354738 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:41.354702    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01fea4c7-b874-49e8-a8d5-c0b9b50413fb-kube-api-access-sqd9b\" (UniqueName: \"kubernetes.io/projected/01fea4c7-b874-49e8-a8d5-c0b9b50413fb-kube-api-access-sqd9b\") pod \"01fea4c7-b874-49e8-a8d5-c0b9b50413fb\" (UID: \"01fea4c7-b874-49e8-a8d5-c0b9b50413fb\") "
Oct 30 07:40:41.354975 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:41.354783    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01fea4c7-b874-49e8-a8d5-c0b9b50413fb-provider-credentials\" (UniqueName: \"kubernetes.io/projected/01fea4c7-b874-49e8-a8d5-c0b9b50413fb-provider-credentials\") pod \"01fea4c7-b874-49e8-a8d5-c0b9b50413fb\" (UID: \"01fea4c7-b874-49e8-a8d5-c0b9b50413fb\") "
Oct 30 07:40:41.354975 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:41.354863    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01fea4c7-b874-49e8-a8d5-c0b9b50413fb-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/01fea4c7-b874-49e8-a8d5-c0b9b50413fb-routing-overlay-volume\") pod \"01fea4c7-b874-49e8-a8d5-c0b9b50413fb\" (UID: \"01fea4c7-b874-49e8-a8d5-c0b9b50413fb\") "
Oct 30 07:40:41.355110 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:41.355020    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01fea4c7-b874-49e8-a8d5-c0b9b50413fb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01fea4c7-b874-49e8-a8d5-c0b9b50413fb-plugins-config-volume\") pod \"01fea4c7-b874-49e8-a8d5-c0b9b50413fb\" (UID: \"01fea4c7-b874-49e8-a8d5-c0b9b50413fb\") "
Oct 30 07:40:41.355367 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:41.355322    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01fea4c7-b874-49e8-a8d5-c0b9b50413fb-routing-overlay-volume" pod "01fea4c7-b874-49e8-a8d5-c0b9b50413fb" (UID: "01fea4c7-b874-49e8-a8d5-c0b9b50413fb"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:40:41.355503 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:41.355417    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01fea4c7-b874-49e8-a8d5-c0b9b50413fb-plugins-config-volume" pod "01fea4c7-b874-49e8-a8d5-c0b9b50413fb" (UID: "01fea4c7-b874-49e8-a8d5-c0b9b50413fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:40:41.355503 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:41.355449    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2fe939f-a5dd-4e9e-801d-ea58e22cadf8-provider-credentials\") pod \"payload-processing-external-model-5b5975486-kzcd4\" (UID: \"c2fe939f-a5dd-4e9e-801d-ea58e22cadf8\") " pod="models-as-a-service/payload-processing-external-model-5b5975486-kzcd4"
Oct 30 07:40:41.355637 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:41.355610    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2fe939f-a5dd-4e9e-801d-ea58e22cadf8-routing-overlay-volume\") pod \"payload-processing-external-model-5b5975486-kzcd4\" (UID: \"c2fe939f-a5dd-4e9e-801d-ea58e22cadf8\") " pod="models-as-a-service/payload-processing-external-model-5b5975486-kzcd4"
Oct 30 07:40:41.355916 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:41.355883    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7jvw\" (UniqueName: \"kubernetes.io/projected/c2fe939f-a5dd-4e9e-801d-ea58e22cadf8-kube-api-access-g7jvw\") pod \"payload-processing-external-model-5b5975486-kzcd4\" (UID: \"c2fe939f-a5dd-4e9e-801d-ea58e22cadf8\") " pod="models-as-a-service/payload-processing-external-model-5b5975486-kzcd4"
Oct 30 07:40:41.356035 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:41.355933    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2fe939f-a5dd-4e9e-801d-ea58e22cadf8-plugins-config-volume\") pod \"payload-processing-external-model-5b5975486-kzcd4\" (UID: \"c2fe939f-a5dd-4e9e-801d-ea58e22cadf8\") " pod="models-as-a-service/payload-processing-external-model-5b5975486-kzcd4"
Oct 30 07:40:41.356185 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:41.356168    6224 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/01fea4c7-b874-49e8-a8d5-c0b9b50413fb-routing-overlay-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:40:41.356399 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:41.356371    6224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01fea4c7-b874-49e8-a8d5-c0b9b50413fb-plugins-config-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:40:41.357666 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:41.357639    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01fea4c7-b874-49e8-a8d5-c0b9b50413fb-provider-credentials" pod "01fea4c7-b874-49e8-a8d5-c0b9b50413fb" (UID: "01fea4c7-b874-49e8-a8d5-c0b9b50413fb"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:40:41.357995 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:41.357969    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01fea4c7-b874-49e8-a8d5-c0b9b50413fb-kube-api-access-sqd9b" pod "01fea4c7-b874-49e8-a8d5-c0b9b50413fb" (UID: "01fea4c7-b874-49e8-a8d5-c0b9b50413fb"). InnerVolumeSpecName "kube-api-access-sqd9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:40:41.458187 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:41.458085    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2fe939f-a5dd-4e9e-801d-ea58e22cadf8-provider-credentials\") pod \"payload-processing-external-model-5b5975486-kzcd4\" (UID: \"c2fe939f-a5dd-4e9e-801d-ea58e22cadf8\") " pod="models-as-a-service/payload-processing-external-model-5b5975486-kzcd4"
Oct 30 07:40:41.458187 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:41.458160    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2fe939f-a5dd-4e9e-801d-ea58e22cadf8-routing-overlay-volume\") pod \"payload-processing-external-model-5b5975486-kzcd4\" (UID: \"c2fe939f-a5dd-4e9e-801d-ea58e22cadf8\") " pod="models-as-a-service/payload-processing-external-model-5b5975486-kzcd4"
Oct 30 07:40:41.458434 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:41.458262    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7jvw\" (UniqueName: \"kubernetes.io/projected/c2fe939f-a5dd-4e9e-801d-ea58e22cadf8-kube-api-access-g7jvw\") pod \"payload-processing-external-model-5b5975486-kzcd4\" (UID: \"c2fe939f-a5dd-4e9e-801d-ea58e22cadf8\") " pod="models-as-a-service/payload-processing-external-model-5b5975486-kzcd4"
Oct 30 07:40:41.458434 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:41.458284    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2fe939f-a5dd-4e9e-801d-ea58e22cadf8-plugins-config-volume\") pod \"payload-processing-external-model-5b5975486-kzcd4\" (UID: \"c2fe939f-a5dd-4e9e-801d-ea58e22cadf8\") " pod="models-as-a-service/payload-processing-external-model-5b5975486-kzcd4"
Oct 30 07:40:41.458434 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:41.458418    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqd9b\" (UniqueName: \"kubernetes.io/projected/01fea4c7-b874-49e8-a8d5-c0b9b50413fb-kube-api-access-sqd9b\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:40:41.458602 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:41.458436    6224 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/01fea4c7-b874-49e8-a8d5-c0b9b50413fb-provider-credentials\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:40:41.459219 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:41.459191    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2fe939f-a5dd-4e9e-801d-ea58e22cadf8-routing-overlay-volume\") pod \"payload-processing-external-model-5b5975486-kzcd4\" (UID: \"c2fe939f-a5dd-4e9e-801d-ea58e22cadf8\") " pod="models-as-a-service/payload-processing-external-model-5b5975486-kzcd4"
Oct 30 07:40:41.459329 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:41.459264    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2fe939f-a5dd-4e9e-801d-ea58e22cadf8-plugins-config-volume\") pod \"payload-processing-external-model-5b5975486-kzcd4\" (UID: \"c2fe939f-a5dd-4e9e-801d-ea58e22cadf8\") " pod="models-as-a-service/payload-processing-external-model-5b5975486-kzcd4"
Oct 30 07:40:41.468611 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:41.461938    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2fe939f-a5dd-4e9e-801d-ea58e22cadf8-provider-credentials\") pod \"payload-processing-external-model-5b5975486-kzcd4\" (UID: \"c2fe939f-a5dd-4e9e-801d-ea58e22cadf8\") " pod="models-as-a-service/payload-processing-external-model-5b5975486-kzcd4"
Oct 30 07:40:41.489821 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:41.489774    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7jvw\" (UniqueName: \"kubernetes.io/projected/c2fe939f-a5dd-4e9e-801d-ea58e22cadf8-kube-api-access-g7jvw\") pod \"payload-processing-external-model-5b5975486-kzcd4\" (UID: \"c2fe939f-a5dd-4e9e-801d-ea58e22cadf8\") " pod="models-as-a-service/payload-processing-external-model-5b5975486-kzcd4"
Oct 30 07:40:41.577585 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:41.577553    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5b5975486-kzcd4"
Oct 30 07:40:41.751109 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:41.751025    6224 generic.go:356] "Generic (PLEG): container finished" podID="01fea4c7-b874-49e8-a8d5-c0b9b50413fb" containerID="6a6497dd3113e468ee4bddac46f13225b86e65344a1e3dac15b292d5b7c6b6aa" exitCode=0
Oct 30 07:40:41.751109 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:41.751085    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8e7948/payload-processing-external-model-e2e-extmod-a-8e7948-6d8dbfhct"
Oct 30 07:40:41.751303 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:41.751100    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8e7948/payload-processing-external-model-e2e-extmod-a-8e7948-6d8dbfhct" event={"ID":"01fea4c7-b874-49e8-a8d5-c0b9b50413fb","Type":"ContainerDied","Data":"6a6497dd3113e468ee4bddac46f13225b86e65344a1e3dac15b292d5b7c6b6aa"}
Oct 30 07:40:41.751303 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:41.751157    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8e7948/payload-processing-external-model-e2e-extmod-a-8e7948-6d8dbfhct" event={"ID":"01fea4c7-b874-49e8-a8d5-c0b9b50413fb","Type":"ContainerDied","Data":"a64a5d4c120e1bf6d09652a10895d98d745641f0906b5c73a3c6919b641d41df"}
Oct 30 07:40:41.751303 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:41.751178    6224 scope.go:122] "RemoveContainer" containerID="6a6497dd3113e468ee4bddac46f13225b86e65344a1e3dac15b292d5b7c6b6aa"
Oct 30 07:40:41.761039 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:41.761022    6224 scope.go:122] "RemoveContainer" containerID="6a6497dd3113e468ee4bddac46f13225b86e65344a1e3dac15b292d5b7c6b6aa"
Oct 30 07:40:41.761337 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:40:41.761318    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a6497dd3113e468ee4bddac46f13225b86e65344a1e3dac15b292d5b7c6b6aa\": container with ID starting with 6a6497dd3113e468ee4bddac46f13225b86e65344a1e3dac15b292d5b7c6b6aa not found: ID does not exist" containerID="6a6497dd3113e468ee4bddac46f13225b86e65344a1e3dac15b292d5b7c6b6aa"
Oct 30 07:40:41.761381 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:41.761344    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a6497dd3113e468ee4bddac46f13225b86e65344a1e3dac15b292d5b7c6b6aa"} err="failed to get container status \"6a6497dd3113e468ee4bddac46f13225b86e65344a1e3dac15b292d5b7c6b6aa\": rpc error: code = NotFound desc = could not find container \"6a6497dd3113e468ee4bddac46f13225b86e65344a1e3dac15b292d5b7c6b6aa\": container with ID starting with 6a6497dd3113e468ee4bddac46f13225b86e65344a1e3dac15b292d5b7c6b6aa not found: ID does not exist"
Oct 30 07:40:41.810379 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:41.810342    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8e7948/payload-processing-external-model-e2e-extmod-a-8e7948-6d8dbfhct"]
Oct 30 07:40:41.816403 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:41.816379    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-8e7948/payload-processing-external-model-e2e-extmod-a-8e7948-6d8dbfhct"]
Oct 30 07:40:41.883195 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:40:41.883168    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2fe939f_a5dd_4e9e_801d_ea58e22cadf8.slice/crio-3ec49d2694fa079ec00132902461b980bab960f54b4c27a76b3caff307096e8b WatchSource:0}: Error finding container 3ec49d2694fa079ec00132902461b980bab960f54b4c27a76b3caff307096e8b: Status 404 returned error can't find the container with id 3ec49d2694fa079ec00132902461b980bab960f54b4c27a76b3caff307096e8b
Oct 30 07:40:41.883864 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:41.883719    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5b5975486-kzcd4"]
Oct 30 07:40:42.760884 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:42.760843    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5b5975486-kzcd4" event={"ID":"c2fe939f-a5dd-4e9e-801d-ea58e22cadf8","Type":"ContainerStarted","Data":"66d4fdd7ce58fdfa6bed19140b5eefa9d900e55fca00456c19536d2a408dcb23"}
Oct 30 07:40:42.761056 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:42.760905    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5b5975486-kzcd4" event={"ID":"c2fe939f-a5dd-4e9e-801d-ea58e22cadf8","Type":"ContainerStarted","Data":"3ec49d2694fa079ec00132902461b980bab960f54b4c27a76b3caff307096e8b"}
Oct 30 07:40:42.761056 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:42.760960    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5b5975486-kzcd4"
Oct 30 07:40:42.809164 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:42.809103    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5b5975486-kzcd4" podStartSLOduration=1.809090037 podStartE2EDuration="1.809090037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:40:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:40:42.808093115 +0000 UTC m=+2601.417500085" watchObservedRunningTime="2026-10-30 07:40:42.809090037 +0000 UTC m=+2601.418497007"
Oct 30 07:40:43.012728 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:43.012652    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-8467w656x"]
Oct 30 07:40:43.017023 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:43.017005    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-8467w656x"
Oct 30 07:40:43.036931 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:43.036907    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-8467w656x"]
Oct 30 07:40:43.038617 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:43.038588    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-c1b5d7\"/\"agc-cred-ad217a913a5d\""
Oct 30 07:40:43.038759 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:43.038739    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-c1b5d7\"/\"payload-processing-external-model-e2e--68f86622-dockercfg-7z9pp\""
Oct 30 07:40:43.038889 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:43.038866    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c1b5d7\"/\"kube-root-ca.crt\""
Oct 30 07:40:43.038960 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:43.038927    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c1b5d7\"/\"payload-processing-external-model-plugins-e2e-extmod-b-c1b5d7\""
Oct 30 07:40:43.038960 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:43.038927    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c1b5d7\"/\"openshift-service-ca.crt\""
Oct 30 07:40:43.039030 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:43.038978    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c1b5d7\"/\"routing-overlay\""
Oct 30 07:40:43.079265 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:43.079236    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0e58f1f7-4972-445e-833c-da31d8b40bf8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c1b5d7-8467w656x\" (UID: \"0e58f1f7-4972-445e-833c-da31d8b40bf8\") " pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-8467w656x"
Oct 30 07:40:43.079516 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:43.079495    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5khcs\" (UniqueName: \"kubernetes.io/projected/0e58f1f7-4972-445e-833c-da31d8b40bf8-kube-api-access-5khcs\") pod \"payload-processing-external-model-e2e-extmod-b-c1b5d7-8467w656x\" (UID: \"0e58f1f7-4972-445e-833c-da31d8b40bf8\") " pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-8467w656x"
Oct 30 07:40:43.079606 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:43.079564    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e58f1f7-4972-445e-833c-da31d8b40bf8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c1b5d7-8467w656x\" (UID: \"0e58f1f7-4972-445e-833c-da31d8b40bf8\") " pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-8467w656x"
Oct 30 07:40:43.079670 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:43.079618    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0e58f1f7-4972-445e-833c-da31d8b40bf8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c1b5d7-8467w656x\" (UID: \"0e58f1f7-4972-445e-833c-da31d8b40bf8\") " pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-8467w656x"
Oct 30 07:40:43.185824 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:43.181606    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5khcs\" (UniqueName: \"kubernetes.io/projected/0e58f1f7-4972-445e-833c-da31d8b40bf8-kube-api-access-5khcs\") pod \"payload-processing-external-model-e2e-extmod-b-c1b5d7-8467w656x\" (UID: \"0e58f1f7-4972-445e-833c-da31d8b40bf8\") " pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-8467w656x"
Oct 30 07:40:43.185824 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:43.181693    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e58f1f7-4972-445e-833c-da31d8b40bf8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c1b5d7-8467w656x\" (UID: \"0e58f1f7-4972-445e-833c-da31d8b40bf8\") " pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-8467w656x"
Oct 30 07:40:43.185824 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:43.181750    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0e58f1f7-4972-445e-833c-da31d8b40bf8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c1b5d7-8467w656x\" (UID: \"0e58f1f7-4972-445e-833c-da31d8b40bf8\") " pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-8467w656x"
Oct 30 07:40:43.185824 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:43.182560    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0e58f1f7-4972-445e-833c-da31d8b40bf8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c1b5d7-8467w656x\" (UID: \"0e58f1f7-4972-445e-833c-da31d8b40bf8\") " pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-8467w656x"
Oct 30 07:40:43.185824 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:43.183654    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e58f1f7-4972-445e-833c-da31d8b40bf8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c1b5d7-8467w656x\" (UID: \"0e58f1f7-4972-445e-833c-da31d8b40bf8\") " pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-8467w656x"
Oct 30 07:40:43.185824 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:43.183708    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0e58f1f7-4972-445e-833c-da31d8b40bf8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c1b5d7-8467w656x\" (UID: \"0e58f1f7-4972-445e-833c-da31d8b40bf8\") " pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-8467w656x"
Oct 30 07:40:43.192657 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:43.192618    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0e58f1f7-4972-445e-833c-da31d8b40bf8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c1b5d7-8467w656x\" (UID: \"0e58f1f7-4972-445e-833c-da31d8b40bf8\") " pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-8467w656x"
Oct 30 07:40:43.220694 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:43.220656    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5khcs\" (UniqueName: \"kubernetes.io/projected/0e58f1f7-4972-445e-833c-da31d8b40bf8-kube-api-access-5khcs\") pod \"payload-processing-external-model-e2e-extmod-b-c1b5d7-8467w656x\" (UID: \"0e58f1f7-4972-445e-833c-da31d8b40bf8\") " pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-8467w656x"
Oct 30 07:40:43.331124 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:43.331079    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-8467w656x"
Oct 30 07:40:43.516743 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:43.516686    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01fea4c7-b874-49e8-a8d5-c0b9b50413fb" path="/var/lib/kubelet/pods/01fea4c7-b874-49e8-a8d5-c0b9b50413fb/volumes"
Oct 30 07:40:43.655880 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:43.655676    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-8467w656x"]
Oct 30 07:40:43.662610 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:40:43.662579    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e58f1f7_4972_445e_833c_da31d8b40bf8.slice/crio-e8e7c8a54aac2a5cc4cdc673b64ee6ce66510da7180e7395235217e4d88e875b WatchSource:0}: Error finding container e8e7c8a54aac2a5cc4cdc673b64ee6ce66510da7180e7395235217e4d88e875b: Status 404 returned error can't find the container with id e8e7c8a54aac2a5cc4cdc673b64ee6ce66510da7180e7395235217e4d88e875b
Oct 30 07:40:43.792876 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:43.787024    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-8467w656x" event={"ID":"0e58f1f7-4972-445e-833c-da31d8b40bf8","Type":"ContainerStarted","Data":"e8e7c8a54aac2a5cc4cdc673b64ee6ce66510da7180e7395235217e4d88e875b"}
Oct 30 07:40:44.807581 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:44.807540    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-8467w656x" event={"ID":"0e58f1f7-4972-445e-833c-da31d8b40bf8","Type":"ContainerStarted","Data":"58faedd6648a956695a4ac2bb96ed903bde853f4648dd033344ea2431ff1862a"}
Oct 30 07:40:44.808093 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:44.807646    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-8467w656x"
Oct 30 07:40:44.857223 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:44.857159    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-8467w656x" podStartSLOduration=2.857142964 podStartE2EDuration="2.857142964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:40:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:40:44.854729318 +0000 UTC m=+2603.464136298" watchObservedRunningTime="2026-10-30 07:40:44.857142964 +0000 UTC m=+2603.466549934"
Oct 30 07:40:50.745692 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:50.745661    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-79fbc7tg8"
Oct 30 07:40:53.791002 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:53.790971    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5b5975486-kzcd4"
Oct 30 07:40:55.817714 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:55.817683    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-8467w656x"
Oct 30 07:40:59.118828 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:59.107605    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-8467w656x"]
Oct 30 07:40:59.792728 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:59.792697    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ad434e-54db8b9d-v5xv5"]
Oct 30 07:40:59.798341 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:59.798324    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ad434e-54db8b9d-v5xv5"
Oct 30 07:40:59.818755 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:59.818729    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ad434e-54db8b9d-v5xv5"]
Oct 30 07:40:59.875855 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:59.866973    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l46qp\" (UniqueName: \"kubernetes.io/projected/ccc61d72-ac4d-4784-9ad1-cea24f30182b-kube-api-access-l46qp\") pod \"payload-processing-e2e-inf-a-ad434e-54db8b9d-v5xv5\" (UID: \"ccc61d72-ac4d-4784-9ad1-cea24f30182b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ad434e-54db8b9d-v5xv5"
Oct 30 07:40:59.875855 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:59.867208    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccc61d72-ac4d-4784-9ad1-cea24f30182b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ad434e-54db8b9d-v5xv5\" (UID: \"ccc61d72-ac4d-4784-9ad1-cea24f30182b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ad434e-54db8b9d-v5xv5"
Oct 30 07:40:59.878628 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:59.878601    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-ad434e-dockercfg-9m5tl\""
Oct 30 07:40:59.878942 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:59.878924    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-ad434e\""
Oct 30 07:40:59.982903 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:59.982866    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l46qp\" (UniqueName: \"kubernetes.io/projected/ccc61d72-ac4d-4784-9ad1-cea24f30182b-kube-api-access-l46qp\") pod \"payload-processing-e2e-inf-a-ad434e-54db8b9d-v5xv5\" (UID: \"ccc61d72-ac4d-4784-9ad1-cea24f30182b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ad434e-54db8b9d-v5xv5"
Oct 30 07:40:59.984980 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:59.984945    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccc61d72-ac4d-4784-9ad1-cea24f30182b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ad434e-54db8b9d-v5xv5\" (UID: \"ccc61d72-ac4d-4784-9ad1-cea24f30182b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ad434e-54db8b9d-v5xv5"
Oct 30 07:40:59.986268 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:40:59.986244    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccc61d72-ac4d-4784-9ad1-cea24f30182b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ad434e-54db8b9d-v5xv5\" (UID: \"ccc61d72-ac4d-4784-9ad1-cea24f30182b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ad434e-54db8b9d-v5xv5"
Oct 30 07:41:00.022233 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:00.022202    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l46qp\" (UniqueName: \"kubernetes.io/projected/ccc61d72-ac4d-4784-9ad1-cea24f30182b-kube-api-access-l46qp\") pod \"payload-processing-e2e-inf-a-ad434e-54db8b9d-v5xv5\" (UID: \"ccc61d72-ac4d-4784-9ad1-cea24f30182b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ad434e-54db8b9d-v5xv5"
Oct 30 07:41:00.028945 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:00.028896    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-79fbc7tg8"]
Oct 30 07:41:00.118460 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:00.118373    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ad434e-54db8b9d-v5xv5"
Oct 30 07:41:00.440466 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:41:00.440433    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccc61d72_ac4d_4784_9ad1_cea24f30182b.slice/crio-9fda6ab0ad9c25936c437fec8c9659f5153f2b7354bbf35da1e5d6743abaa664 WatchSource:0}: Error finding container 9fda6ab0ad9c25936c437fec8c9659f5153f2b7354bbf35da1e5d6743abaa664: Status 404 returned error can't find the container with id 9fda6ab0ad9c25936c437fec8c9659f5153f2b7354bbf35da1e5d6743abaa664
Oct 30 07:41:00.442840 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:00.442798    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ad434e-54db8b9d-v5xv5"]
Oct 30 07:41:01.017383 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:01.017342    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ad434e-54db8b9d-v5xv5" event={"ID":"ccc61d72-ac4d-4784-9ad1-cea24f30182b","Type":"ContainerStarted","Data":"a3cd90918074d4fc48f0f7a1edd596f36e73433982f2ae7d1e2156ec3fe38976"}
Oct 30 07:41:01.017567 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:01.017393    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ad434e-54db8b9d-v5xv5" event={"ID":"ccc61d72-ac4d-4784-9ad1-cea24f30182b","Type":"ContainerStarted","Data":"9fda6ab0ad9c25936c437fec8c9659f5153f2b7354bbf35da1e5d6743abaa664"}
Oct 30 07:41:01.017567 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:01.017508    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ad434e-54db8b9d-v5xv5"
Oct 30 07:41:01.062633 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:01.062576    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-ad434e-54db8b9d-v5xv5" podStartSLOduration=2.06255866 podStartE2EDuration="2.06255866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:40:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:41:01.061526042 +0000 UTC m=+2619.670933013" watchObservedRunningTime="2026-10-30 07:41:01.06255866 +0000 UTC m=+2619.671965632"
Oct 30 07:41:01.860976 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:41:01.860938    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:41:04.108150 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:04.108039    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-8467w656x" podUID="0e58f1f7-4972-445e-833c-da31d8b40bf8" containerName="payload-processing" containerID="cri-o://58faedd6648a956695a4ac2bb96ed903bde853f4648dd033344ea2431ff1862a" gracePeriod=25
Oct 30 07:41:04.378688 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:04.378627    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-8467w656x"
Oct 30 07:41:04.444580 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:04.444510    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e58f1f7-4972-445e-833c-da31d8b40bf8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0e58f1f7-4972-445e-833c-da31d8b40bf8-provider-credentials\") pod \"0e58f1f7-4972-445e-833c-da31d8b40bf8\" (UID: \"0e58f1f7-4972-445e-833c-da31d8b40bf8\") "
Oct 30 07:41:04.444863 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:04.444829    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e58f1f7-4972-445e-833c-da31d8b40bf8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e58f1f7-4972-445e-833c-da31d8b40bf8-plugins-config-volume\") pod \"0e58f1f7-4972-445e-833c-da31d8b40bf8\" (UID: \"0e58f1f7-4972-445e-833c-da31d8b40bf8\") "
Oct 30 07:41:04.444988 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:04.444891    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e58f1f7-4972-445e-833c-da31d8b40bf8-kube-api-access-5khcs\" (UniqueName: \"kubernetes.io/projected/0e58f1f7-4972-445e-833c-da31d8b40bf8-kube-api-access-5khcs\") pod \"0e58f1f7-4972-445e-833c-da31d8b40bf8\" (UID: \"0e58f1f7-4972-445e-833c-da31d8b40bf8\") "
Oct 30 07:41:04.444988 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:04.444917    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e58f1f7-4972-445e-833c-da31d8b40bf8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0e58f1f7-4972-445e-833c-da31d8b40bf8-routing-overlay-volume\") pod \"0e58f1f7-4972-445e-833c-da31d8b40bf8\" (UID: \"0e58f1f7-4972-445e-833c-da31d8b40bf8\") "
Oct 30 07:41:04.445817 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:04.445777    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e58f1f7-4972-445e-833c-da31d8b40bf8-routing-overlay-volume" pod "0e58f1f7-4972-445e-833c-da31d8b40bf8" (UID: "0e58f1f7-4972-445e-833c-da31d8b40bf8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:41:04.446974 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:04.446939    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e58f1f7-4972-445e-833c-da31d8b40bf8-plugins-config-volume" pod "0e58f1f7-4972-445e-833c-da31d8b40bf8" (UID: "0e58f1f7-4972-445e-833c-da31d8b40bf8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:41:04.448290 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:04.448267    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e58f1f7-4972-445e-833c-da31d8b40bf8-provider-credentials" pod "0e58f1f7-4972-445e-833c-da31d8b40bf8" (UID: "0e58f1f7-4972-445e-833c-da31d8b40bf8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:41:04.449719 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:04.449050    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e58f1f7-4972-445e-833c-da31d8b40bf8-kube-api-access-5khcs" pod "0e58f1f7-4972-445e-833c-da31d8b40bf8" (UID: "0e58f1f7-4972-445e-833c-da31d8b40bf8"). InnerVolumeSpecName "kube-api-access-5khcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:41:04.548751 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:04.548719    6224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e58f1f7-4972-445e-833c-da31d8b40bf8-plugins-config-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:41:04.548751 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:04.548756    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5khcs\" (UniqueName: \"kubernetes.io/projected/0e58f1f7-4972-445e-833c-da31d8b40bf8-kube-api-access-5khcs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:41:04.549077 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:04.548774    6224 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0e58f1f7-4972-445e-833c-da31d8b40bf8-routing-overlay-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:41:04.549077 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:04.548787    6224 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0e58f1f7-4972-445e-833c-da31d8b40bf8-provider-credentials\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:41:05.030569 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:05.030506    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-79fbc7tg8" podUID="e93f3828-749e-4c9c-95e5-e826ec289fb6" containerName="payload-processing" containerID="cri-o://0ba88591e215276434c7c8f9f5152000b7917687130c38e08fbbc020a21b346a" gracePeriod=25
Oct 30 07:41:05.047104 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:05.047077    6224 generic.go:356] "Generic (PLEG): container finished" podID="0e58f1f7-4972-445e-833c-da31d8b40bf8" containerID="58faedd6648a956695a4ac2bb96ed903bde853f4648dd033344ea2431ff1862a" exitCode=0
Oct 30 07:41:05.047233 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:05.047130    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-8467w656x"
Oct 30 07:41:05.047233 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:05.047150    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-8467w656x" event={"ID":"0e58f1f7-4972-445e-833c-da31d8b40bf8","Type":"ContainerDied","Data":"58faedd6648a956695a4ac2bb96ed903bde853f4648dd033344ea2431ff1862a"}
Oct 30 07:41:05.047233 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:05.047200    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-8467w656x" event={"ID":"0e58f1f7-4972-445e-833c-da31d8b40bf8","Type":"ContainerDied","Data":"e8e7c8a54aac2a5cc4cdc673b64ee6ce66510da7180e7395235217e4d88e875b"}
Oct 30 07:41:05.047233 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:05.047223    6224 scope.go:122] "RemoveContainer" containerID="58faedd6648a956695a4ac2bb96ed903bde853f4648dd033344ea2431ff1862a"
Oct 30 07:41:05.093314 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:05.093289    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5b5975486-kzcd4"]
Oct 30 07:41:05.142614 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:05.142585    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-8467w656x"]
Oct 30 07:41:05.145165 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:05.144953    6224 scope.go:122] "RemoveContainer" containerID="58faedd6648a956695a4ac2bb96ed903bde853f4648dd033344ea2431ff1862a"
Oct 30 07:41:05.145318 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:41:05.145276    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58faedd6648a956695a4ac2bb96ed903bde853f4648dd033344ea2431ff1862a\": container with ID starting with 58faedd6648a956695a4ac2bb96ed903bde853f4648dd033344ea2431ff1862a not found: ID does not exist" containerID="58faedd6648a956695a4ac2bb96ed903bde853f4648dd033344ea2431ff1862a"
Oct 30 07:41:05.145396 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:05.145310    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58faedd6648a956695a4ac2bb96ed903bde853f4648dd033344ea2431ff1862a"} err="failed to get container status \"58faedd6648a956695a4ac2bb96ed903bde853f4648dd033344ea2431ff1862a\": rpc error: code = NotFound desc = could not find container \"58faedd6648a956695a4ac2bb96ed903bde853f4648dd033344ea2431ff1862a\": container with ID starting with 58faedd6648a956695a4ac2bb96ed903bde853f4648dd033344ea2431ff1862a not found: ID does not exist"
Oct 30 07:41:05.147619 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:05.147599    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-c1b5d7/payload-processing-external-model-e2e-extmod-b-c1b5d7-8467w656x"]
Oct 30 07:41:05.307380 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:05.307358    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-79fbc7tg8"
Oct 30 07:41:05.358877 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:05.358845    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e93f3828-749e-4c9c-95e5-e826ec289fb6-kube-api-access-bn2mb\" (UniqueName: \"kubernetes.io/projected/e93f3828-749e-4c9c-95e5-e826ec289fb6-kube-api-access-bn2mb\") pod \"e93f3828-749e-4c9c-95e5-e826ec289fb6\" (UID: \"e93f3828-749e-4c9c-95e5-e826ec289fb6\") "
Oct 30 07:41:05.359049 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:05.359026    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e93f3828-749e-4c9c-95e5-e826ec289fb6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e93f3828-749e-4c9c-95e5-e826ec289fb6-routing-overlay-volume\") pod \"e93f3828-749e-4c9c-95e5-e826ec289fb6\" (UID: \"e93f3828-749e-4c9c-95e5-e826ec289fb6\") "
Oct 30 07:41:05.359233 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:05.359209    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e93f3828-749e-4c9c-95e5-e826ec289fb6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e93f3828-749e-4c9c-95e5-e826ec289fb6-provider-credentials\") pod \"e93f3828-749e-4c9c-95e5-e826ec289fb6\" (UID: \"e93f3828-749e-4c9c-95e5-e826ec289fb6\") "
Oct 30 07:41:05.359387 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:05.359349    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e93f3828-749e-4c9c-95e5-e826ec289fb6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e93f3828-749e-4c9c-95e5-e826ec289fb6-plugins-config-volume\") pod \"e93f3828-749e-4c9c-95e5-e826ec289fb6\" (UID: \"e93f3828-749e-4c9c-95e5-e826ec289fb6\") "
Oct 30 07:41:05.359560 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:05.359513    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e93f3828-749e-4c9c-95e5-e826ec289fb6-routing-overlay-volume" pod "e93f3828-749e-4c9c-95e5-e826ec289fb6" (UID: "e93f3828-749e-4c9c-95e5-e826ec289fb6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:41:05.360065 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:05.360039    6224 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e93f3828-749e-4c9c-95e5-e826ec289fb6-routing-overlay-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:41:05.360277 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:05.360248    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e93f3828-749e-4c9c-95e5-e826ec289fb6-plugins-config-volume" pod "e93f3828-749e-4c9c-95e5-e826ec289fb6" (UID: "e93f3828-749e-4c9c-95e5-e826ec289fb6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:41:05.365146 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:05.365113    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e93f3828-749e-4c9c-95e5-e826ec289fb6-provider-credentials" pod "e93f3828-749e-4c9c-95e5-e826ec289fb6" (UID: "e93f3828-749e-4c9c-95e5-e826ec289fb6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:41:05.366146 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:05.366118    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e93f3828-749e-4c9c-95e5-e826ec289fb6-kube-api-access-bn2mb" pod "e93f3828-749e-4c9c-95e5-e826ec289fb6" (UID: "e93f3828-749e-4c9c-95e5-e826ec289fb6"). InnerVolumeSpecName "kube-api-access-bn2mb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:41:05.461098 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:05.461071    6224 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e93f3828-749e-4c9c-95e5-e826ec289fb6-provider-credentials\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:41:05.461098 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:05.461096    6224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e93f3828-749e-4c9c-95e5-e826ec289fb6-plugins-config-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:41:05.461273 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:05.461108    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn2mb\" (UniqueName: \"kubernetes.io/projected/e93f3828-749e-4c9c-95e5-e826ec289fb6-kube-api-access-bn2mb\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:41:05.504508 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:05.504481    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e58f1f7-4972-445e-833c-da31d8b40bf8" path="/var/lib/kubelet/pods/0e58f1f7-4972-445e-833c-da31d8b40bf8/volumes"
Oct 30 07:41:06.055899 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:06.055867    6224 generic.go:356] "Generic (PLEG): container finished" podID="e93f3828-749e-4c9c-95e5-e826ec289fb6" containerID="0ba88591e215276434c7c8f9f5152000b7917687130c38e08fbbc020a21b346a" exitCode=0
Oct 30 07:41:06.056093 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:06.055910    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-79fbc7tg8"
Oct 30 07:41:06.056093 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:06.055959    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-79fbc7tg8" event={"ID":"e93f3828-749e-4c9c-95e5-e826ec289fb6","Type":"ContainerDied","Data":"0ba88591e215276434c7c8f9f5152000b7917687130c38e08fbbc020a21b346a"}
Oct 30 07:41:06.056093 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:06.056011    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-79fbc7tg8" event={"ID":"e93f3828-749e-4c9c-95e5-e826ec289fb6","Type":"ContainerDied","Data":"620bedb279cec7c8ee777eb66bb6445b5714eef9c128f2cd534a7ed08d3adff9"}
Oct 30 07:41:06.056093 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:06.056041    6224 scope.go:122] "RemoveContainer" containerID="0ba88591e215276434c7c8f9f5152000b7917687130c38e08fbbc020a21b346a"
Oct 30 07:41:06.066129 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:06.066109    6224 scope.go:122] "RemoveContainer" containerID="0ba88591e215276434c7c8f9f5152000b7917687130c38e08fbbc020a21b346a"
Oct 30 07:41:06.066409 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:41:06.066386    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ba88591e215276434c7c8f9f5152000b7917687130c38e08fbbc020a21b346a\": container with ID starting with 0ba88591e215276434c7c8f9f5152000b7917687130c38e08fbbc020a21b346a not found: ID does not exist" containerID="0ba88591e215276434c7c8f9f5152000b7917687130c38e08fbbc020a21b346a"
Oct 30 07:41:06.066517 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:06.066414    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ba88591e215276434c7c8f9f5152000b7917687130c38e08fbbc020a21b346a"} err="failed to get container status \"0ba88591e215276434c7c8f9f5152000b7917687130c38e08fbbc020a21b346a\": rpc error: code = NotFound desc = could not find container \"0ba88591e215276434c7c8f9f5152000b7917687130c38e08fbbc020a21b346a\": container with ID starting with 0ba88591e215276434c7c8f9f5152000b7917687130c38e08fbbc020a21b346a not found: ID does not exist"
Oct 30 07:41:06.116593 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:06.116566    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-79fbc7tg8"]
Oct 30 07:41:06.130236 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:06.130209    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-79fbc7tg8"]
Oct 30 07:41:07.373142 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:07.373098    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj"]
Oct 30 07:41:07.373970 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:07.373949    6224 cpu_manager.go:414] "containerMap: removing container" podUID="0e58f1f7-4972-445e-833c-da31d8b40bf8" containerName="payload-processing"
Oct 30 07:41:07.373970 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:07.373970    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e58f1f7-4972-445e-833c-da31d8b40bf8" containerName="payload-processing"
Oct 30 07:41:07.374177 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:07.374016    6224 cpu_manager.go:414] "containerMap: removing container" podUID="e93f3828-749e-4c9c-95e5-e826ec289fb6" containerName="payload-processing"
Oct 30 07:41:07.374177 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:07.374027    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e93f3828-749e-4c9c-95e5-e826ec289fb6" containerName="payload-processing"
Oct 30 07:41:07.374286 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:07.374209    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e58f1f7-4972-445e-833c-da31d8b40bf8" containerName="payload-processing"
Oct 30 07:41:07.374286 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:07.374225    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="e93f3828-749e-4c9c-95e5-e826ec289fb6" containerName="payload-processing"
Oct 30 07:41:07.379258 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:07.379235    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj"
Oct 30 07:41:07.395934 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:07.395912    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-0f5081-openshift-default-dockercfg-ntqwn\""
Oct 30 07:41:07.415593 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:07.415552    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj"]
Oct 30 07:41:07.485485 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:07.485449    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cf36810-0393-4f8d-97f5-3a191f94020a-istiod-ca-cert\") pod \"e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj\" (UID: \"6cf36810-0393-4f8d-97f5-3a191f94020a\") " pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj"
Oct 30 07:41:07.485653 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:07.485513    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cf36810-0393-4f8d-97f5-3a191f94020a-istio-data\") pod \"e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj\" (UID: \"6cf36810-0393-4f8d-97f5-3a191f94020a\") " pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj"
Oct 30 07:41:07.485653 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:07.485609    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp5dw\" (UniqueName: \"kubernetes.io/projected/6cf36810-0393-4f8d-97f5-3a191f94020a-kube-api-access-cp5dw\") pod \"e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj\" (UID: \"6cf36810-0393-4f8d-97f5-3a191f94020a\") " pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj"
Oct 30 07:41:07.485750 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:07.485653    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cf36810-0393-4f8d-97f5-3a191f94020a-workload-certs\") pod \"e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj\" (UID: \"6cf36810-0393-4f8d-97f5-3a191f94020a\") " pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj"
Oct 30 07:41:07.485828 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:07.485782    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cf36810-0393-4f8d-97f5-3a191f94020a-workload-socket\") pod \"e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj\" (UID: \"6cf36810-0393-4f8d-97f5-3a191f94020a\") " pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj"
Oct 30 07:41:07.485888 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:07.485874    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cf36810-0393-4f8d-97f5-3a191f94020a-istio-envoy\") pod \"e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj\" (UID: \"6cf36810-0393-4f8d-97f5-3a191f94020a\") " pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj"
Oct 30 07:41:07.486049 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:07.486029    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cf36810-0393-4f8d-97f5-3a191f94020a-istio-token\") pod \"e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj\" (UID: \"6cf36810-0393-4f8d-97f5-3a191f94020a\") " pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj"
Oct 30 07:41:07.486118 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:07.486085    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cf36810-0393-4f8d-97f5-3a191f94020a-credential-socket\") pod \"e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj\" (UID: \"6cf36810-0393-4f8d-97f5-3a191f94020a\") " pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj"
Oct 30 07:41:07.486168 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:07.486148    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cf36810-0393-4f8d-97f5-3a191f94020a-istio-podinfo\") pod \"e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj\" (UID: \"6cf36810-0393-4f8d-97f5-3a191f94020a\") " pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj"
Oct 30 07:41:07.507401 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:07.507363    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e93f3828-749e-4c9c-95e5-e826ec289fb6" path="/var/lib/kubelet/pods/e93f3828-749e-4c9c-95e5-e826ec289fb6/volumes"
Oct 30 07:41:07.587904 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:07.587870    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cf36810-0393-4f8d-97f5-3a191f94020a-istio-token\") pod \"e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj\" (UID: \"6cf36810-0393-4f8d-97f5-3a191f94020a\") " pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj"
Oct 30 07:41:07.588094 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:07.587932    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cf36810-0393-4f8d-97f5-3a191f94020a-credential-socket\") pod \"e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj\" (UID: \"6cf36810-0393-4f8d-97f5-3a191f94020a\") " pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj"
Oct 30 07:41:07.588094 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:07.587968    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cf36810-0393-4f8d-97f5-3a191f94020a-istio-podinfo\") pod \"e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj\" (UID: \"6cf36810-0393-4f8d-97f5-3a191f94020a\") " pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj"
Oct 30 07:41:07.588094 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:07.588032    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cf36810-0393-4f8d-97f5-3a191f94020a-istiod-ca-cert\") pod \"e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj\" (UID: \"6cf36810-0393-4f8d-97f5-3a191f94020a\") " pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj"
Oct 30 07:41:07.588094 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:07.588077    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cf36810-0393-4f8d-97f5-3a191f94020a-istio-data\") pod \"e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj\" (UID: \"6cf36810-0393-4f8d-97f5-3a191f94020a\") " pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj"
Oct 30 07:41:07.588322 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:07.588275    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp5dw\" (UniqueName: \"kubernetes.io/projected/6cf36810-0393-4f8d-97f5-3a191f94020a-kube-api-access-cp5dw\") pod \"e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj\" (UID: \"6cf36810-0393-4f8d-97f5-3a191f94020a\") " pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj"
Oct 30 07:41:07.588386 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:07.588323    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cf36810-0393-4f8d-97f5-3a191f94020a-credential-socket\") pod \"e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj\" (UID: \"6cf36810-0393-4f8d-97f5-3a191f94020a\") " pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj"
Oct 30 07:41:07.588386 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:07.588340    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cf36810-0393-4f8d-97f5-3a191f94020a-workload-certs\") pod \"e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj\" (UID: \"6cf36810-0393-4f8d-97f5-3a191f94020a\") " pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj"
Oct 30 07:41:07.588495 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:07.588447    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cf36810-0393-4f8d-97f5-3a191f94020a-workload-socket\") pod \"e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj\" (UID: \"6cf36810-0393-4f8d-97f5-3a191f94020a\") " pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj"
Oct 30 07:41:07.588495 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:07.588463    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cf36810-0393-4f8d-97f5-3a191f94020a-istio-data\") pod \"e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj\" (UID: \"6cf36810-0393-4f8d-97f5-3a191f94020a\") " pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj"
Oct 30 07:41:07.588692 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:07.588667    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cf36810-0393-4f8d-97f5-3a191f94020a-workload-certs\") pod \"e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj\" (UID: \"6cf36810-0393-4f8d-97f5-3a191f94020a\") " pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj"
Oct 30 07:41:07.588943 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:07.588907    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cf36810-0393-4f8d-97f5-3a191f94020a-istiod-ca-cert\") pod \"e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj\" (UID: \"6cf36810-0393-4f8d-97f5-3a191f94020a\") " pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj"
Oct 30 07:41:07.589136 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:07.589109    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cf36810-0393-4f8d-97f5-3a191f94020a-istio-envoy\") pod \"e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj\" (UID: \"6cf36810-0393-4f8d-97f5-3a191f94020a\") " pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj"
Oct 30 07:41:07.589211 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:07.589182    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cf36810-0393-4f8d-97f5-3a191f94020a-workload-socket\") pod \"e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj\" (UID: \"6cf36810-0393-4f8d-97f5-3a191f94020a\") " pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj"
Oct 30 07:41:07.590632 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:07.590607    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cf36810-0393-4f8d-97f5-3a191f94020a-istio-podinfo\") pod \"e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj\" (UID: \"6cf36810-0393-4f8d-97f5-3a191f94020a\") " pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj"
Oct 30 07:41:07.591038 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:07.591021    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cf36810-0393-4f8d-97f5-3a191f94020a-istio-envoy\") pod \"e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj\" (UID: \"6cf36810-0393-4f8d-97f5-3a191f94020a\") " pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj"
Oct 30 07:41:07.613352 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:07.613327    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cf36810-0393-4f8d-97f5-3a191f94020a-istio-token\") pod \"e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj\" (UID: \"6cf36810-0393-4f8d-97f5-3a191f94020a\") " pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj"
Oct 30 07:41:07.613471 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:07.613430    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp5dw\" (UniqueName: \"kubernetes.io/projected/6cf36810-0393-4f8d-97f5-3a191f94020a-kube-api-access-cp5dw\") pod \"e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj\" (UID: \"6cf36810-0393-4f8d-97f5-3a191f94020a\") " pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj"
Oct 30 07:41:07.693245 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:07.693156    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj"
Oct 30 07:41:08.027908 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:41:08.027878    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cf36810_0393_4f8d_97f5_3a191f94020a.slice/crio-b14a51a6f072cd00e1db38da7ad4e9819f76b61d23fe6d19d6f701e16f9de860 WatchSource:0}: Error finding container b14a51a6f072cd00e1db38da7ad4e9819f76b61d23fe6d19d6f701e16f9de860: Status 404 returned error can't find the container with id b14a51a6f072cd00e1db38da7ad4e9819f76b61d23fe6d19d6f701e16f9de860
Oct 30 07:41:08.029514 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:08.029490    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj"]
Oct 30 07:41:08.031214 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:08.031178    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:41:08.031322 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:08.031259    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:41:08.031322 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:08.031306    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:41:08.073405 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:08.073371    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj" event={"ID":"6cf36810-0393-4f8d-97f5-3a191f94020a","Type":"ContainerStarted","Data":"b14a51a6f072cd00e1db38da7ad4e9819f76b61d23fe6d19d6f701e16f9de860"}
Oct 30 07:41:09.096113 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:09.096074    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj" event={"ID":"6cf36810-0393-4f8d-97f5-3a191f94020a","Type":"ContainerStarted","Data":"7b31518820437b54397e8530d818e34cceadcb7d7908119f1224a87440dd30c0"}
Oct 30 07:41:09.162713 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:09.162655    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj" podStartSLOduration=2.162641267 podStartE2EDuration="2.162641267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:41:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:41:09.158868871 +0000 UTC m=+2627.768275843" watchObservedRunningTime="2026-10-30 07:41:09.162641267 +0000 UTC m=+2627.772048237"
Oct 30 07:41:09.694277 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:09.694193    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj"
Oct 30 07:41:09.695729 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:09.695699    6224 patch_prober.go:29] interesting pod/e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Oct 30 07:41:09.695903 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:09.695752    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj" podUID="6cf36810-0393-4f8d-97f5-3a191f94020a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Oct 30 07:41:10.093641 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:10.093586    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5b5975486-kzcd4" podUID="c2fe939f-a5dd-4e9e-801d-ea58e22cadf8" containerName="payload-processing" containerID="cri-o://66d4fdd7ce58fdfa6bed19140b5eefa9d900e55fca00456c19536d2a408dcb23" gracePeriod=25
Oct 30 07:41:10.406675 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:10.406647    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5b5975486-kzcd4"
Oct 30 07:41:10.530452 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:10.530418    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2fe939f-a5dd-4e9e-801d-ea58e22cadf8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2fe939f-a5dd-4e9e-801d-ea58e22cadf8-provider-credentials\") pod \"c2fe939f-a5dd-4e9e-801d-ea58e22cadf8\" (UID: \"c2fe939f-a5dd-4e9e-801d-ea58e22cadf8\") "
Oct 30 07:41:10.530613 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:10.530503    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2fe939f-a5dd-4e9e-801d-ea58e22cadf8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2fe939f-a5dd-4e9e-801d-ea58e22cadf8-plugins-config-volume\") pod \"c2fe939f-a5dd-4e9e-801d-ea58e22cadf8\" (UID: \"c2fe939f-a5dd-4e9e-801d-ea58e22cadf8\") "
Oct 30 07:41:10.530651 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:10.530623    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2fe939f-a5dd-4e9e-801d-ea58e22cadf8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2fe939f-a5dd-4e9e-801d-ea58e22cadf8-routing-overlay-volume\") pod \"c2fe939f-a5dd-4e9e-801d-ea58e22cadf8\" (UID: \"c2fe939f-a5dd-4e9e-801d-ea58e22cadf8\") "
Oct 30 07:41:10.530704 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:10.530676    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2fe939f-a5dd-4e9e-801d-ea58e22cadf8-kube-api-access-g7jvw\" (UniqueName: \"kubernetes.io/projected/c2fe939f-a5dd-4e9e-801d-ea58e22cadf8-kube-api-access-g7jvw\") pod \"c2fe939f-a5dd-4e9e-801d-ea58e22cadf8\" (UID: \"c2fe939f-a5dd-4e9e-801d-ea58e22cadf8\") "
Oct 30 07:41:10.530995 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:10.530966    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2fe939f-a5dd-4e9e-801d-ea58e22cadf8-routing-overlay-volume" pod "c2fe939f-a5dd-4e9e-801d-ea58e22cadf8" (UID: "c2fe939f-a5dd-4e9e-801d-ea58e22cadf8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:41:10.530995 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:10.530979    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2fe939f-a5dd-4e9e-801d-ea58e22cadf8-plugins-config-volume" pod "c2fe939f-a5dd-4e9e-801d-ea58e22cadf8" (UID: "c2fe939f-a5dd-4e9e-801d-ea58e22cadf8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:41:10.531372 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:10.531352    6224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2fe939f-a5dd-4e9e-801d-ea58e22cadf8-plugins-config-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:41:10.531437 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:10.531376    6224 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2fe939f-a5dd-4e9e-801d-ea58e22cadf8-routing-overlay-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:41:10.532672 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:10.532648    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2fe939f-a5dd-4e9e-801d-ea58e22cadf8-provider-credentials" pod "c2fe939f-a5dd-4e9e-801d-ea58e22cadf8" (UID: "c2fe939f-a5dd-4e9e-801d-ea58e22cadf8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:41:10.532760 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:10.532680    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2fe939f-a5dd-4e9e-801d-ea58e22cadf8-kube-api-access-g7jvw" pod "c2fe939f-a5dd-4e9e-801d-ea58e22cadf8" (UID: "c2fe939f-a5dd-4e9e-801d-ea58e22cadf8"). InnerVolumeSpecName "kube-api-access-g7jvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:41:10.633194 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:10.633060    6224 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2fe939f-a5dd-4e9e-801d-ea58e22cadf8-provider-credentials\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:41:10.633194 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:10.633097    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7jvw\" (UniqueName: \"kubernetes.io/projected/c2fe939f-a5dd-4e9e-801d-ea58e22cadf8-kube-api-access-g7jvw\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:41:10.695078 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:10.694199    6224 patch_prober.go:29] interesting pod/e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Oct 30 07:41:10.695078 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:10.694260    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj" podUID="6cf36810-0393-4f8d-97f5-3a191f94020a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Oct 30 07:41:11.113859 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:11.113789    6224 generic.go:356] "Generic (PLEG): container finished" podID="c2fe939f-a5dd-4e9e-801d-ea58e22cadf8" containerID="66d4fdd7ce58fdfa6bed19140b5eefa9d900e55fca00456c19536d2a408dcb23" exitCode=0
Oct 30 07:41:11.113859 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:11.113839    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5b5975486-kzcd4" event={"ID":"c2fe939f-a5dd-4e9e-801d-ea58e22cadf8","Type":"ContainerDied","Data":"66d4fdd7ce58fdfa6bed19140b5eefa9d900e55fca00456c19536d2a408dcb23"}
Oct 30 07:41:11.113859 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:11.113861    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5b5975486-kzcd4"
Oct 30 07:41:11.114180 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:11.113903    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5b5975486-kzcd4" event={"ID":"c2fe939f-a5dd-4e9e-801d-ea58e22cadf8","Type":"ContainerDied","Data":"3ec49d2694fa079ec00132902461b980bab960f54b4c27a76b3caff307096e8b"}
Oct 30 07:41:11.114180 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:11.113932    6224 scope.go:122] "RemoveContainer" containerID="66d4fdd7ce58fdfa6bed19140b5eefa9d900e55fca00456c19536d2a408dcb23"
Oct 30 07:41:11.125787 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:11.125769    6224 scope.go:122] "RemoveContainer" containerID="66d4fdd7ce58fdfa6bed19140b5eefa9d900e55fca00456c19536d2a408dcb23"
Oct 30 07:41:11.126076 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:41:11.126060    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66d4fdd7ce58fdfa6bed19140b5eefa9d900e55fca00456c19536d2a408dcb23\": container with ID starting with 66d4fdd7ce58fdfa6bed19140b5eefa9d900e55fca00456c19536d2a408dcb23 not found: ID does not exist" containerID="66d4fdd7ce58fdfa6bed19140b5eefa9d900e55fca00456c19536d2a408dcb23"
Oct 30 07:41:11.126126 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:11.126084    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66d4fdd7ce58fdfa6bed19140b5eefa9d900e55fca00456c19536d2a408dcb23"} err="failed to get container status \"66d4fdd7ce58fdfa6bed19140b5eefa9d900e55fca00456c19536d2a408dcb23\": rpc error: code = NotFound desc = could not find container \"66d4fdd7ce58fdfa6bed19140b5eefa9d900e55fca00456c19536d2a408dcb23\": container with ID starting with 66d4fdd7ce58fdfa6bed19140b5eefa9d900e55fca00456c19536d2a408dcb23 not found: ID does not exist"
Oct 30 07:41:11.191452 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:11.190787    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5b5975486-kzcd4"]
Oct 30 07:41:11.196913 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:11.196884    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5b5975486-kzcd4"]
Oct 30 07:41:11.513994 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:11.513915    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2fe939f-a5dd-4e9e-801d-ea58e22cadf8" path="/var/lib/kubelet/pods/c2fe939f-a5dd-4e9e-801d-ea58e22cadf8/volumes"
Oct 30 07:41:11.694393 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:11.694361    6224 patch_prober.go:29] interesting pod/e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Oct 30 07:41:11.694567 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:11.694413    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj" podUID="6cf36810-0393-4f8d-97f5-3a191f94020a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Oct 30 07:41:12.028103 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:12.028071    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ad434e-54db8b9d-v5xv5"
Oct 30 07:41:12.699831 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:12.699675    6224 patch_prober.go:29] interesting pod/e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Oct 30 07:41:12.699831 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:12.699737    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj" podUID="6cf36810-0393-4f8d-97f5-3a191f94020a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Oct 30 07:41:13.694483 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:13.694446    6224 patch_prober.go:29] interesting pod/e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Oct 30 07:41:13.694681 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:13.694502    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj" podUID="6cf36810-0393-4f8d-97f5-3a191f94020a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Oct 30 07:41:14.699492 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:14.699459    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj"
Oct 30 07:41:14.699992 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:14.699894    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj"
Oct 30 07:41:14.700666 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:14.700649    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj"
Oct 30 07:41:31.858275 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:41:31.858238    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:41:34.817731 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:34.817699    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:41:48.563860 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:48.563826    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ad434e-6b5b5484cf-g5m97"]
Oct 30 07:41:48.564543 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:48.564527    6224 cpu_manager.go:414] "containerMap: removing container" podUID="c2fe939f-a5dd-4e9e-801d-ea58e22cadf8" containerName="payload-processing"
Oct 30 07:41:48.564610 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:48.564544    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2fe939f-a5dd-4e9e-801d-ea58e22cadf8" containerName="payload-processing"
Oct 30 07:41:48.564676 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:48.564662    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2fe939f-a5dd-4e9e-801d-ea58e22cadf8" containerName="payload-processing"
Oct 30 07:41:48.570003 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:48.569986    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad434e-6b5b5484cf-g5m97"
Oct 30 07:41:48.586718 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:48.586693    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-ad434e-serving-cert\""
Oct 30 07:41:48.605442 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:48.605409    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ad434e-6b5b5484cf-g5m97"]
Oct 30 07:41:48.686014 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:48.685981    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2db85c9-1904-4eb2-928d-3b8588b8082c-metrics-certs\") pod \"maas-api-e2e-inf-a-ad434e-6b5b5484cf-g5m97\" (UID: \"b2db85c9-1904-4eb2-928d-3b8588b8082c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad434e-6b5b5484cf-g5m97"
Oct 30 07:41:48.686198 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:48.686086    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2db85c9-1904-4eb2-928d-3b8588b8082c-maas-api-tls\") pod \"maas-api-e2e-inf-a-ad434e-6b5b5484cf-g5m97\" (UID: \"b2db85c9-1904-4eb2-928d-3b8588b8082c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad434e-6b5b5484cf-g5m97"
Oct 30 07:41:48.686253 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:48.686213    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvv5j\" (UniqueName: \"kubernetes.io/projected/b2db85c9-1904-4eb2-928d-3b8588b8082c-kube-api-access-bvv5j\") pod \"maas-api-e2e-inf-a-ad434e-6b5b5484cf-g5m97\" (UID: \"b2db85c9-1904-4eb2-928d-3b8588b8082c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad434e-6b5b5484cf-g5m97"
Oct 30 07:41:48.686294 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:48.686252    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2db85c9-1904-4eb2-928d-3b8588b8082c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ad434e-6b5b5484cf-g5m97\" (UID: \"b2db85c9-1904-4eb2-928d-3b8588b8082c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad434e-6b5b5484cf-g5m97"
Oct 30 07:41:48.787221 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:48.787180    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvv5j\" (UniqueName: \"kubernetes.io/projected/b2db85c9-1904-4eb2-928d-3b8588b8082c-kube-api-access-bvv5j\") pod \"maas-api-e2e-inf-a-ad434e-6b5b5484cf-g5m97\" (UID: \"b2db85c9-1904-4eb2-928d-3b8588b8082c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad434e-6b5b5484cf-g5m97"
Oct 30 07:41:48.787221 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:48.787223    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2db85c9-1904-4eb2-928d-3b8588b8082c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ad434e-6b5b5484cf-g5m97\" (UID: \"b2db85c9-1904-4eb2-928d-3b8588b8082c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad434e-6b5b5484cf-g5m97"
Oct 30 07:41:48.787509 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:48.787348    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2db85c9-1904-4eb2-928d-3b8588b8082c-metrics-certs\") pod \"maas-api-e2e-inf-a-ad434e-6b5b5484cf-g5m97\" (UID: \"b2db85c9-1904-4eb2-928d-3b8588b8082c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad434e-6b5b5484cf-g5m97"
Oct 30 07:41:48.787509 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:48.787422    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2db85c9-1904-4eb2-928d-3b8588b8082c-maas-api-tls\") pod \"maas-api-e2e-inf-a-ad434e-6b5b5484cf-g5m97\" (UID: \"b2db85c9-1904-4eb2-928d-3b8588b8082c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad434e-6b5b5484cf-g5m97"
Oct 30 07:41:48.788008 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:48.787979    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2db85c9-1904-4eb2-928d-3b8588b8082c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ad434e-6b5b5484cf-g5m97\" (UID: \"b2db85c9-1904-4eb2-928d-3b8588b8082c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad434e-6b5b5484cf-g5m97"
Oct 30 07:41:48.789863 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:48.789841    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2db85c9-1904-4eb2-928d-3b8588b8082c-metrics-certs\") pod \"maas-api-e2e-inf-a-ad434e-6b5b5484cf-g5m97\" (UID: \"b2db85c9-1904-4eb2-928d-3b8588b8082c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad434e-6b5b5484cf-g5m97"
Oct 30 07:41:48.789996 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:48.789977    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2db85c9-1904-4eb2-928d-3b8588b8082c-maas-api-tls\") pod \"maas-api-e2e-inf-a-ad434e-6b5b5484cf-g5m97\" (UID: \"b2db85c9-1904-4eb2-928d-3b8588b8082c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad434e-6b5b5484cf-g5m97"
Oct 30 07:41:48.812956 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:48.812923    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvv5j\" (UniqueName: \"kubernetes.io/projected/b2db85c9-1904-4eb2-928d-3b8588b8082c-kube-api-access-bvv5j\") pod \"maas-api-e2e-inf-a-ad434e-6b5b5484cf-g5m97\" (UID: \"b2db85c9-1904-4eb2-928d-3b8588b8082c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad434e-6b5b5484cf-g5m97"
Oct 30 07:41:48.885247 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:48.884791    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad434e-6b5b5484cf-g5m97"
Oct 30 07:41:49.197758 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:41:49.197729    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2db85c9_1904_4eb2_928d_3b8588b8082c.slice/crio-851d7b055bf2743c3ae359bf7fca785b04de7d3298fffcb086b4ace38574abe7 WatchSource:0}: Error finding container 851d7b055bf2743c3ae359bf7fca785b04de7d3298fffcb086b4ace38574abe7: Status 404 returned error can't find the container with id 851d7b055bf2743c3ae359bf7fca785b04de7d3298fffcb086b4ace38574abe7
Oct 30 07:41:49.198340 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:49.198317    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ad434e-6b5b5484cf-g5m97"]
Oct 30 07:41:49.199152 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:49.199133    6224 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 07:41:49.436822 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:49.436773    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad434e-6b5b5484cf-g5m97" event={"ID":"b2db85c9-1904-4eb2-928d-3b8588b8082c","Type":"ContainerStarted","Data":"851d7b055bf2743c3ae359bf7fca785b04de7d3298fffcb086b4ace38574abe7"}
Oct 30 07:41:50.445443 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:50.445411    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad434e-6b5b5484cf-g5m97" event={"ID":"b2db85c9-1904-4eb2-928d-3b8588b8082c","Type":"ContainerStarted","Data":"50413d74795df7feab1b2ff2f20e5af60687b5b313819425339a803caa47db32"}
Oct 30 07:41:50.445882 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:50.445601    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad434e-6b5b5484cf-g5m97"
Oct 30 07:41:50.493544 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:50.493499    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad434e-6b5b5484cf-g5m97" podStartSLOduration=2.127632665 podStartE2EDuration="2.493485442s" totalImagesPullingTime="365.852777ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:41:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:41:50.492694632 +0000 UTC m=+2669.102101603" watchObservedRunningTime="2026-10-30 07:41:50.493485442 +0000 UTC m=+2669.102892409"
Oct 30 07:41:53.863145 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:53.863114    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-extmod-shr-user-27d4cd/e2e-usage-sim-32bc7f-7cf9bb66c-mxc4b"]
Oct 30 07:41:53.867654 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:53.867632    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-27d4cd/e2e-usage-sim-32bc7f-7cf9bb66c-mxc4b"
Oct 30 07:41:53.886634 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:53.886608    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-27d4cd/e2e-usage-sim-32bc7f-7cf9bb66c-mxc4b"]
Oct 30 07:41:53.896614 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:53.896592    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-27d4cd\"/\"openshift-service-ca.crt\""
Oct 30 07:41:53.896730 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:53.896654    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-27d4cd\"/\"kube-root-ca.crt\""
Oct 30 07:41:53.911125 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:53.911100    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zvdb\" (UniqueName: \"kubernetes.io/projected/5e4adbdd-072a-4cad-9a46-4bd3c51f04f8-kube-api-access-4zvdb\") pod \"e2e-usage-sim-32bc7f-7cf9bb66c-mxc4b\" (UID: \"5e4adbdd-072a-4cad-9a46-4bd3c51f04f8\") " pod="e2e-extmod-shr-user-27d4cd/e2e-usage-sim-32bc7f-7cf9bb66c-mxc4b"
Oct 30 07:41:53.911294 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:53.911276    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/5e4adbdd-072a-4cad-9a46-4bd3c51f04f8-tls\") pod \"e2e-usage-sim-32bc7f-7cf9bb66c-mxc4b\" (UID: \"5e4adbdd-072a-4cad-9a46-4bd3c51f04f8\") " pod="e2e-extmod-shr-user-27d4cd/e2e-usage-sim-32bc7f-7cf9bb66c-mxc4b"
Oct 30 07:41:53.956834 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:53.956785    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-27d4cd\"/\"default-dockercfg-dmb49\""
Oct 30 07:41:53.956949 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:53.956876    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-27d4cd\"/\"e2e-usage-sim-32bc7f-cert\""
Oct 30 07:41:54.012561 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:54.012535    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/5e4adbdd-072a-4cad-9a46-4bd3c51f04f8-tls\") pod \"e2e-usage-sim-32bc7f-7cf9bb66c-mxc4b\" (UID: \"5e4adbdd-072a-4cad-9a46-4bd3c51f04f8\") " pod="e2e-extmod-shr-user-27d4cd/e2e-usage-sim-32bc7f-7cf9bb66c-mxc4b"
Oct 30 07:41:54.012668 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:54.012654    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zvdb\" (UniqueName: \"kubernetes.io/projected/5e4adbdd-072a-4cad-9a46-4bd3c51f04f8-kube-api-access-4zvdb\") pod \"e2e-usage-sim-32bc7f-7cf9bb66c-mxc4b\" (UID: \"5e4adbdd-072a-4cad-9a46-4bd3c51f04f8\") " pod="e2e-extmod-shr-user-27d4cd/e2e-usage-sim-32bc7f-7cf9bb66c-mxc4b"
Oct 30 07:41:54.015031 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:54.015010    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/5e4adbdd-072a-4cad-9a46-4bd3c51f04f8-tls\") pod \"e2e-usage-sim-32bc7f-7cf9bb66c-mxc4b\" (UID: \"5e4adbdd-072a-4cad-9a46-4bd3c51f04f8\") " pod="e2e-extmod-shr-user-27d4cd/e2e-usage-sim-32bc7f-7cf9bb66c-mxc4b"
Oct 30 07:41:54.038689 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:54.038669    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zvdb\" (UniqueName: \"kubernetes.io/projected/5e4adbdd-072a-4cad-9a46-4bd3c51f04f8-kube-api-access-4zvdb\") pod \"e2e-usage-sim-32bc7f-7cf9bb66c-mxc4b\" (UID: \"5e4adbdd-072a-4cad-9a46-4bd3c51f04f8\") " pod="e2e-extmod-shr-user-27d4cd/e2e-usage-sim-32bc7f-7cf9bb66c-mxc4b"
Oct 30 07:41:54.181648 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:54.181582    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-27d4cd/e2e-usage-sim-32bc7f-7cf9bb66c-mxc4b"
Oct 30 07:41:54.483890 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:41:54.483861    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5e4adbdd_072a_4cad_9a46_4bd3c51f04f8.slice/crio-5f47d08234da9ec1694a2ad2bf55486b6578fd2be98dcb16443b007648092f18 WatchSource:0}: Error finding container 5f47d08234da9ec1694a2ad2bf55486b6578fd2be98dcb16443b007648092f18: Status 404 returned error can't find the container with id 5f47d08234da9ec1694a2ad2bf55486b6578fd2be98dcb16443b007648092f18
Oct 30 07:41:54.486243 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:54.486217    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-27d4cd/e2e-usage-sim-32bc7f-7cf9bb66c-mxc4b"]
Oct 30 07:41:55.482653 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:55.482602    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-27d4cd/e2e-usage-sim-32bc7f-7cf9bb66c-mxc4b" event={"ID":"5e4adbdd-072a-4cad-9a46-4bd3c51f04f8","Type":"ContainerStarted","Data":"b52e36d7232ccbf95c88453a8a2ec812645e65898940c4d535b4cff1773f6576"}
Oct 30 07:41:55.483145 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:55.482662    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-extmod-shr-user-27d4cd/e2e-usage-sim-32bc7f-7cf9bb66c-mxc4b"
Oct 30 07:41:55.483145 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:55.482675    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-27d4cd/e2e-usage-sim-32bc7f-7cf9bb66c-mxc4b" event={"ID":"5e4adbdd-072a-4cad-9a46-4bd3c51f04f8","Type":"ContainerStarted","Data":"5f47d08234da9ec1694a2ad2bf55486b6578fd2be98dcb16443b007648092f18"}
Oct 30 07:41:55.487713 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:55.487690    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-extmod-shr-user-27d4cd/e2e-usage-sim-32bc7f-7cf9bb66c-mxc4b"
Oct 30 07:41:55.528082 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:55.528038    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-extmod-shr-user-27d4cd/e2e-usage-sim-32bc7f-7cf9bb66c-mxc4b" podStartSLOduration=2.528024005 podStartE2EDuration="2.528024005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:41:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:41:55.526216367 +0000 UTC m=+2674.135623348" watchObservedRunningTime="2026-10-30 07:41:55.528024005 +0000 UTC m=+2674.137430975"
Oct 30 07:41:56.457324 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:41:56.457294    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad434e-6b5b5484cf-g5m97"
Oct 30 07:42:01.859534 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:42:01.859496    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:42:20.055729 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:42:20.055684    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:42:31.861020 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:42:31.860983    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:42:49.289544 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:42:49.289506    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8fef73-99b947487-lf4k2"]
Oct 30 07:42:49.295761 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:42:49.295738    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8fef73-99b947487-lf4k2"
Oct 30 07:42:49.315496 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:42:49.315467    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8fef73-99b947487-lf4k2"]
Oct 30 07:42:49.325445 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:42:49.325405    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-8fef73\""
Oct 30 07:42:49.325445 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:42:49.325445    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-8fef73-dockercfg-rppfg\""
Oct 30 07:42:49.434730 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:42:49.434682    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0df77ed5-4c16-40d9-9f1c-6dba3150a7e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-8fef73-99b947487-lf4k2\" (UID: \"0df77ed5-4c16-40d9-9f1c-6dba3150a7e6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8fef73-99b947487-lf4k2"
Oct 30 07:42:49.435252 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:42:49.435221    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slhhj\" (UniqueName: \"kubernetes.io/projected/0df77ed5-4c16-40d9-9f1c-6dba3150a7e6-kube-api-access-slhhj\") pod \"payload-pre-processing-e2e-inf-a-8fef73-99b947487-lf4k2\" (UID: \"0df77ed5-4c16-40d9-9f1c-6dba3150a7e6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8fef73-99b947487-lf4k2"
Oct 30 07:42:49.510749 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:42:49.510701    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8fef73-656cd4fdb-st5tl"]
Oct 30 07:42:49.516657 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:42:49.516607    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8fef73-656cd4fdb-st5tl"]
Oct 30 07:42:49.516917 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:42:49.516736    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8fef73-656cd4fdb-st5tl"
Oct 30 07:42:49.536515 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:42:49.536485    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slhhj\" (UniqueName: \"kubernetes.io/projected/0df77ed5-4c16-40d9-9f1c-6dba3150a7e6-kube-api-access-slhhj\") pod \"payload-pre-processing-e2e-inf-a-8fef73-99b947487-lf4k2\" (UID: \"0df77ed5-4c16-40d9-9f1c-6dba3150a7e6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8fef73-99b947487-lf4k2"
Oct 30 07:42:49.536668 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:42:49.536648    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0df77ed5-4c16-40d9-9f1c-6dba3150a7e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-8fef73-99b947487-lf4k2\" (UID: \"0df77ed5-4c16-40d9-9f1c-6dba3150a7e6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8fef73-99b947487-lf4k2"
Oct 30 07:42:49.537766 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:42:49.537743    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0df77ed5-4c16-40d9-9f1c-6dba3150a7e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-8fef73-99b947487-lf4k2\" (UID: \"0df77ed5-4c16-40d9-9f1c-6dba3150a7e6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8fef73-99b947487-lf4k2"
Oct 30 07:42:49.562405 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:42:49.562350    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slhhj\" (UniqueName: \"kubernetes.io/projected/0df77ed5-4c16-40d9-9f1c-6dba3150a7e6-kube-api-access-slhhj\") pod \"payload-pre-processing-e2e-inf-a-8fef73-99b947487-lf4k2\" (UID: \"0df77ed5-4c16-40d9-9f1c-6dba3150a7e6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8fef73-99b947487-lf4k2"
Oct 30 07:42:49.610510 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:42:49.610482    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8fef73-99b947487-lf4k2"
Oct 30 07:42:49.644849 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:42:49.642290    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1edc6428-e830-42ed-868b-bdb7079e0c33-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8fef73-656cd4fdb-st5tl\" (UID: \"1edc6428-e830-42ed-868b-bdb7079e0c33\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8fef73-656cd4fdb-st5tl"
Oct 30 07:42:49.644849 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:42:49.642426    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btrpr\" (UniqueName: \"kubernetes.io/projected/1edc6428-e830-42ed-868b-bdb7079e0c33-kube-api-access-btrpr\") pod \"payload-processing-e2e-inf-a-8fef73-656cd4fdb-st5tl\" (UID: \"1edc6428-e830-42ed-868b-bdb7079e0c33\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8fef73-656cd4fdb-st5tl"
Oct 30 07:42:49.744185 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:42:49.744149    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btrpr\" (UniqueName: \"kubernetes.io/projected/1edc6428-e830-42ed-868b-bdb7079e0c33-kube-api-access-btrpr\") pod \"payload-processing-e2e-inf-a-8fef73-656cd4fdb-st5tl\" (UID: \"1edc6428-e830-42ed-868b-bdb7079e0c33\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8fef73-656cd4fdb-st5tl"
Oct 30 07:42:49.744372 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:42:49.744337    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1edc6428-e830-42ed-868b-bdb7079e0c33-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8fef73-656cd4fdb-st5tl\" (UID: \"1edc6428-e830-42ed-868b-bdb7079e0c33\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8fef73-656cd4fdb-st5tl"
Oct 30 07:42:49.745061 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:42:49.745038    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1edc6428-e830-42ed-868b-bdb7079e0c33-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8fef73-656cd4fdb-st5tl\" (UID: \"1edc6428-e830-42ed-868b-bdb7079e0c33\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8fef73-656cd4fdb-st5tl"
Oct 30 07:42:49.771422 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:42:49.771396    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btrpr\" (UniqueName: \"kubernetes.io/projected/1edc6428-e830-42ed-868b-bdb7079e0c33-kube-api-access-btrpr\") pod \"payload-processing-e2e-inf-a-8fef73-656cd4fdb-st5tl\" (UID: \"1edc6428-e830-42ed-868b-bdb7079e0c33\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8fef73-656cd4fdb-st5tl"
Oct 30 07:42:49.838944 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:42:49.838873    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8fef73-656cd4fdb-st5tl"
Oct 30 07:42:49.926850 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:42:49.926794    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8fef73-99b947487-lf4k2"]
Oct 30 07:42:49.990636 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:42:49.990603    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8fef73-99b947487-lf4k2" event={"ID":"0df77ed5-4c16-40d9-9f1c-6dba3150a7e6","Type":"ContainerStarted","Data":"93a4b2b264a0092f178af267a606f7ff1c1a270953d07993d12fb355eba125a0"}
Oct 30 07:42:50.141759 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:42:50.141730    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1edc6428_e830_42ed_868b_bdb7079e0c33.slice/crio-d2535108df63eed5787745cf5b20aca3283526b17ad7f460bf69d1191fa7d469 WatchSource:0}: Error finding container d2535108df63eed5787745cf5b20aca3283526b17ad7f460bf69d1191fa7d469: Status 404 returned error can't find the container with id d2535108df63eed5787745cf5b20aca3283526b17ad7f460bf69d1191fa7d469
Oct 30 07:42:50.143162 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:42:50.143136    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8fef73-656cd4fdb-st5tl"]
Oct 30 07:42:50.999650 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:42:50.999606    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8fef73-656cd4fdb-st5tl" event={"ID":"1edc6428-e830-42ed-868b-bdb7079e0c33","Type":"ContainerStarted","Data":"aa9a43ab9c21ba0788735277a5dd16d77dab644b59870016a797d2465391dd03"}
Oct 30 07:42:51.000121 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:42:50.999680    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-8fef73-656cd4fdb-st5tl"
Oct 30 07:42:51.000121 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:42:50.999703    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8fef73-656cd4fdb-st5tl" event={"ID":"1edc6428-e830-42ed-868b-bdb7079e0c33","Type":"ContainerStarted","Data":"d2535108df63eed5787745cf5b20aca3283526b17ad7f460bf69d1191fa7d469"}
Oct 30 07:42:51.001768 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:42:51.001741    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8fef73-99b947487-lf4k2" event={"ID":"0df77ed5-4c16-40d9-9f1c-6dba3150a7e6","Type":"ContainerStarted","Data":"fc3630ab64d4fa5cbb0cd0e4c9d4d2515c46aa50e703a1a6e91dcc798a30a377"}
Oct 30 07:42:51.001909 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:42:51.001855    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8fef73-99b947487-lf4k2"
Oct 30 07:42:51.050285 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:42:51.050233    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-8fef73-656cd4fdb-st5tl" podStartSLOduration=2.050220216 podStartE2EDuration="2.050220216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:42:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:42:51.044983598 +0000 UTC m=+2729.654390571" watchObservedRunningTime="2026-10-30 07:42:51.050220216 +0000 UTC m=+2729.659627186"
Oct 30 07:42:51.091527 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:42:51.091473    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8fef73-99b947487-lf4k2" podStartSLOduration=2.091458515 podStartE2EDuration="2.091458515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:42:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:42:51.090628259 +0000 UTC m=+2729.700035230" watchObservedRunningTime="2026-10-30 07:42:51.091458515 +0000 UTC m=+2729.700865484"
Oct 30 07:42:51.555655 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:42:51.555622    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4ab37e-7cdb99fdd6-mlnbs"]
Oct 30 07:42:51.561068 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:42:51.561024    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4ab37e-7cdb99fdd6-mlnbs"
Oct 30 07:42:51.613890 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:42:51.608822    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-4ab37e\""
Oct 30 07:42:51.638832 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:42:51.630351    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4ab37e-7cdb99fdd6-mlnbs"]
Oct 30 07:42:51.638832 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:42:51.630591    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-4ab37e-dockercfg-kld95\""
Oct 30 07:42:51.677947 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:42:51.677478    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pndnp\" (UniqueName: \"kubernetes.io/projected/8a2822e1-a884-41cb-a5c2-34972b1d4de4-kube-api-access-pndnp\") pod \"payload-processing-e2e-inf-b-4ab37e-7cdb99fdd6-mlnbs\" (UID: \"8a2822e1-a884-41cb-a5c2-34972b1d4de4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4ab37e-7cdb99fdd6-mlnbs"
Oct 30 07:42:51.677947 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:42:51.677530    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a2822e1-a884-41cb-a5c2-34972b1d4de4-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4ab37e-7cdb99fdd6-mlnbs\" (UID: \"8a2822e1-a884-41cb-a5c2-34972b1d4de4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4ab37e-7cdb99fdd6-mlnbs"
Oct 30 07:42:51.779667 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:42:51.779630    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pndnp\" (UniqueName: \"kubernetes.io/projected/8a2822e1-a884-41cb-a5c2-34972b1d4de4-kube-api-access-pndnp\") pod \"payload-processing-e2e-inf-b-4ab37e-7cdb99fdd6-mlnbs\" (UID: \"8a2822e1-a884-41cb-a5c2-34972b1d4de4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4ab37e-7cdb99fdd6-mlnbs"
Oct 30 07:42:51.779667 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:42:51.779669    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a2822e1-a884-41cb-a5c2-34972b1d4de4-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4ab37e-7cdb99fdd6-mlnbs\" (UID: \"8a2822e1-a884-41cb-a5c2-34972b1d4de4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4ab37e-7cdb99fdd6-mlnbs"
Oct 30 07:42:51.780446 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:42:51.780426    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a2822e1-a884-41cb-a5c2-34972b1d4de4-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4ab37e-7cdb99fdd6-mlnbs\" (UID: \"8a2822e1-a884-41cb-a5c2-34972b1d4de4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4ab37e-7cdb99fdd6-mlnbs"
Oct 30 07:42:51.808316 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:42:51.808240    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pndnp\" (UniqueName: \"kubernetes.io/projected/8a2822e1-a884-41cb-a5c2-34972b1d4de4-kube-api-access-pndnp\") pod \"payload-processing-e2e-inf-b-4ab37e-7cdb99fdd6-mlnbs\" (UID: \"8a2822e1-a884-41cb-a5c2-34972b1d4de4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4ab37e-7cdb99fdd6-mlnbs"
Oct 30 07:42:51.944082 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:42:51.944049    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4ab37e-7cdb99fdd6-mlnbs"
Oct 30 07:42:52.245144 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:42:52.245100    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a2822e1_a884_41cb_a5c2_34972b1d4de4.slice/crio-530ff575fbdb1524a8a335a85fffccb3e9843741e40a634c9aefbc2c6afc393b WatchSource:0}: Error finding container 530ff575fbdb1524a8a335a85fffccb3e9843741e40a634c9aefbc2c6afc393b: Status 404 returned error can't find the container with id 530ff575fbdb1524a8a335a85fffccb3e9843741e40a634c9aefbc2c6afc393b
Oct 30 07:42:52.246459 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:42:52.246437    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4ab37e-7cdb99fdd6-mlnbs"]
Oct 30 07:42:53.020877 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:42:53.020839    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4ab37e-7cdb99fdd6-mlnbs" event={"ID":"8a2822e1-a884-41cb-a5c2-34972b1d4de4","Type":"ContainerStarted","Data":"07185a558bf3277002cf38b81c4f9d24dd252c37c718a9d1120049d9a505e56c"}
Oct 30 07:42:53.021077 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:42:53.020907    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4ab37e-7cdb99fdd6-mlnbs" event={"ID":"8a2822e1-a884-41cb-a5c2-34972b1d4de4","Type":"ContainerStarted","Data":"530ff575fbdb1524a8a335a85fffccb3e9843741e40a634c9aefbc2c6afc393b"}
Oct 30 07:42:53.021077 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:42:53.021053    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-4ab37e-7cdb99fdd6-mlnbs"
Oct 30 07:42:53.066493 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:42:53.066442    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-4ab37e-7cdb99fdd6-mlnbs" podStartSLOduration=2.066424843 podStartE2EDuration="2.066424843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:42:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:42:53.06468158 +0000 UTC m=+2731.674088552" watchObservedRunningTime="2026-10-30 07:42:53.066424843 +0000 UTC m=+2731.675831858"
Oct 30 07:43:01.860866 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:43:01.860825    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:43:02.016959 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:43:02.016929    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-8fef73-656cd4fdb-st5tl"
Oct 30 07:43:02.017361 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:43:02.017339    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8fef73-99b947487-lf4k2"
Oct 30 07:43:04.041013 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:43:04.040977    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-4ab37e-7cdb99fdd6-mlnbs"
Oct 30 07:43:31.859818 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:43:31.859764    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:43:34.406705 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:43:34.406668    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:43:41.893243 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:43:41.893212    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4ab37e-65c46fb659-dbr69"]
Oct 30 07:43:41.898781 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:43:41.898763    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4ab37e-65c46fb659-dbr69"
Oct 30 07:43:41.918461 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:43:41.918437    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4ab37e-65c46fb659-dbr69"]
Oct 30 07:43:41.928273 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:43:41.928249    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-4ab37e-serving-cert\""
Oct 30 07:43:42.037305 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:43:42.037272    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgqh2\" (UniqueName: \"kubernetes.io/projected/efa670c6-14aa-4c5e-be1c-9976b974bdff-kube-api-access-kgqh2\") pod \"maas-api-e2e-inf-b-4ab37e-65c46fb659-dbr69\" (UID: \"efa670c6-14aa-4c5e-be1c-9976b974bdff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4ab37e-65c46fb659-dbr69"
Oct 30 07:43:42.037492 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:43:42.037313    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efa670c6-14aa-4c5e-be1c-9976b974bdff-metrics-certs\") pod \"maas-api-e2e-inf-b-4ab37e-65c46fb659-dbr69\" (UID: \"efa670c6-14aa-4c5e-be1c-9976b974bdff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4ab37e-65c46fb659-dbr69"
Oct 30 07:43:42.037492 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:43:42.037336    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efa670c6-14aa-4c5e-be1c-9976b974bdff-maas-api-tls\") pod \"maas-api-e2e-inf-b-4ab37e-65c46fb659-dbr69\" (UID: \"efa670c6-14aa-4c5e-be1c-9976b974bdff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4ab37e-65c46fb659-dbr69"
Oct 30 07:43:42.037582 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:43:42.037542    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efa670c6-14aa-4c5e-be1c-9976b974bdff-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4ab37e-65c46fb659-dbr69\" (UID: \"efa670c6-14aa-4c5e-be1c-9976b974bdff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4ab37e-65c46fb659-dbr69"
Oct 30 07:43:42.139919 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:43:42.139860    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efa670c6-14aa-4c5e-be1c-9976b974bdff-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4ab37e-65c46fb659-dbr69\" (UID: \"efa670c6-14aa-4c5e-be1c-9976b974bdff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4ab37e-65c46fb659-dbr69"
Oct 30 07:43:42.140347 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:43:42.140312    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgqh2\" (UniqueName: \"kubernetes.io/projected/efa670c6-14aa-4c5e-be1c-9976b974bdff-kube-api-access-kgqh2\") pod \"maas-api-e2e-inf-b-4ab37e-65c46fb659-dbr69\" (UID: \"efa670c6-14aa-4c5e-be1c-9976b974bdff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4ab37e-65c46fb659-dbr69"
Oct 30 07:43:42.140511 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:43:42.140358    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efa670c6-14aa-4c5e-be1c-9976b974bdff-metrics-certs\") pod \"maas-api-e2e-inf-b-4ab37e-65c46fb659-dbr69\" (UID: \"efa670c6-14aa-4c5e-be1c-9976b974bdff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4ab37e-65c46fb659-dbr69"
Oct 30 07:43:42.140511 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:43:42.140395    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efa670c6-14aa-4c5e-be1c-9976b974bdff-maas-api-tls\") pod \"maas-api-e2e-inf-b-4ab37e-65c46fb659-dbr69\" (UID: \"efa670c6-14aa-4c5e-be1c-9976b974bdff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4ab37e-65c46fb659-dbr69"
Oct 30 07:43:42.141041 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:43:42.141008    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efa670c6-14aa-4c5e-be1c-9976b974bdff-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4ab37e-65c46fb659-dbr69\" (UID: \"efa670c6-14aa-4c5e-be1c-9976b974bdff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4ab37e-65c46fb659-dbr69"
Oct 30 07:43:42.144460 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:43:42.144366    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efa670c6-14aa-4c5e-be1c-9976b974bdff-metrics-certs\") pod \"maas-api-e2e-inf-b-4ab37e-65c46fb659-dbr69\" (UID: \"efa670c6-14aa-4c5e-be1c-9976b974bdff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4ab37e-65c46fb659-dbr69"
Oct 30 07:43:42.146191 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:43:42.146121    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efa670c6-14aa-4c5e-be1c-9976b974bdff-maas-api-tls\") pod \"maas-api-e2e-inf-b-4ab37e-65c46fb659-dbr69\" (UID: \"efa670c6-14aa-4c5e-be1c-9976b974bdff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4ab37e-65c46fb659-dbr69"
Oct 30 07:43:42.168090 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:43:42.168057    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgqh2\" (UniqueName: \"kubernetes.io/projected/efa670c6-14aa-4c5e-be1c-9976b974bdff-kube-api-access-kgqh2\") pod \"maas-api-e2e-inf-b-4ab37e-65c46fb659-dbr69\" (UID: \"efa670c6-14aa-4c5e-be1c-9976b974bdff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4ab37e-65c46fb659-dbr69"
Oct 30 07:43:42.212743 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:43:42.212710    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4ab37e-65c46fb659-dbr69"
Oct 30 07:43:42.517755 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:43:42.517724    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefa670c6_14aa_4c5e_be1c_9976b974bdff.slice/crio-2626bab648bc167dc1f3861bc8df0989d6c15f4ba7a7c8810d58b2a6d84fe696 WatchSource:0}: Error finding container 2626bab648bc167dc1f3861bc8df0989d6c15f4ba7a7c8810d58b2a6d84fe696: Status 404 returned error can't find the container with id 2626bab648bc167dc1f3861bc8df0989d6c15f4ba7a7c8810d58b2a6d84fe696
Oct 30 07:43:42.517960 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:43:42.517939    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4ab37e-65c46fb659-dbr69"]
Oct 30 07:43:43.424758 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:43:43.424719    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4ab37e-65c46fb659-dbr69" event={"ID":"efa670c6-14aa-4c5e-be1c-9976b974bdff","Type":"ContainerStarted","Data":"fcf444f4238faafad50883aba8ab24c820ed10f9af9648152b12394a2c7d4b08"}
Oct 30 07:43:43.425222 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:43:43.424798    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4ab37e-65c46fb659-dbr69" event={"ID":"efa670c6-14aa-4c5e-be1c-9976b974bdff","Type":"ContainerStarted","Data":"2626bab648bc167dc1f3861bc8df0989d6c15f4ba7a7c8810d58b2a6d84fe696"}
Oct 30 07:43:43.425222 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:43:43.424860    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4ab37e-65c46fb659-dbr69"
Oct 30 07:43:43.472886 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:43:43.472835    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4ab37e-65c46fb659-dbr69" podStartSLOduration=2.19951825 podStartE2EDuration="2.472823199s" totalImagesPullingTime="273.304949ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:43:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:43:43.47070857 +0000 UTC m=+2782.080115539" watchObservedRunningTime="2026-10-30 07:43:43.472823199 +0000 UTC m=+2782.082230162"
Oct 30 07:43:49.437950 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:43:49.437919    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4ab37e-65c46fb659-dbr69"
Oct 30 07:44:01.859896 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:44:01.859853    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:44:18.952734 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:18.952633    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-866cjr7gk"]
Oct 30 07:44:23.953700 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:23.953646    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-866cjr7gk" podUID="23d95410-a777-4c72-a377-3a3495fa0a15" containerName="payload-processing" containerID="cri-o://afab0df3e9b78baf31c86511ea425f7447e6829134f0af12dead17c9363f1ad6" gracePeriod=25
Oct 30 07:44:24.253442 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:24.253413    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-866cjr7gk"
Oct 30 07:44:24.327726 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:24.327688    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23d95410-a777-4c72-a377-3a3495fa0a15-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/23d95410-a777-4c72-a377-3a3495fa0a15-routing-overlay-volume\") pod \"23d95410-a777-4c72-a377-3a3495fa0a15\" (UID: \"23d95410-a777-4c72-a377-3a3495fa0a15\") "
Oct 30 07:44:24.327921 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:24.327752    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23d95410-a777-4c72-a377-3a3495fa0a15-kube-api-access-lkqhn\" (UniqueName: \"kubernetes.io/projected/23d95410-a777-4c72-a377-3a3495fa0a15-kube-api-access-lkqhn\") pod \"23d95410-a777-4c72-a377-3a3495fa0a15\" (UID: \"23d95410-a777-4c72-a377-3a3495fa0a15\") "
Oct 30 07:44:24.327921 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:24.327794    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23d95410-a777-4c72-a377-3a3495fa0a15-provider-credentials\" (UniqueName: \"kubernetes.io/projected/23d95410-a777-4c72-a377-3a3495fa0a15-provider-credentials\") pod \"23d95410-a777-4c72-a377-3a3495fa0a15\" (UID: \"23d95410-a777-4c72-a377-3a3495fa0a15\") "
Oct 30 07:44:24.327921 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:24.327867    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23d95410-a777-4c72-a377-3a3495fa0a15-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23d95410-a777-4c72-a377-3a3495fa0a15-plugins-config-volume\") pod \"23d95410-a777-4c72-a377-3a3495fa0a15\" (UID: \"23d95410-a777-4c72-a377-3a3495fa0a15\") "
Oct 30 07:44:24.328184 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:24.328053    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23d95410-a777-4c72-a377-3a3495fa0a15-routing-overlay-volume" pod "23d95410-a777-4c72-a377-3a3495fa0a15" (UID: "23d95410-a777-4c72-a377-3a3495fa0a15"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:44:24.328335 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:24.328308    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23d95410-a777-4c72-a377-3a3495fa0a15-plugins-config-volume" pod "23d95410-a777-4c72-a377-3a3495fa0a15" (UID: "23d95410-a777-4c72-a377-3a3495fa0a15"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:44:24.328697 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:24.328676    6224 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/23d95410-a777-4c72-a377-3a3495fa0a15-routing-overlay-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:44:24.328797 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:24.328702    6224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23d95410-a777-4c72-a377-3a3495fa0a15-plugins-config-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:44:24.330031 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:24.330004    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23d95410-a777-4c72-a377-3a3495fa0a15-kube-api-access-lkqhn" pod "23d95410-a777-4c72-a377-3a3495fa0a15" (UID: "23d95410-a777-4c72-a377-3a3495fa0a15"). InnerVolumeSpecName "kube-api-access-lkqhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:44:24.330950 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:24.330925    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23d95410-a777-4c72-a377-3a3495fa0a15-provider-credentials" pod "23d95410-a777-4c72-a377-3a3495fa0a15" (UID: "23d95410-a777-4c72-a377-3a3495fa0a15"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:44:24.430624 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:24.430594    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkqhn\" (UniqueName: \"kubernetes.io/projected/23d95410-a777-4c72-a377-3a3495fa0a15-kube-api-access-lkqhn\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:44:24.430624 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:24.430621    6224 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/23d95410-a777-4c72-a377-3a3495fa0a15-provider-credentials\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:44:24.756854 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:24.756794    6224 generic.go:356] "Generic (PLEG): container finished" podID="23d95410-a777-4c72-a377-3a3495fa0a15" containerID="afab0df3e9b78baf31c86511ea425f7447e6829134f0af12dead17c9363f1ad6" exitCode=0
Oct 30 07:44:24.757023 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:24.756855    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-866cjr7gk"
Oct 30 07:44:24.757023 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:24.756855    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-866cjr7gk" event={"ID":"23d95410-a777-4c72-a377-3a3495fa0a15","Type":"ContainerDied","Data":"afab0df3e9b78baf31c86511ea425f7447e6829134f0af12dead17c9363f1ad6"}
Oct 30 07:44:24.757023 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:24.756924    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-866cjr7gk" event={"ID":"23d95410-a777-4c72-a377-3a3495fa0a15","Type":"ContainerDied","Data":"8e5ae3d6df8deb6d110926d04c0eec359f7f5092b6afbdf3ed855abc49d488f4"}
Oct 30 07:44:24.757023 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:24.756956    6224 scope.go:122] "RemoveContainer" containerID="afab0df3e9b78baf31c86511ea425f7447e6829134f0af12dead17c9363f1ad6"
Oct 30 07:44:24.769320 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:24.769296    6224 scope.go:122] "RemoveContainer" containerID="afab0df3e9b78baf31c86511ea425f7447e6829134f0af12dead17c9363f1ad6"
Oct 30 07:44:24.769601 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:44:24.769579    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afab0df3e9b78baf31c86511ea425f7447e6829134f0af12dead17c9363f1ad6\": container with ID starting with afab0df3e9b78baf31c86511ea425f7447e6829134f0af12dead17c9363f1ad6 not found: ID does not exist" containerID="afab0df3e9b78baf31c86511ea425f7447e6829134f0af12dead17c9363f1ad6"
Oct 30 07:44:24.769676 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:24.769610    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afab0df3e9b78baf31c86511ea425f7447e6829134f0af12dead17c9363f1ad6"} err="failed to get container status \"afab0df3e9b78baf31c86511ea425f7447e6829134f0af12dead17c9363f1ad6\": rpc error: code = NotFound desc = could not find container \"afab0df3e9b78baf31c86511ea425f7447e6829134f0af12dead17c9363f1ad6\": container with ID starting with afab0df3e9b78baf31c86511ea425f7447e6829134f0af12dead17c9363f1ad6 not found: ID does not exist"
Oct 30 07:44:24.816218 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:24.816190    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-866cjr7gk"]
Oct 30 07:44:24.829945 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:24.829917    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-e27e0c/payload-processing-external-model-e2e-extmod-b-e27e0c-866cjr7gk"]
Oct 30 07:44:25.504312 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:25.504280    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23d95410-a777-4c72-a377-3a3495fa0a15" path="/var/lib/kubelet/pods/23d95410-a777-4c72-a377-3a3495fa0a15/volumes"
Oct 30 07:44:27.324160 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:27.324127    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-79fbnqckc"]
Oct 30 07:44:27.324820 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:27.324793    6224 cpu_manager.go:414] "containerMap: removing container" podUID="23d95410-a777-4c72-a377-3a3495fa0a15" containerName="payload-processing"
Oct 30 07:44:27.324879 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:27.324823    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23d95410-a777-4c72-a377-3a3495fa0a15" containerName="payload-processing"
Oct 30 07:44:27.325035 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:27.325023    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="23d95410-a777-4c72-a377-3a3495fa0a15" containerName="payload-processing"
Oct 30 07:44:27.329758 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:27.329739    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-79fbnqckc"
Oct 30 07:44:27.349891 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:27.349864    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-79fbnqckc"]
Oct 30 07:44:27.370922 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:27.369819    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9bec5c\"/\"openshift-service-ca.crt\""
Oct 30 07:44:27.370922 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:27.370579    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9bec5c\"/\"routing-overlay\""
Oct 30 07:44:27.370922 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:27.370655    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-9bec5c\"/\"agc-cred-6cd15efdbb3d\""
Oct 30 07:44:27.370922 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:27.370729    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9bec5c\"/\"kube-root-ca.crt\""
Oct 30 07:44:27.370922 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:27.370873    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-9bec5c\"/\"payload-processing-external-model-e2e--314725d2-dockercfg-fpjr8\""
Oct 30 07:44:27.371252 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:27.370958    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9bec5c\"/\"payload-processing-external-model-plugins-e2e-extmod-a-9bec5c\""
Oct 30 07:44:27.477328 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:27.477291    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3ba99f81-9643-4331-9f03-a71987812566-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-79fbnqckc\" (UID: \"3ba99f81-9643-4331-9f03-a71987812566\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-79fbnqckc"
Oct 30 07:44:27.477523 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:27.477438    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ba99f81-9643-4331-9f03-a71987812566-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-79fbnqckc\" (UID: \"3ba99f81-9643-4331-9f03-a71987812566\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-79fbnqckc"
Oct 30 07:44:27.477639 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:27.477622    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3ba99f81-9643-4331-9f03-a71987812566-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-79fbnqckc\" (UID: \"3ba99f81-9643-4331-9f03-a71987812566\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-79fbnqckc"
Oct 30 07:44:27.477691 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:27.477654    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwbh2\" (UniqueName: \"kubernetes.io/projected/3ba99f81-9643-4331-9f03-a71987812566-kube-api-access-mwbh2\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-79fbnqckc\" (UID: \"3ba99f81-9643-4331-9f03-a71987812566\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-79fbnqckc"
Oct 30 07:44:27.578658 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:27.578571    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3ba99f81-9643-4331-9f03-a71987812566-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-79fbnqckc\" (UID: \"3ba99f81-9643-4331-9f03-a71987812566\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-79fbnqckc"
Oct 30 07:44:27.578658 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:27.578620    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwbh2\" (UniqueName: \"kubernetes.io/projected/3ba99f81-9643-4331-9f03-a71987812566-kube-api-access-mwbh2\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-79fbnqckc\" (UID: \"3ba99f81-9643-4331-9f03-a71987812566\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-79fbnqckc"
Oct 30 07:44:27.578972 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:27.578682    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3ba99f81-9643-4331-9f03-a71987812566-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-79fbnqckc\" (UID: \"3ba99f81-9643-4331-9f03-a71987812566\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-79fbnqckc"
Oct 30 07:44:27.578972 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:27.578772    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ba99f81-9643-4331-9f03-a71987812566-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-79fbnqckc\" (UID: \"3ba99f81-9643-4331-9f03-a71987812566\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-79fbnqckc"
Oct 30 07:44:27.579457 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:27.579433    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3ba99f81-9643-4331-9f03-a71987812566-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-79fbnqckc\" (UID: \"3ba99f81-9643-4331-9f03-a71987812566\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-79fbnqckc"
Oct 30 07:44:27.579573 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:27.579535    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ba99f81-9643-4331-9f03-a71987812566-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-79fbnqckc\" (UID: \"3ba99f81-9643-4331-9f03-a71987812566\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-79fbnqckc"
Oct 30 07:44:27.581253 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:27.581227    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3ba99f81-9643-4331-9f03-a71987812566-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-79fbnqckc\" (UID: \"3ba99f81-9643-4331-9f03-a71987812566\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-79fbnqckc"
Oct 30 07:44:27.603848 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:27.603827    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwbh2\" (UniqueName: \"kubernetes.io/projected/3ba99f81-9643-4331-9f03-a71987812566-kube-api-access-mwbh2\") pod \"payload-processing-external-model-e2e-extmod-a-9bec5c-79fbnqckc\" (UID: \"3ba99f81-9643-4331-9f03-a71987812566\") " pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-79fbnqckc"
Oct 30 07:44:27.644606 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:27.644583    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-79fbnqckc"
Oct 30 07:44:27.953327 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:44:27.953297    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ba99f81_9643_4331_9f03_a71987812566.slice/crio-91564fd3a5b6be726c435e51c8cebcd0ecdbed2e286c3053434dfbafaa8e5793 WatchSource:0}: Error finding container 91564fd3a5b6be726c435e51c8cebcd0ecdbed2e286c3053434dfbafaa8e5793: Status 404 returned error can't find the container with id 91564fd3a5b6be726c435e51c8cebcd0ecdbed2e286c3053434dfbafaa8e5793
Oct 30 07:44:27.953967 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:27.953943    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-79fbnqckc"]
Oct 30 07:44:28.919314 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:28.919267    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-79fbnqckc" event={"ID":"3ba99f81-9643-4331-9f03-a71987812566","Type":"ContainerStarted","Data":"87614ee16ce7f1fd04cd0b38cb996dc80ea6547831907f9226d617b175687e37"}
Oct 30 07:44:28.919888 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:28.919343    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-79fbnqckc" event={"ID":"3ba99f81-9643-4331-9f03-a71987812566","Type":"ContainerStarted","Data":"91564fd3a5b6be726c435e51c8cebcd0ecdbed2e286c3053434dfbafaa8e5793"}
Oct 30 07:44:28.921784 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:28.920479    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-79fbnqckc"
Oct 30 07:44:28.980578 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:28.980523    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-79fbnqckc" podStartSLOduration=1.980502801 podStartE2EDuration="1.980502801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:44:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:44:28.977883617 +0000 UTC m=+2827.587290592" watchObservedRunningTime="2026-10-30 07:44:28.980502801 +0000 UTC m=+2827.589910185"
Oct 30 07:44:29.048295 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:44:29.048256    6224 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-9bec5c/agc-cred-6cd15efdbb3d: secret "agc-cred-6cd15efdbb3d" not found
Oct 30 07:44:29.048295 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:44:29.048289    6224 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-79fbnqckc: secret "agc-cred-6cd15efdbb3d" not found
Oct 30 07:44:29.048538 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:44:29.048371    6224 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ba99f81-9643-4331-9f03-a71987812566-provider-credentials podName:3ba99f81-9643-4331-9f03-a71987812566 nodeName:}" failed. No retries permitted until 2026-10-30 07:44:29.548350536 +0000 UTC m=+2828.157757505 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3ba99f81-9643-4331-9f03-a71987812566-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-9bec5c-79fbnqckc" (UID: "3ba99f81-9643-4331-9f03-a71987812566") : secret "agc-cred-6cd15efdbb3d" not found
Oct 30 07:44:29.210668 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:29.210591    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-79fbnqckc"]
Oct 30 07:44:29.559887 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:44:29.559092    6224 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-9bec5c/agc-cred-6cd15efdbb3d: secret "agc-cred-6cd15efdbb3d" not found
Oct 30 07:44:29.559887 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:44:29.559133    6224 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-79fbnqckc: secret "agc-cred-6cd15efdbb3d" not found
Oct 30 07:44:29.559887 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:44:29.559200    6224 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ba99f81-9643-4331-9f03-a71987812566-provider-credentials podName:3ba99f81-9643-4331-9f03-a71987812566 nodeName:}" failed. No retries permitted until 2026-10-30 07:44:30.559178562 +0000 UTC m=+2829.168585513 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3ba99f81-9643-4331-9f03-a71987812566-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-9bec5c-79fbnqckc" (UID: "3ba99f81-9643-4331-9f03-a71987812566") : secret "agc-cred-6cd15efdbb3d" not found
Oct 30 07:44:29.927063 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:29.926963    6224 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-79fbnqckc" secret="" err="secret \"payload-processing-external-model-e2e--314725d2-dockercfg-fpjr8\" not found"
Oct 30 07:44:30.066269 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:44:30.066232    6224 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-plugins-e2e-extmod-a-9bec5c: configmap "payload-processing-external-model-plugins-e2e-extmod-a-9bec5c" not found
Oct 30 07:44:30.066453 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:44:30.066338    6224 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3ba99f81-9643-4331-9f03-a71987812566-plugins-config-volume podName:3ba99f81-9643-4331-9f03-a71987812566 nodeName:}" failed. No retries permitted until 2026-10-30 07:44:30.566315975 +0000 UTC m=+2829.175722936 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/3ba99f81-9643-4331-9f03-a71987812566-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-9bec5c-79fbnqckc" (UID: "3ba99f81-9643-4331-9f03-a71987812566") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-9bec5c" not found
Oct 30 07:44:30.575300 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:44:30.575261    6224 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-9bec5c/agc-cred-6cd15efdbb3d: secret "agc-cred-6cd15efdbb3d" not found
Oct 30 07:44:30.575300 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:44:30.575297    6224 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-79fbnqckc: secret "agc-cred-6cd15efdbb3d" not found
Oct 30 07:44:30.575560 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:44:30.575359    6224 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ba99f81-9643-4331-9f03-a71987812566-provider-credentials podName:3ba99f81-9643-4331-9f03-a71987812566 nodeName:}" failed. No retries permitted until 2026-10-30 07:44:32.575340949 +0000 UTC m=+2831.184747899 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3ba99f81-9643-4331-9f03-a71987812566-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-9bec5c-79fbnqckc" (UID: "3ba99f81-9643-4331-9f03-a71987812566") : secret "agc-cred-6cd15efdbb3d" not found
Oct 30 07:44:30.578841 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:44:30.575735    6224 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-plugins-e2e-extmod-a-9bec5c: configmap "payload-processing-external-model-plugins-e2e-extmod-a-9bec5c" not found
Oct 30 07:44:30.578841 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:44:30.575800    6224 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3ba99f81-9643-4331-9f03-a71987812566-plugins-config-volume podName:3ba99f81-9643-4331-9f03-a71987812566 nodeName:}" failed. No retries permitted until 2026-10-30 07:44:31.575784723 +0000 UTC m=+2830.185191692 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/3ba99f81-9643-4331-9f03-a71987812566-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-9bec5c-79fbnqckc" (UID: "3ba99f81-9643-4331-9f03-a71987812566") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-9bec5c" not found
Oct 30 07:44:31.589914 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:44:31.589881    6224 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-plugins-e2e-extmod-a-9bec5c: configmap "payload-processing-external-model-plugins-e2e-extmod-a-9bec5c" not found
Oct 30 07:44:31.590328 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:44:31.589962    6224 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3ba99f81-9643-4331-9f03-a71987812566-plugins-config-volume podName:3ba99f81-9643-4331-9f03-a71987812566 nodeName:}" failed. No retries permitted until 2026-10-30 07:44:33.589946074 +0000 UTC m=+2832.199353056 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/3ba99f81-9643-4331-9f03-a71987812566-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-9bec5c-79fbnqckc" (UID: "3ba99f81-9643-4331-9f03-a71987812566") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-9bec5c" not found
Oct 30 07:44:31.858430 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:44:31.858346    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:44:32.603877 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:44:32.603847    6224 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-9bec5c/agc-cred-6cd15efdbb3d: secret "agc-cred-6cd15efdbb3d" not found
Oct 30 07:44:32.603877 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:44:32.603875    6224 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-79fbnqckc: secret "agc-cred-6cd15efdbb3d" not found
Oct 30 07:44:32.604315 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:44:32.603938    6224 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ba99f81-9643-4331-9f03-a71987812566-provider-credentials podName:3ba99f81-9643-4331-9f03-a71987812566 nodeName:}" failed. No retries permitted until 2026-10-30 07:44:36.603919872 +0000 UTC m=+2835.213326844 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3ba99f81-9643-4331-9f03-a71987812566-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-9bec5c-79fbnqckc" (UID: "3ba99f81-9643-4331-9f03-a71987812566") : secret "agc-cred-6cd15efdbb3d" not found
Oct 30 07:44:33.617380 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:44:33.617343    6224 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-plugins-e2e-extmod-a-9bec5c: configmap "payload-processing-external-model-plugins-e2e-extmod-a-9bec5c" not found
Oct 30 07:44:33.617879 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:44:33.617424    6224 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3ba99f81-9643-4331-9f03-a71987812566-plugins-config-volume podName:3ba99f81-9643-4331-9f03-a71987812566 nodeName:}" failed. No retries permitted until 2026-10-30 07:44:37.617407138 +0000 UTC m=+2836.226814086 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/3ba99f81-9643-4331-9f03-a71987812566-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-9bec5c-79fbnqckc" (UID: "3ba99f81-9643-4331-9f03-a71987812566") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-9bec5c" not found
Oct 30 07:44:35.934725 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:35.934681    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-79fbnqckc" podUID="3ba99f81-9643-4331-9f03-a71987812566" containerName="payload-processing" containerID="cri-o://87614ee16ce7f1fd04cd0b38cb996dc80ea6547831907f9226d617b175687e37" gracePeriod=25
Oct 30 07:44:36.212631 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:36.212606    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-79fbnqckc"
Oct 30 07:44:36.249046 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:36.249016    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ba99f81-9643-4331-9f03-a71987812566-kube-api-access-mwbh2\" (UniqueName: \"kubernetes.io/projected/3ba99f81-9643-4331-9f03-a71987812566-kube-api-access-mwbh2\") pod \"3ba99f81-9643-4331-9f03-a71987812566\" (UID: \"3ba99f81-9643-4331-9f03-a71987812566\") "
Oct 30 07:44:36.249189 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:36.249108    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ba99f81-9643-4331-9f03-a71987812566-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3ba99f81-9643-4331-9f03-a71987812566-provider-credentials\") pod \"3ba99f81-9643-4331-9f03-a71987812566\" (UID: \"3ba99f81-9643-4331-9f03-a71987812566\") "
Oct 30 07:44:36.249254 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:36.249237    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ba99f81-9643-4331-9f03-a71987812566-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3ba99f81-9643-4331-9f03-a71987812566-routing-overlay-volume\") pod \"3ba99f81-9643-4331-9f03-a71987812566\" (UID: \"3ba99f81-9643-4331-9f03-a71987812566\") "
Oct 30 07:44:36.249326 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:36.249298    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ba99f81-9643-4331-9f03-a71987812566-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ba99f81-9643-4331-9f03-a71987812566-plugins-config-volume\") pod \"3ba99f81-9643-4331-9f03-a71987812566\" (UID: \"3ba99f81-9643-4331-9f03-a71987812566\") "
Oct 30 07:44:36.249652 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:36.249625    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ba99f81-9643-4331-9f03-a71987812566-routing-overlay-volume" pod "3ba99f81-9643-4331-9f03-a71987812566" (UID: "3ba99f81-9643-4331-9f03-a71987812566"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:44:36.249762 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:36.249645    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ba99f81-9643-4331-9f03-a71987812566-plugins-config-volume" pod "3ba99f81-9643-4331-9f03-a71987812566" (UID: "3ba99f81-9643-4331-9f03-a71987812566"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:44:36.249952 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:36.249936    6224 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3ba99f81-9643-4331-9f03-a71987812566-routing-overlay-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:44:36.250002 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:36.249955    6224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ba99f81-9643-4331-9f03-a71987812566-plugins-config-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:44:36.251364 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:36.251345    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ba99f81-9643-4331-9f03-a71987812566-kube-api-access-mwbh2" pod "3ba99f81-9643-4331-9f03-a71987812566" (UID: "3ba99f81-9643-4331-9f03-a71987812566"). InnerVolumeSpecName "kube-api-access-mwbh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:44:36.251440 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:36.251377    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ba99f81-9643-4331-9f03-a71987812566-provider-credentials" pod "3ba99f81-9643-4331-9f03-a71987812566" (UID: "3ba99f81-9643-4331-9f03-a71987812566"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:44:36.350940 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:36.350903    6224 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3ba99f81-9643-4331-9f03-a71987812566-provider-credentials\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:44:36.350940 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:36.350931    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwbh2\" (UniqueName: \"kubernetes.io/projected/3ba99f81-9643-4331-9f03-a71987812566-kube-api-access-mwbh2\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:44:36.981760 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:36.981677    6224 generic.go:356] "Generic (PLEG): container finished" podID="3ba99f81-9643-4331-9f03-a71987812566" containerID="87614ee16ce7f1fd04cd0b38cb996dc80ea6547831907f9226d617b175687e37" exitCode=0
Oct 30 07:44:36.982494 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:36.981743    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-79fbnqckc"
Oct 30 07:44:36.983833 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:36.981763    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-79fbnqckc" event={"ID":"3ba99f81-9643-4331-9f03-a71987812566","Type":"ContainerDied","Data":"87614ee16ce7f1fd04cd0b38cb996dc80ea6547831907f9226d617b175687e37"}
Oct 30 07:44:36.983833 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:36.983679    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-79fbnqckc" event={"ID":"3ba99f81-9643-4331-9f03-a71987812566","Type":"ContainerDied","Data":"91564fd3a5b6be726c435e51c8cebcd0ecdbed2e286c3053434dfbafaa8e5793"}
Oct 30 07:44:36.983833 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:36.983715    6224 scope.go:122] "RemoveContainer" containerID="87614ee16ce7f1fd04cd0b38cb996dc80ea6547831907f9226d617b175687e37"
Oct 30 07:44:37.025535 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:37.025158    6224 scope.go:122] "RemoveContainer" containerID="87614ee16ce7f1fd04cd0b38cb996dc80ea6547831907f9226d617b175687e37"
Oct 30 07:44:37.039120 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:44:37.039004    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87614ee16ce7f1fd04cd0b38cb996dc80ea6547831907f9226d617b175687e37\": container with ID starting with 87614ee16ce7f1fd04cd0b38cb996dc80ea6547831907f9226d617b175687e37 not found: ID does not exist" containerID="87614ee16ce7f1fd04cd0b38cb996dc80ea6547831907f9226d617b175687e37"
Oct 30 07:44:37.039120 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:37.039051    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87614ee16ce7f1fd04cd0b38cb996dc80ea6547831907f9226d617b175687e37"} err="failed to get container status \"87614ee16ce7f1fd04cd0b38cb996dc80ea6547831907f9226d617b175687e37\": rpc error: code = NotFound desc = could not find container \"87614ee16ce7f1fd04cd0b38cb996dc80ea6547831907f9226d617b175687e37\": container with ID starting with 87614ee16ce7f1fd04cd0b38cb996dc80ea6547831907f9226d617b175687e37 not found: ID does not exist"
Oct 30 07:44:37.086828 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:37.084563    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-79fbnqckc"]
Oct 30 07:44:37.121212 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:37.121167    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-9bec5c/payload-processing-external-model-e2e-extmod-a-9bec5c-79fbnqckc"]
Oct 30 07:44:37.508099 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:37.508060    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ba99f81-9643-4331-9f03-a71987812566" path="/var/lib/kubelet/pods/3ba99f81-9643-4331-9f03-a71987812566/volumes"
Oct 30 07:44:42.879890 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:42.879857    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:44:44.546520 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:44.546470    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-91c8c2-5b7b6b7c59-zlzl6"]
Oct 30 07:44:44.548212 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:44.548182    6224 cpu_manager.go:414] "containerMap: removing container" podUID="3ba99f81-9643-4331-9f03-a71987812566" containerName="payload-processing"
Oct 30 07:44:44.548377 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:44.548361    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ba99f81-9643-4331-9f03-a71987812566" containerName="payload-processing"
Oct 30 07:44:44.548701 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:44.548683    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ba99f81-9643-4331-9f03-a71987812566" containerName="payload-processing"
Oct 30 07:44:44.555911 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:44.554617    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-91c8c2-5b7b6b7c59-zlzl6"
Oct 30 07:44:44.575084 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:44.575057    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-91c8c2-5b7b6b7c59-zlzl6"]
Oct 30 07:44:44.582858 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:44.580050    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-91c8c2\""
Oct 30 07:44:44.582858 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:44.580177    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-91c8c2-dockercfg-mczkx\""
Oct 30 07:44:44.686750 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:44.686706    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg2j6\" (UniqueName: \"kubernetes.io/projected/6626da66-f9b9-4838-8f9e-02ece551f087-kube-api-access-kg2j6\") pod \"payload-pre-processing-e2e-inf-a-91c8c2-5b7b6b7c59-zlzl6\" (UID: \"6626da66-f9b9-4838-8f9e-02ece551f087\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-91c8c2-5b7b6b7c59-zlzl6"
Oct 30 07:44:44.686953 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:44.686873    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6626da66-f9b9-4838-8f9e-02ece551f087-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-91c8c2-5b7b6b7c59-zlzl6\" (UID: \"6626da66-f9b9-4838-8f9e-02ece551f087\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-91c8c2-5b7b6b7c59-zlzl6"
Oct 30 07:44:44.714531 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:44.714493    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-91c8c2-67cf6564dd-z4qql"]
Oct 30 07:44:44.720378 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:44.720352    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-91c8c2-67cf6564dd-z4qql"
Oct 30 07:44:44.737744 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:44.737714    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-91c8c2-67cf6564dd-z4qql"]
Oct 30 07:44:44.787799 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:44.787752    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a38806b-e470-45b3-a949-c457876d72e4-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-91c8c2-67cf6564dd-z4qql\" (UID: \"5a38806b-e470-45b3-a949-c457876d72e4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-91c8c2-67cf6564dd-z4qql"
Oct 30 07:44:44.787976 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:44.787931    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tk9d\" (UniqueName: \"kubernetes.io/projected/5a38806b-e470-45b3-a949-c457876d72e4-kube-api-access-5tk9d\") pod \"payload-processing-e2e-inf-a-91c8c2-67cf6564dd-z4qql\" (UID: \"5a38806b-e470-45b3-a949-c457876d72e4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-91c8c2-67cf6564dd-z4qql"
Oct 30 07:44:44.787976 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:44.787959    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg2j6\" (UniqueName: \"kubernetes.io/projected/6626da66-f9b9-4838-8f9e-02ece551f087-kube-api-access-kg2j6\") pod \"payload-pre-processing-e2e-inf-a-91c8c2-5b7b6b7c59-zlzl6\" (UID: \"6626da66-f9b9-4838-8f9e-02ece551f087\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-91c8c2-5b7b6b7c59-zlzl6"
Oct 30 07:44:44.788109 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:44.788084    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6626da66-f9b9-4838-8f9e-02ece551f087-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-91c8c2-5b7b6b7c59-zlzl6\" (UID: \"6626da66-f9b9-4838-8f9e-02ece551f087\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-91c8c2-5b7b6b7c59-zlzl6"
Oct 30 07:44:44.788778 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:44.788755    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6626da66-f9b9-4838-8f9e-02ece551f087-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-91c8c2-5b7b6b7c59-zlzl6\" (UID: \"6626da66-f9b9-4838-8f9e-02ece551f087\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-91c8c2-5b7b6b7c59-zlzl6"
Oct 30 07:44:44.813745 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:44.813669    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg2j6\" (UniqueName: \"kubernetes.io/projected/6626da66-f9b9-4838-8f9e-02ece551f087-kube-api-access-kg2j6\") pod \"payload-pre-processing-e2e-inf-a-91c8c2-5b7b6b7c59-zlzl6\" (UID: \"6626da66-f9b9-4838-8f9e-02ece551f087\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-91c8c2-5b7b6b7c59-zlzl6"
Oct 30 07:44:44.876076 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:44.876042    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-91c8c2-5b7b6b7c59-zlzl6"
Oct 30 07:44:44.890256 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:44.890219    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tk9d\" (UniqueName: \"kubernetes.io/projected/5a38806b-e470-45b3-a949-c457876d72e4-kube-api-access-5tk9d\") pod \"payload-processing-e2e-inf-a-91c8c2-67cf6564dd-z4qql\" (UID: \"5a38806b-e470-45b3-a949-c457876d72e4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-91c8c2-67cf6564dd-z4qql"
Oct 30 07:44:44.890633 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:44.890587    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a38806b-e470-45b3-a949-c457876d72e4-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-91c8c2-67cf6564dd-z4qql\" (UID: \"5a38806b-e470-45b3-a949-c457876d72e4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-91c8c2-67cf6564dd-z4qql"
Oct 30 07:44:44.894741 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:44.891557    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a38806b-e470-45b3-a949-c457876d72e4-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-91c8c2-67cf6564dd-z4qql\" (UID: \"5a38806b-e470-45b3-a949-c457876d72e4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-91c8c2-67cf6564dd-z4qql"
Oct 30 07:44:44.916949 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:44.916918    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tk9d\" (UniqueName: \"kubernetes.io/projected/5a38806b-e470-45b3-a949-c457876d72e4-kube-api-access-5tk9d\") pod \"payload-processing-e2e-inf-a-91c8c2-67cf6564dd-z4qql\" (UID: \"5a38806b-e470-45b3-a949-c457876d72e4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-91c8c2-67cf6564dd-z4qql"
Oct 30 07:44:45.037628 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:45.037596    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-91c8c2-67cf6564dd-z4qql"
Oct 30 07:44:45.189702 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:44:45.189673    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6626da66_f9b9_4838_8f9e_02ece551f087.slice/crio-d54a8a74bd770ee46baf9bc759ef2e99dcf98e1fabcca87c2216cdd0c2ca0507 WatchSource:0}: Error finding container d54a8a74bd770ee46baf9bc759ef2e99dcf98e1fabcca87c2216cdd0c2ca0507: Status 404 returned error can't find the container with id d54a8a74bd770ee46baf9bc759ef2e99dcf98e1fabcca87c2216cdd0c2ca0507
Oct 30 07:44:45.190950 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:45.190927    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-91c8c2-5b7b6b7c59-zlzl6"]
Oct 30 07:44:45.341308 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:44:45.341247    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a38806b_e470_45b3_a949_c457876d72e4.slice/crio-b8a3bd7dd66bbf9e9b20178dd3c8bb9c3e49f79e7240a1905c2b6189221a041a WatchSource:0}: Error finding container b8a3bd7dd66bbf9e9b20178dd3c8bb9c3e49f79e7240a1905c2b6189221a041a: Status 404 returned error can't find the container with id b8a3bd7dd66bbf9e9b20178dd3c8bb9c3e49f79e7240a1905c2b6189221a041a
Oct 30 07:44:45.342475 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:45.342452    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-91c8c2-67cf6564dd-z4qql"]
Oct 30 07:44:46.066916 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:46.066872    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-91c8c2-67cf6564dd-z4qql" event={"ID":"5a38806b-e470-45b3-a949-c457876d72e4","Type":"ContainerStarted","Data":"c66018fdc4671b66bf676cfa94e39c14644065cb4a0bb321aaa2f7bd53bc5336"}
Oct 30 07:44:46.067414 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:46.066962    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-91c8c2-67cf6564dd-z4qql"
Oct 30 07:44:46.067414 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:46.066984    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-91c8c2-67cf6564dd-z4qql" event={"ID":"5a38806b-e470-45b3-a949-c457876d72e4","Type":"ContainerStarted","Data":"b8a3bd7dd66bbf9e9b20178dd3c8bb9c3e49f79e7240a1905c2b6189221a041a"}
Oct 30 07:44:46.068533 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:46.068503    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-91c8c2-5b7b6b7c59-zlzl6" event={"ID":"6626da66-f9b9-4838-8f9e-02ece551f087","Type":"ContainerStarted","Data":"3fbccbd3744945d22b4935d7d83748a23d2ad3f5f846d858d5ede75f7f2ac62e"}
Oct 30 07:44:46.068669 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:46.068544    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-91c8c2-5b7b6b7c59-zlzl6" event={"ID":"6626da66-f9b9-4838-8f9e-02ece551f087","Type":"ContainerStarted","Data":"d54a8a74bd770ee46baf9bc759ef2e99dcf98e1fabcca87c2216cdd0c2ca0507"}
Oct 30 07:44:46.068669 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:46.068562    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-91c8c2-5b7b6b7c59-zlzl6"
Oct 30 07:44:46.137689 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:46.137640    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-91c8c2-67cf6564dd-z4qql" podStartSLOduration=2.137627051 podStartE2EDuration="2.137627051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:44:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:44:46.135619053 +0000 UTC m=+2844.745026023" watchObservedRunningTime="2026-10-30 07:44:46.137627051 +0000 UTC m=+2844.747034050"
Oct 30 07:44:46.200320 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:46.200273    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-91c8c2-5b7b6b7c59-zlzl6" podStartSLOduration=2.200260487 podStartE2EDuration="2.200260487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:44:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:44:46.197085839 +0000 UTC m=+2844.806492812" watchObservedRunningTime="2026-10-30 07:44:46.200260487 +0000 UTC m=+2844.809667453"
Oct 30 07:44:49.280385 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:49.280341    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-ad434e/test-model-ad434e-kserve-9d799b5f7-zcfvt"]
Oct 30 07:44:49.285376 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:49.285351    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ad434e/test-model-ad434e-kserve-9d799b5f7-zcfvt"
Oct 30 07:44:49.306648 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:49.306620    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ad434e/test-model-ad434e-kserve-9d799b5f7-zcfvt"]
Oct 30 07:44:49.314498 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:49.314474    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ad434e\"/\"kube-root-ca.crt\""
Oct 30 07:44:49.314625 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:49.314571    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ad434e\"/\"openshift-service-ca.crt\""
Oct 30 07:44:49.314625 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:49.314595    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ad434e\"/\"default-dockercfg-hhsvl\""
Oct 30 07:44:49.317382 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:49.317362    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ad434e\"/\"test-model-ad434e-kserve-self-signed-certs\""
Oct 30 07:44:49.369909 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:49.369872    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fd17eaf-b363-4ddb-8dc3-29489cb00a87-dshm\") pod \"test-model-ad434e-kserve-9d799b5f7-zcfvt\" (UID: \"6fd17eaf-b363-4ddb-8dc3-29489cb00a87\") " pod="ai-tenant-e2e-inf-a-ad434e/test-model-ad434e-kserve-9d799b5f7-zcfvt"
Oct 30 07:44:49.370056 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:49.369927    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48wpx\" (UniqueName: \"kubernetes.io/projected/6fd17eaf-b363-4ddb-8dc3-29489cb00a87-kube-api-access-48wpx\") pod \"test-model-ad434e-kserve-9d799b5f7-zcfvt\" (UID: \"6fd17eaf-b363-4ddb-8dc3-29489cb00a87\") " pod="ai-tenant-e2e-inf-a-ad434e/test-model-ad434e-kserve-9d799b5f7-zcfvt"
Oct 30 07:44:49.370056 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:49.369994    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fd17eaf-b363-4ddb-8dc3-29489cb00a87-home\") pod \"test-model-ad434e-kserve-9d799b5f7-zcfvt\" (UID: \"6fd17eaf-b363-4ddb-8dc3-29489cb00a87\") " pod="ai-tenant-e2e-inf-a-ad434e/test-model-ad434e-kserve-9d799b5f7-zcfvt"
Oct 30 07:44:49.370056 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:49.370045    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fd17eaf-b363-4ddb-8dc3-29489cb00a87-tls-certs\") pod \"test-model-ad434e-kserve-9d799b5f7-zcfvt\" (UID: \"6fd17eaf-b363-4ddb-8dc3-29489cb00a87\") " pod="ai-tenant-e2e-inf-a-ad434e/test-model-ad434e-kserve-9d799b5f7-zcfvt"
Oct 30 07:44:49.370231 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:49.370118    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fd17eaf-b363-4ddb-8dc3-29489cb00a87-model-cache\") pod \"test-model-ad434e-kserve-9d799b5f7-zcfvt\" (UID: \"6fd17eaf-b363-4ddb-8dc3-29489cb00a87\") " pod="ai-tenant-e2e-inf-a-ad434e/test-model-ad434e-kserve-9d799b5f7-zcfvt"
Oct 30 07:44:49.370231 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:49.370169    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fd17eaf-b363-4ddb-8dc3-29489cb00a87-tmp-dir\") pod \"test-model-ad434e-kserve-9d799b5f7-zcfvt\" (UID: \"6fd17eaf-b363-4ddb-8dc3-29489cb00a87\") " pod="ai-tenant-e2e-inf-a-ad434e/test-model-ad434e-kserve-9d799b5f7-zcfvt"
Oct 30 07:44:49.471579 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:49.471540    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fd17eaf-b363-4ddb-8dc3-29489cb00a87-dshm\") pod \"test-model-ad434e-kserve-9d799b5f7-zcfvt\" (UID: \"6fd17eaf-b363-4ddb-8dc3-29489cb00a87\") " pod="ai-tenant-e2e-inf-a-ad434e/test-model-ad434e-kserve-9d799b5f7-zcfvt"
Oct 30 07:44:49.471779 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:49.471590    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48wpx\" (UniqueName: \"kubernetes.io/projected/6fd17eaf-b363-4ddb-8dc3-29489cb00a87-kube-api-access-48wpx\") pod \"test-model-ad434e-kserve-9d799b5f7-zcfvt\" (UID: \"6fd17eaf-b363-4ddb-8dc3-29489cb00a87\") " pod="ai-tenant-e2e-inf-a-ad434e/test-model-ad434e-kserve-9d799b5f7-zcfvt"
Oct 30 07:44:49.471779 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:49.471644    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fd17eaf-b363-4ddb-8dc3-29489cb00a87-home\") pod \"test-model-ad434e-kserve-9d799b5f7-zcfvt\" (UID: \"6fd17eaf-b363-4ddb-8dc3-29489cb00a87\") " pod="ai-tenant-e2e-inf-a-ad434e/test-model-ad434e-kserve-9d799b5f7-zcfvt"
Oct 30 07:44:49.471779 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:49.471690    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fd17eaf-b363-4ddb-8dc3-29489cb00a87-tls-certs\") pod \"test-model-ad434e-kserve-9d799b5f7-zcfvt\" (UID: \"6fd17eaf-b363-4ddb-8dc3-29489cb00a87\") " pod="ai-tenant-e2e-inf-a-ad434e/test-model-ad434e-kserve-9d799b5f7-zcfvt"
Oct 30 07:44:49.471990 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:49.471786    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fd17eaf-b363-4ddb-8dc3-29489cb00a87-model-cache\") pod \"test-model-ad434e-kserve-9d799b5f7-zcfvt\" (UID: \"6fd17eaf-b363-4ddb-8dc3-29489cb00a87\") " pod="ai-tenant-e2e-inf-a-ad434e/test-model-ad434e-kserve-9d799b5f7-zcfvt"
Oct 30 07:44:49.471990 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:49.471889    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fd17eaf-b363-4ddb-8dc3-29489cb00a87-tmp-dir\") pod \"test-model-ad434e-kserve-9d799b5f7-zcfvt\" (UID: \"6fd17eaf-b363-4ddb-8dc3-29489cb00a87\") " pod="ai-tenant-e2e-inf-a-ad434e/test-model-ad434e-kserve-9d799b5f7-zcfvt"
Oct 30 07:44:49.472133 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:49.472108    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fd17eaf-b363-4ddb-8dc3-29489cb00a87-home\") pod \"test-model-ad434e-kserve-9d799b5f7-zcfvt\" (UID: \"6fd17eaf-b363-4ddb-8dc3-29489cb00a87\") " pod="ai-tenant-e2e-inf-a-ad434e/test-model-ad434e-kserve-9d799b5f7-zcfvt"
Oct 30 07:44:49.472202 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:49.472137    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fd17eaf-b363-4ddb-8dc3-29489cb00a87-model-cache\") pod \"test-model-ad434e-kserve-9d799b5f7-zcfvt\" (UID: \"6fd17eaf-b363-4ddb-8dc3-29489cb00a87\") " pod="ai-tenant-e2e-inf-a-ad434e/test-model-ad434e-kserve-9d799b5f7-zcfvt"
Oct 30 07:44:49.472268 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:49.472246    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fd17eaf-b363-4ddb-8dc3-29489cb00a87-tmp-dir\") pod \"test-model-ad434e-kserve-9d799b5f7-zcfvt\" (UID: \"6fd17eaf-b363-4ddb-8dc3-29489cb00a87\") " pod="ai-tenant-e2e-inf-a-ad434e/test-model-ad434e-kserve-9d799b5f7-zcfvt"
Oct 30 07:44:49.474520 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:49.474494    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fd17eaf-b363-4ddb-8dc3-29489cb00a87-dshm\") pod \"test-model-ad434e-kserve-9d799b5f7-zcfvt\" (UID: \"6fd17eaf-b363-4ddb-8dc3-29489cb00a87\") " pod="ai-tenant-e2e-inf-a-ad434e/test-model-ad434e-kserve-9d799b5f7-zcfvt"
Oct 30 07:44:49.474759 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:49.474741    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fd17eaf-b363-4ddb-8dc3-29489cb00a87-tls-certs\") pod \"test-model-ad434e-kserve-9d799b5f7-zcfvt\" (UID: \"6fd17eaf-b363-4ddb-8dc3-29489cb00a87\") " pod="ai-tenant-e2e-inf-a-ad434e/test-model-ad434e-kserve-9d799b5f7-zcfvt"
Oct 30 07:44:49.498078 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:49.498040    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48wpx\" (UniqueName: \"kubernetes.io/projected/6fd17eaf-b363-4ddb-8dc3-29489cb00a87-kube-api-access-48wpx\") pod \"test-model-ad434e-kserve-9d799b5f7-zcfvt\" (UID: \"6fd17eaf-b363-4ddb-8dc3-29489cb00a87\") " pod="ai-tenant-e2e-inf-a-ad434e/test-model-ad434e-kserve-9d799b5f7-zcfvt"
Oct 30 07:44:49.603831 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:49.602942    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ad434e/test-model-ad434e-kserve-9d799b5f7-zcfvt"
Oct 30 07:44:49.942800 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:44:49.942772    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6fd17eaf_b363_4ddb_8dc3_29489cb00a87.slice/crio-39f4e929e3720b4cd95c42b86fca7934c75957753c423e3e9bea4b1491dba547 WatchSource:0}: Error finding container 39f4e929e3720b4cd95c42b86fca7934c75957753c423e3e9bea4b1491dba547: Status 404 returned error can't find the container with id 39f4e929e3720b4cd95c42b86fca7934c75957753c423e3e9bea4b1491dba547
Oct 30 07:44:49.943364 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:49.943342    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ad434e/test-model-ad434e-kserve-9d799b5f7-zcfvt"]
Oct 30 07:44:50.105013 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:50.104973    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ad434e/test-model-ad434e-kserve-9d799b5f7-zcfvt" event={"ID":"6fd17eaf-b363-4ddb-8dc3-29489cb00a87","Type":"ContainerStarted","Data":"733de4fa65bb32571beb7c0a1ce7353a9ced9e13a1177d29b573d615c25589e2"}
Oct 30 07:44:50.105202 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:50.105025    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ad434e/test-model-ad434e-kserve-9d799b5f7-zcfvt" event={"ID":"6fd17eaf-b363-4ddb-8dc3-29489cb00a87","Type":"ContainerStarted","Data":"39f4e929e3720b4cd95c42b86fca7934c75957753c423e3e9bea4b1491dba547"}
Oct 30 07:44:50.151065 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:50.151009    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-ad434e/test-model-ad434e-kserve-9d799b5f7-zcfvt" podStartSLOduration=1.150995992 podStartE2EDuration="1.150995992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:44:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:44:50.150743263 +0000 UTC m=+2848.760150234" watchObservedRunningTime="2026-10-30 07:44:50.150995992 +0000 UTC m=+2848.760402962"
Oct 30 07:44:50.603498 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:50.603462    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-ad434e/test-model-ad434e-kserve-9d799b5f7-zcfvt"
Oct 30 07:44:57.080322 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:57.080289    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-91c8c2-5b7b6b7c59-zlzl6"
Oct 30 07:44:57.080753 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:57.080360    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-91c8c2-67cf6564dd-z4qql"
Oct 30 07:44:59.603974 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:59.603943    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-ad434e/test-model-ad434e-kserve-9d799b5f7-zcfvt"
Oct 30 07:44:59.616420 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:44:59.616394    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-ad434e/test-model-ad434e-kserve-9d799b5f7-zcfvt"
Oct 30 07:45:00.191363 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.191331    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-ad434e/test-model-ad434e-kserve-9d799b5f7-zcfvt"
Oct 30 07:45:00.234081 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.234048    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8e7948-29889105-m6vwb"]
Oct 30 07:45:00.239754 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.239729    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8e7948-29889105-m6vwb"
Oct 30 07:45:00.244965 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.244939    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f5081-29889105-4t64q"]
Oct 30 07:45:00.252964 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.252304    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f5081-29889105-4t64q"
Oct 30 07:45:00.282229 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.282198    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8e7948-29889105-m6vwb"]
Oct 30 07:45:00.292434 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.292350    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f5081-29889105-4t64q"]
Oct 30 07:45:00.297029 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.296998    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889105-6dfr9"]
Oct 30 07:45:00.304093 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.304061    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889105-fmdqz"]
Oct 30 07:45:00.304368 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.304341    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889105-6dfr9"
Oct 30 07:45:00.305452 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.305433    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 07:45:00.321488 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.321222    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889105-fmdqz"
Oct 30 07:45:00.321990 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.321960    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889105-pkg77"]
Oct 30 07:45:00.326803 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.326776    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1641d01b-b971-4bf0-b3af-e1523a410dc5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-0f5081-29889105-4t64q\" (UID: \"1641d01b-b971-4bf0-b3af-e1523a410dc5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f5081-29889105-4t64q"
Oct 30 07:45:00.328949 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.327582    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdhqc\" (UniqueName: \"kubernetes.io/projected/1641d01b-b971-4bf0-b3af-e1523a410dc5-kube-api-access-sdhqc\") pod \"maas-api-key-cleanup-e2e-extmod-a-0f5081-29889105-4t64q\" (UID: \"1641d01b-b971-4bf0-b3af-e1523a410dc5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f5081-29889105-4t64q"
Oct 30 07:45:00.328949 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.327656    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87432693-c1f2-42bb-9a66-7c148d63f2fa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-8e7948-29889105-m6vwb\" (UID: \"87432693-c1f2-42bb-9a66-7c148d63f2fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8e7948-29889105-m6vwb"
Oct 30 07:45:00.336678 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.327721    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph9nd\" (UniqueName: \"kubernetes.io/projected/87432693-c1f2-42bb-9a66-7c148d63f2fa-kube-api-access-ph9nd\") pod \"maas-api-key-cleanup-e2e-extmod-a-8e7948-29889105-m6vwb\" (UID: \"87432693-c1f2-42bb-9a66-7c148d63f2fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8e7948-29889105-m6vwb"
Oct 30 07:45:00.347096 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.347051    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889105-pkg77"
Oct 30 07:45:00.353525 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.353213    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889105-6dfr9"]
Oct 30 07:45:00.362044 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.362010    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889105-fmdqz"]
Oct 30 07:45:00.382871 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.382843    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889105-4x5vf"]
Oct 30 07:45:00.389216 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.389185    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889105-4x5vf"
Oct 30 07:45:00.391499 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.391474    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889105-pkg77"]
Oct 30 07:45:00.425014 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.421607    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8fef73-29889105-xmpv9"]
Oct 30 07:45:00.441848 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.441183    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8fef73-29889105-xmpv9"
Oct 30 07:45:00.448313 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.448100    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbt6r\" (UniqueName: \"kubernetes.io/projected/08398783-5da2-4d6b-b0d0-4d141510de4b-kube-api-access-jbt6r\") pod \"maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889105-4x5vf\" (UID: \"08398783-5da2-4d6b-b0d0-4d141510de4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889105-4x5vf"
Oct 30 07:45:00.448313 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.448180    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afde60aa-712f-46da-a245-9df1d7e8326f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889105-6dfr9\" (UID: \"afde60aa-712f-46da-a245-9df1d7e8326f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889105-6dfr9"
Oct 30 07:45:00.448313 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.448226    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889105-4x5vf"]
Oct 30 07:45:00.448637 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.448330    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1641d01b-b971-4bf0-b3af-e1523a410dc5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-0f5081-29889105-4t64q\" (UID: \"1641d01b-b971-4bf0-b3af-e1523a410dc5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f5081-29889105-4t64q"
Oct 30 07:45:00.448637 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.448414    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb2jw\" (UniqueName: \"kubernetes.io/projected/afde60aa-712f-46da-a245-9df1d7e8326f-kube-api-access-kb2jw\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889105-6dfr9\" (UID: \"afde60aa-712f-46da-a245-9df1d7e8326f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889105-6dfr9"
Oct 30 07:45:00.448637 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.448513    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdhqc\" (UniqueName: \"kubernetes.io/projected/1641d01b-b971-4bf0-b3af-e1523a410dc5-kube-api-access-sdhqc\") pod \"maas-api-key-cleanup-e2e-extmod-a-0f5081-29889105-4t64q\" (UID: \"1641d01b-b971-4bf0-b3af-e1523a410dc5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f5081-29889105-4t64q"
Oct 30 07:45:00.448637 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.448552    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1966a9a-09be-440c-a430-8c79b8c89684-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-137122-29889105-fmdqz\" (UID: \"b1966a9a-09be-440c-a430-8c79b8c89684\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889105-fmdqz"
Oct 30 07:45:00.448637 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.448585    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxp4q\" (UniqueName: \"kubernetes.io/projected/b1966a9a-09be-440c-a430-8c79b8c89684-kube-api-access-lxp4q\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-137122-29889105-fmdqz\" (UID: \"b1966a9a-09be-440c-a430-8c79b8c89684\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889105-fmdqz"
Oct 30 07:45:00.448637 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.448620    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxwm5\" (UniqueName: \"kubernetes.io/projected/abc98592-b397-43d3-8169-91d59a211e6d-kube-api-access-wxwm5\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889105-pkg77\" (UID: \"abc98592-b397-43d3-8169-91d59a211e6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889105-pkg77"
Oct 30 07:45:00.449016 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.448644    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87432693-c1f2-42bb-9a66-7c148d63f2fa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-8e7948-29889105-m6vwb\" (UID: \"87432693-c1f2-42bb-9a66-7c148d63f2fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8e7948-29889105-m6vwb"
Oct 30 07:45:00.449016 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.448671    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph9nd\" (UniqueName: \"kubernetes.io/projected/87432693-c1f2-42bb-9a66-7c148d63f2fa-kube-api-access-ph9nd\") pod \"maas-api-key-cleanup-e2e-extmod-a-8e7948-29889105-m6vwb\" (UID: \"87432693-c1f2-42bb-9a66-7c148d63f2fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8e7948-29889105-m6vwb"
Oct 30 07:45:00.449016 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.448716    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abc98592-b397-43d3-8169-91d59a211e6d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889105-pkg77\" (UID: \"abc98592-b397-43d3-8169-91d59a211e6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889105-pkg77"
Oct 30 07:45:00.449016 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.448835    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08398783-5da2-4d6b-b0d0-4d141510de4b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889105-4x5vf\" (UID: \"08398783-5da2-4d6b-b0d0-4d141510de4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889105-4x5vf"
Oct 30 07:45:00.450290 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.449634    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1641d01b-b971-4bf0-b3af-e1523a410dc5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-0f5081-29889105-4t64q\" (UID: \"1641d01b-b971-4bf0-b3af-e1523a410dc5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f5081-29889105-4t64q"
Oct 30 07:45:00.450290 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.450174    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87432693-c1f2-42bb-9a66-7c148d63f2fa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-8e7948-29889105-m6vwb\" (UID: \"87432693-c1f2-42bb-9a66-7c148d63f2fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8e7948-29889105-m6vwb"
Oct 30 07:45:00.469108 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.469086    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8fef73-29889105-xmpv9"]
Oct 30 07:45:00.475777 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.475740    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ad434e-29889105-fljfl"]
Oct 30 07:45:00.480309 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.480281    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889105-gmwl4"]
Oct 30 07:45:00.481079 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.480425    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ad434e-29889105-fljfl"
Oct 30 07:45:00.493379 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.493248    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdhqc\" (UniqueName: \"kubernetes.io/projected/1641d01b-b971-4bf0-b3af-e1523a410dc5-kube-api-access-sdhqc\") pod \"maas-api-key-cleanup-e2e-extmod-a-0f5081-29889105-4t64q\" (UID: \"1641d01b-b971-4bf0-b3af-e1523a410dc5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f5081-29889105-4t64q"
Oct 30 07:45:00.493379 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.493329    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph9nd\" (UniqueName: \"kubernetes.io/projected/87432693-c1f2-42bb-9a66-7c148d63f2fa-kube-api-access-ph9nd\") pod \"maas-api-key-cleanup-e2e-extmod-a-8e7948-29889105-m6vwb\" (UID: \"87432693-c1f2-42bb-9a66-7c148d63f2fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8e7948-29889105-m6vwb"
Oct 30 07:45:00.496120 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.495435    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889105-gmwl4"
Oct 30 07:45:00.501278 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.501206    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ad434e-29889105-fljfl"]
Oct 30 07:45:00.518996 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.518962    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889105-gmwl4"]
Oct 30 07:45:00.550478 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.550450    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1966a9a-09be-440c-a430-8c79b8c89684-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-137122-29889105-fmdqz\" (UID: \"b1966a9a-09be-440c-a430-8c79b8c89684\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889105-fmdqz"
Oct 30 07:45:00.550617 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.550487    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxp4q\" (UniqueName: \"kubernetes.io/projected/b1966a9a-09be-440c-a430-8c79b8c89684-kube-api-access-lxp4q\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-137122-29889105-fmdqz\" (UID: \"b1966a9a-09be-440c-a430-8c79b8c89684\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889105-fmdqz"
Oct 30 07:45:00.550702 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.550620    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxwm5\" (UniqueName: \"kubernetes.io/projected/abc98592-b397-43d3-8169-91d59a211e6d-kube-api-access-wxwm5\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889105-pkg77\" (UID: \"abc98592-b397-43d3-8169-91d59a211e6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889105-pkg77"
Oct 30 07:45:00.550702 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.550695    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abc98592-b397-43d3-8169-91d59a211e6d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889105-pkg77\" (UID: \"abc98592-b397-43d3-8169-91d59a211e6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889105-pkg77"
Oct 30 07:45:00.550872 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.550851    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08398783-5da2-4d6b-b0d0-4d141510de4b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889105-4x5vf\" (UID: \"08398783-5da2-4d6b-b0d0-4d141510de4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889105-4x5vf"
Oct 30 07:45:00.550946 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.550908    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4rnh\" (UniqueName: \"kubernetes.io/projected/49a8dd13-feae-465b-a031-aea87e1d8a59-kube-api-access-f4rnh\") pod \"maas-api-key-cleanup-e2e-inf-a-8fef73-29889105-xmpv9\" (UID: \"49a8dd13-feae-465b-a031-aea87e1d8a59\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8fef73-29889105-xmpv9"
Oct 30 07:45:00.551009 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.550987    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbt6r\" (UniqueName: \"kubernetes.io/projected/08398783-5da2-4d6b-b0d0-4d141510de4b-kube-api-access-jbt6r\") pod \"maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889105-4x5vf\" (UID: \"08398783-5da2-4d6b-b0d0-4d141510de4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889105-4x5vf"
Oct 30 07:45:00.551077 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.551043    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afde60aa-712f-46da-a245-9df1d7e8326f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889105-6dfr9\" (UID: \"afde60aa-712f-46da-a245-9df1d7e8326f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889105-6dfr9"
Oct 30 07:45:00.551146 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.551106    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j27d8\" (UniqueName: \"kubernetes.io/projected/f487dd5e-0777-4bfb-98c9-81b2602e63a3-kube-api-access-j27d8\") pod \"maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889105-gmwl4\" (UID: \"f487dd5e-0777-4bfb-98c9-81b2602e63a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889105-gmwl4"
Oct 30 07:45:00.551205 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.551142    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b63a195-1c69-4fdd-ba65-07fdc7996d37-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-ad434e-29889105-fljfl\" (UID: \"7b63a195-1c69-4fdd-ba65-07fdc7996d37\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ad434e-29889105-fljfl"
Oct 30 07:45:00.551205 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.551166    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p76t4\" (UniqueName: \"kubernetes.io/projected/7b63a195-1c69-4fdd-ba65-07fdc7996d37-kube-api-access-p76t4\") pod \"maas-api-key-cleanup-e2e-inf-a-ad434e-29889105-fljfl\" (UID: \"7b63a195-1c69-4fdd-ba65-07fdc7996d37\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ad434e-29889105-fljfl"
Oct 30 07:45:00.551316 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.551206    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49a8dd13-feae-465b-a031-aea87e1d8a59-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-8fef73-29889105-xmpv9\" (UID: \"49a8dd13-feae-465b-a031-aea87e1d8a59\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8fef73-29889105-xmpv9"
Oct 30 07:45:00.551316 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.551237    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f487dd5e-0777-4bfb-98c9-81b2602e63a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889105-gmwl4\" (UID: \"f487dd5e-0777-4bfb-98c9-81b2602e63a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889105-gmwl4"
Oct 30 07:45:00.551316 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.551282    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1966a9a-09be-440c-a430-8c79b8c89684-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-137122-29889105-fmdqz\" (UID: \"b1966a9a-09be-440c-a430-8c79b8c89684\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889105-fmdqz"
Oct 30 07:45:00.551316 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.551309    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb2jw\" (UniqueName: \"kubernetes.io/projected/afde60aa-712f-46da-a245-9df1d7e8326f-kube-api-access-kb2jw\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889105-6dfr9\" (UID: \"afde60aa-712f-46da-a245-9df1d7e8326f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889105-6dfr9"
Oct 30 07:45:00.551532 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.551504    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08398783-5da2-4d6b-b0d0-4d141510de4b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889105-4x5vf\" (UID: \"08398783-5da2-4d6b-b0d0-4d141510de4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889105-4x5vf"
Oct 30 07:45:00.551585 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.551546    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abc98592-b397-43d3-8169-91d59a211e6d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889105-pkg77\" (UID: \"abc98592-b397-43d3-8169-91d59a211e6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889105-pkg77"
Oct 30 07:45:00.552091 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.552072    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afde60aa-712f-46da-a245-9df1d7e8326f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889105-6dfr9\" (UID: \"afde60aa-712f-46da-a245-9df1d7e8326f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889105-6dfr9"
Oct 30 07:45:00.559078 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.559058    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8e7948-29889105-m6vwb"
Oct 30 07:45:00.570261 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.570239    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f5081-29889105-4t64q"
Oct 30 07:45:00.582110 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.582087    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxp4q\" (UniqueName: \"kubernetes.io/projected/b1966a9a-09be-440c-a430-8c79b8c89684-kube-api-access-lxp4q\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-137122-29889105-fmdqz\" (UID: \"b1966a9a-09be-440c-a430-8c79b8c89684\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889105-fmdqz"
Oct 30 07:45:00.582258 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.582168    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb2jw\" (UniqueName: \"kubernetes.io/projected/afde60aa-712f-46da-a245-9df1d7e8326f-kube-api-access-kb2jw\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889105-6dfr9\" (UID: \"afde60aa-712f-46da-a245-9df1d7e8326f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889105-6dfr9"
Oct 30 07:45:00.582362 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.582274    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxwm5\" (UniqueName: \"kubernetes.io/projected/abc98592-b397-43d3-8169-91d59a211e6d-kube-api-access-wxwm5\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889105-pkg77\" (UID: \"abc98592-b397-43d3-8169-91d59a211e6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889105-pkg77"
Oct 30 07:45:00.584422 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.584400    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbt6r\" (UniqueName: \"kubernetes.io/projected/08398783-5da2-4d6b-b0d0-4d141510de4b-kube-api-access-jbt6r\") pod \"maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889105-4x5vf\" (UID: \"08398783-5da2-4d6b-b0d0-4d141510de4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889105-4x5vf"
Oct 30 07:45:00.654863 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.654824    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4rnh\" (UniqueName: \"kubernetes.io/projected/49a8dd13-feae-465b-a031-aea87e1d8a59-kube-api-access-f4rnh\") pod \"maas-api-key-cleanup-e2e-inf-a-8fef73-29889105-xmpv9\" (UID: \"49a8dd13-feae-465b-a031-aea87e1d8a59\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8fef73-29889105-xmpv9"
Oct 30 07:45:00.655251 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.654961    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j27d8\" (UniqueName: \"kubernetes.io/projected/f487dd5e-0777-4bfb-98c9-81b2602e63a3-kube-api-access-j27d8\") pod \"maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889105-gmwl4\" (UID: \"f487dd5e-0777-4bfb-98c9-81b2602e63a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889105-gmwl4"
Oct 30 07:45:00.655251 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.655002    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b63a195-1c69-4fdd-ba65-07fdc7996d37-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-ad434e-29889105-fljfl\" (UID: \"7b63a195-1c69-4fdd-ba65-07fdc7996d37\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ad434e-29889105-fljfl"
Oct 30 07:45:00.655251 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.655028    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p76t4\" (UniqueName: \"kubernetes.io/projected/7b63a195-1c69-4fdd-ba65-07fdc7996d37-kube-api-access-p76t4\") pod \"maas-api-key-cleanup-e2e-inf-a-ad434e-29889105-fljfl\" (UID: \"7b63a195-1c69-4fdd-ba65-07fdc7996d37\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ad434e-29889105-fljfl"
Oct 30 07:45:00.655251 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.655065    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49a8dd13-feae-465b-a031-aea87e1d8a59-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-8fef73-29889105-xmpv9\" (UID: \"49a8dd13-feae-465b-a031-aea87e1d8a59\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8fef73-29889105-xmpv9"
Oct 30 07:45:00.655251 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.655101    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f487dd5e-0777-4bfb-98c9-81b2602e63a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889105-gmwl4\" (UID: \"f487dd5e-0777-4bfb-98c9-81b2602e63a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889105-gmwl4"
Oct 30 07:45:00.655905 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.655881    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889105-6dfr9"
Oct 30 07:45:00.656073 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.656047    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49a8dd13-feae-465b-a031-aea87e1d8a59-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-8fef73-29889105-xmpv9\" (UID: \"49a8dd13-feae-465b-a031-aea87e1d8a59\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8fef73-29889105-xmpv9"
Oct 30 07:45:00.656149 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.656125    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b63a195-1c69-4fdd-ba65-07fdc7996d37-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-ad434e-29889105-fljfl\" (UID: \"7b63a195-1c69-4fdd-ba65-07fdc7996d37\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ad434e-29889105-fljfl"
Oct 30 07:45:00.661251 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.661159    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f487dd5e-0777-4bfb-98c9-81b2602e63a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889105-gmwl4\" (UID: \"f487dd5e-0777-4bfb-98c9-81b2602e63a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889105-gmwl4"
Oct 30 07:45:00.665205 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.665181    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889105-fmdqz"
Oct 30 07:45:00.678357 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.678336    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889105-pkg77"
Oct 30 07:45:00.682451 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.682406    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j27d8\" (UniqueName: \"kubernetes.io/projected/f487dd5e-0777-4bfb-98c9-81b2602e63a3-kube-api-access-j27d8\") pod \"maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889105-gmwl4\" (UID: \"f487dd5e-0777-4bfb-98c9-81b2602e63a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889105-gmwl4"
Oct 30 07:45:00.682451 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.682404    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p76t4\" (UniqueName: \"kubernetes.io/projected/7b63a195-1c69-4fdd-ba65-07fdc7996d37-kube-api-access-p76t4\") pod \"maas-api-key-cleanup-e2e-inf-a-ad434e-29889105-fljfl\" (UID: \"7b63a195-1c69-4fdd-ba65-07fdc7996d37\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ad434e-29889105-fljfl"
Oct 30 07:45:00.685090 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.685053    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4rnh\" (UniqueName: \"kubernetes.io/projected/49a8dd13-feae-465b-a031-aea87e1d8a59-kube-api-access-f4rnh\") pod \"maas-api-key-cleanup-e2e-inf-a-8fef73-29889105-xmpv9\" (UID: \"49a8dd13-feae-465b-a031-aea87e1d8a59\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8fef73-29889105-xmpv9"
Oct 30 07:45:00.715744 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.715548    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889105-4x5vf"
Oct 30 07:45:00.781043 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.777036    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8fef73-29889105-xmpv9"
Oct 30 07:45:00.842541 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.842512    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ad434e-29889105-fljfl"
Oct 30 07:45:00.858594 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.858565    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889105-gmwl4"
Oct 30 07:45:00.887835 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.887575    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8e7948-29889105-m6vwb"]
Oct 30 07:45:00.893986 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.893956    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f5081-29889105-4t64q"]
Oct 30 07:45:00.909861 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:45:00.909757    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1641d01b_b971_4bf0_b3af_e1523a410dc5.slice/crio-b5a60519afc34082f130ccd32b985b7c7ae3584b90576aabc6c1b68e87ac0d05 WatchSource:0}: Error finding container b5a60519afc34082f130ccd32b985b7c7ae3584b90576aabc6c1b68e87ac0d05: Status 404 returned error can't find the container with id b5a60519afc34082f130ccd32b985b7c7ae3584b90576aabc6c1b68e87ac0d05
Oct 30 07:45:00.993992 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:00.993951    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889105-6dfr9"]
Oct 30 07:45:00.996093 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:45:00.996049    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafde60aa_712f_46da_a245_9df1d7e8326f.slice/crio-132101cce56bcafcf4f0bcccbeb750bd7efe143009b480beda4119b39273242b WatchSource:0}: Error finding container 132101cce56bcafcf4f0bcccbeb750bd7efe143009b480beda4119b39273242b: Status 404 returned error can't find the container with id 132101cce56bcafcf4f0bcccbeb750bd7efe143009b480beda4119b39273242b
Oct 30 07:45:01.024797 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:45:01.023850    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabc98592_b397_43d3_8169_91d59a211e6d.slice/crio-54650e0ad8ef0db03e087db37b65c86ce12a5a5817cf4d2788239804c7acaad6 WatchSource:0}: Error finding container 54650e0ad8ef0db03e087db37b65c86ce12a5a5817cf4d2788239804c7acaad6: Status 404 returned error can't find the container with id 54650e0ad8ef0db03e087db37b65c86ce12a5a5817cf4d2788239804c7acaad6
Oct 30 07:45:01.029790 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:01.029761    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889105-pkg77"]
Oct 30 07:45:01.039219 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:01.039193    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889105-fmdqz"]
Oct 30 07:45:01.076695 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:01.075380    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889105-4x5vf"]
Oct 30 07:45:01.136084 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:01.136054    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8fef73-29889105-xmpv9"]
Oct 30 07:45:01.168330 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:45:01.168296    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08398783_5da2_4d6b_b0d0_4d141510de4b.slice/crio-52602708aac77f774fc197ea5e33b8c34d001a9aebf8260fda8aab6808cdcc32 WatchSource:0}: Error finding container 52602708aac77f774fc197ea5e33b8c34d001a9aebf8260fda8aab6808cdcc32: Status 404 returned error can't find the container with id 52602708aac77f774fc197ea5e33b8c34d001a9aebf8260fda8aab6808cdcc32
Oct 30 07:45:01.178652 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:45:01.178617    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49a8dd13_feae_465b_a031_aea87e1d8a59.slice/crio-02c3df0cb8ae75b003fb89b7cc015f3a152d2bc0d0d48e8ac88919e032b69a05 WatchSource:0}: Error finding container 02c3df0cb8ae75b003fb89b7cc015f3a152d2bc0d0d48e8ac88919e032b69a05: Status 404 returned error can't find the container with id 02c3df0cb8ae75b003fb89b7cc015f3a152d2bc0d0d48e8ac88919e032b69a05
Oct 30 07:45:01.188545 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:01.188514    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8e7948-29889105-m6vwb" event={"ID":"87432693-c1f2-42bb-9a66-7c148d63f2fa","Type":"ContainerStarted","Data":"47a6a9df9e78a6bca65e139cfe3bafc1f0d155c91767c32c2beabc98e809842b"}
Oct 30 07:45:01.190445 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:01.190415    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889105-fmdqz" event={"ID":"b1966a9a-09be-440c-a430-8c79b8c89684","Type":"ContainerStarted","Data":"be9ff56a94957b0c5324a245dbc9b1e30e0baa5f4c76768bb6814ae8ec3bb51c"}
Oct 30 07:45:01.193083 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:01.193059    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889105-4x5vf" event={"ID":"08398783-5da2-4d6b-b0d0-4d141510de4b","Type":"ContainerStarted","Data":"52602708aac77f774fc197ea5e33b8c34d001a9aebf8260fda8aab6808cdcc32"}
Oct 30 07:45:01.194379 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:01.194357    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889105-pkg77" event={"ID":"abc98592-b397-43d3-8169-91d59a211e6d","Type":"ContainerStarted","Data":"54650e0ad8ef0db03e087db37b65c86ce12a5a5817cf4d2788239804c7acaad6"}
Oct 30 07:45:01.195677 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:01.195649    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889105-6dfr9" event={"ID":"afde60aa-712f-46da-a245-9df1d7e8326f","Type":"ContainerStarted","Data":"132101cce56bcafcf4f0bcccbeb750bd7efe143009b480beda4119b39273242b"}
Oct 30 07:45:01.196792 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:01.196766    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f5081-29889105-4t64q" event={"ID":"1641d01b-b971-4bf0-b3af-e1523a410dc5","Type":"ContainerStarted","Data":"b5a60519afc34082f130ccd32b985b7c7ae3584b90576aabc6c1b68e87ac0d05"}
Oct 30 07:45:01.234409 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:01.234387    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ad434e-29889105-fljfl"]
Oct 30 07:45:01.248985 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:01.248963    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889105-gmwl4"]
Oct 30 07:45:01.267302 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:45:01.267274    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b63a195_1c69_4fdd_ba65_07fdc7996d37.slice/crio-7919f07cd39529c024bb7c472258a909c45032e3876aad37b3d4c302f0239739 WatchSource:0}: Error finding container 7919f07cd39529c024bb7c472258a909c45032e3876aad37b3d4c302f0239739: Status 404 returned error can't find the container with id 7919f07cd39529c024bb7c472258a909c45032e3876aad37b3d4c302f0239739
Oct 30 07:45:01.267554 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:45:01.267526    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf487dd5e_0777_4bfb_98c9_81b2602e63a3.slice/crio-23196585e56eb9b0961f3a0994c8d7dc89ef12e924caf3ef17d08061d561a0fe WatchSource:0}: Error finding container 23196585e56eb9b0961f3a0994c8d7dc89ef12e924caf3ef17d08061d561a0fe: Status 404 returned error can't find the container with id 23196585e56eb9b0961f3a0994c8d7dc89ef12e924caf3ef17d08061d561a0fe
Oct 30 07:45:01.966068 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:45:01.965904    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:45:02.210766 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:02.210667    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889105-4x5vf" event={"ID":"08398783-5da2-4d6b-b0d0-4d141510de4b","Type":"ContainerStarted","Data":"d63778381dcd0de2ffdd39b0f2b1b2f450aaad2fc36cff4cfafe7fb0dbafad15"}
Oct 30 07:45:02.212697 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:02.212672    6224 generic.go:356] "Generic (PLEG): container finished" podID="abc98592-b397-43d3-8169-91d59a211e6d" containerID="6a6d032dfd93909eb06c4710b589bfe21041fe987378a419bf37aa9c2dab2021" exitCode=60
Oct 30 07:45:02.212894 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:02.212751    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889105-pkg77" event={"ID":"abc98592-b397-43d3-8169-91d59a211e6d","Type":"ContainerDied","Data":"6a6d032dfd93909eb06c4710b589bfe21041fe987378a419bf37aa9c2dab2021"}
Oct 30 07:45:02.213207 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:02.213174    6224 scope.go:122] "RemoveContainer" containerID="6a6d032dfd93909eb06c4710b589bfe21041fe987378a419bf37aa9c2dab2021"
Oct 30 07:45:02.215206 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:02.215110    6224 generic.go:356] "Generic (PLEG): container finished" podID="afde60aa-712f-46da-a245-9df1d7e8326f" containerID="61362905f5fe85dbb211de98a50b0f0ade19492401a48cf5017fa9fa1cee3af2" exitCode=60
Oct 30 07:45:02.215206 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:02.215183    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889105-6dfr9" event={"ID":"afde60aa-712f-46da-a245-9df1d7e8326f","Type":"ContainerDied","Data":"61362905f5fe85dbb211de98a50b0f0ade19492401a48cf5017fa9fa1cee3af2"}
Oct 30 07:45:02.215427 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:02.215399    6224 scope.go:122] "RemoveContainer" containerID="61362905f5fe85dbb211de98a50b0f0ade19492401a48cf5017fa9fa1cee3af2"
Oct 30 07:45:02.219100 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:02.217312    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f5081-29889105-4t64q" event={"ID":"1641d01b-b971-4bf0-b3af-e1523a410dc5","Type":"ContainerStarted","Data":"267c2bf7110a17f67a0f301ba624e199bc37cfdcad6ec9f7a948866ebf634d51"}
Oct 30 07:45:02.219720 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:02.219698    6224 generic.go:356] "Generic (PLEG): container finished" podID="87432693-c1f2-42bb-9a66-7c148d63f2fa" containerID="393e6611b4459fd58feee08d599c0f1edbdea38796274dc2b277f127222a3304" exitCode=60
Oct 30 07:45:02.219836 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:02.219756    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8e7948-29889105-m6vwb" event={"ID":"87432693-c1f2-42bb-9a66-7c148d63f2fa","Type":"ContainerDied","Data":"393e6611b4459fd58feee08d599c0f1edbdea38796274dc2b277f127222a3304"}
Oct 30 07:45:02.220026 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:02.220008    6224 scope.go:122] "RemoveContainer" containerID="393e6611b4459fd58feee08d599c0f1edbdea38796274dc2b277f127222a3304"
Oct 30 07:45:02.221999 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:02.221978    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889105-gmwl4" event={"ID":"f487dd5e-0777-4bfb-98c9-81b2602e63a3","Type":"ContainerStarted","Data":"23196585e56eb9b0961f3a0994c8d7dc89ef12e924caf3ef17d08061d561a0fe"}
Oct 30 07:45:02.223672 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:02.223650    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889105-fmdqz" event={"ID":"b1966a9a-09be-440c-a430-8c79b8c89684","Type":"ContainerStarted","Data":"3750f8b9203ea9f73bbb0b0ac74f786d520b6018bfd624f078a2aaa21ea69469"}
Oct 30 07:45:02.225063 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:02.225026    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8fef73-29889105-xmpv9" event={"ID":"49a8dd13-feae-465b-a031-aea87e1d8a59","Type":"ContainerStarted","Data":"02c3df0cb8ae75b003fb89b7cc015f3a152d2bc0d0d48e8ac88919e032b69a05"}
Oct 30 07:45:02.226611 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:02.226590    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ad434e-29889105-fljfl" event={"ID":"7b63a195-1c69-4fdd-ba65-07fdc7996d37","Type":"ContainerStarted","Data":"6d75ed5da272510c436400896dde321df92ebe45aff1ef19c4af31688ee16d11"}
Oct 30 07:45:02.226727 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:02.226633    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ad434e-29889105-fljfl" event={"ID":"7b63a195-1c69-4fdd-ba65-07fdc7996d37","Type":"ContainerStarted","Data":"7919f07cd39529c024bb7c472258a909c45032e3876aad37b3d4c302f0239739"}
Oct 30 07:45:02.258578 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:02.258524    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889105-4x5vf" podStartSLOduration=2.258505971 podStartE2EDuration="2.258505971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:45:02.254172537 +0000 UTC m=+2860.863579509" watchObservedRunningTime="2026-10-30 07:45:02.258505971 +0000 UTC m=+2860.867912941"
Oct 30 07:45:02.305470 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:02.305419    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889105-fmdqz" podStartSLOduration=2.305406866 podStartE2EDuration="2.305406866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:45:02.304109708 +0000 UTC m=+2860.913516680" watchObservedRunningTime="2026-10-30 07:45:02.305406866 +0000 UTC m=+2860.914813836"
Oct 30 07:45:02.350707 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:02.350656    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f5081-29889105-4t64q" podStartSLOduration=2.350643877 podStartE2EDuration="2.350643877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:45:02.347153979 +0000 UTC m=+2860.956560950" watchObservedRunningTime="2026-10-30 07:45:02.350643877 +0000 UTC m=+2860.960050867"
Oct 30 07:45:02.440086 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:02.440024    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ad434e-29889105-fljfl" podStartSLOduration=2.440003009 podStartE2EDuration="2.440003009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:45:02.433260529 +0000 UTC m=+2861.042667499" watchObservedRunningTime="2026-10-30 07:45:02.440003009 +0000 UTC m=+2861.049409980"
Oct 30 07:45:03.235722 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:03.235682    6224 generic.go:356] "Generic (PLEG): container finished" podID="b1966a9a-09be-440c-a430-8c79b8c89684" containerID="3750f8b9203ea9f73bbb0b0ac74f786d520b6018bfd624f078a2aaa21ea69469" exitCode=60
Oct 30 07:45:03.236245 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:03.235777    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889105-fmdqz" event={"ID":"b1966a9a-09be-440c-a430-8c79b8c89684","Type":"ContainerDied","Data":"3750f8b9203ea9f73bbb0b0ac74f786d520b6018bfd624f078a2aaa21ea69469"}
Oct 30 07:45:03.236245 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:03.236133    6224 scope.go:122] "RemoveContainer" containerID="3750f8b9203ea9f73bbb0b0ac74f786d520b6018bfd624f078a2aaa21ea69469"
Oct 30 07:45:03.237678 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:03.237654    6224 generic.go:356] "Generic (PLEG): container finished" podID="49a8dd13-feae-465b-a031-aea87e1d8a59" containerID="afc85b428f5ba608cf74790ef5315b750f541a795c11e4f1509a9eb3ea8d4dc6" exitCode=60
Oct 30 07:45:03.237856 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:03.237764    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8fef73-29889105-xmpv9" event={"ID":"49a8dd13-feae-465b-a031-aea87e1d8a59","Type":"ContainerDied","Data":"afc85b428f5ba608cf74790ef5315b750f541a795c11e4f1509a9eb3ea8d4dc6"}
Oct 30 07:45:03.238491 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:03.238470    6224 scope.go:122] "RemoveContainer" containerID="afc85b428f5ba608cf74790ef5315b750f541a795c11e4f1509a9eb3ea8d4dc6"
Oct 30 07:45:03.240084 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:03.240060    6224 generic.go:356] "Generic (PLEG): container finished" podID="7b63a195-1c69-4fdd-ba65-07fdc7996d37" containerID="6d75ed5da272510c436400896dde321df92ebe45aff1ef19c4af31688ee16d11" exitCode=60
Oct 30 07:45:03.240245 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:03.240184    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ad434e-29889105-fljfl" event={"ID":"7b63a195-1c69-4fdd-ba65-07fdc7996d37","Type":"ContainerDied","Data":"6d75ed5da272510c436400896dde321df92ebe45aff1ef19c4af31688ee16d11"}
Oct 30 07:45:03.240427 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:03.240410    6224 scope.go:122] "RemoveContainer" containerID="6d75ed5da272510c436400896dde321df92ebe45aff1ef19c4af31688ee16d11"
Oct 30 07:45:03.242240 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:03.242206    6224 generic.go:356] "Generic (PLEG): container finished" podID="08398783-5da2-4d6b-b0d0-4d141510de4b" containerID="d63778381dcd0de2ffdd39b0f2b1b2f450aaad2fc36cff4cfafe7fb0dbafad15" exitCode=60
Oct 30 07:45:03.242288 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:03.242232    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889105-4x5vf" event={"ID":"08398783-5da2-4d6b-b0d0-4d141510de4b","Type":"ContainerDied","Data":"d63778381dcd0de2ffdd39b0f2b1b2f450aaad2fc36cff4cfafe7fb0dbafad15"}
Oct 30 07:45:03.242531 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:03.242509    6224 scope.go:122] "RemoveContainer" containerID="d63778381dcd0de2ffdd39b0f2b1b2f450aaad2fc36cff4cfafe7fb0dbafad15"
Oct 30 07:45:03.244481 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:03.244456    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889105-pkg77" event={"ID":"abc98592-b397-43d3-8169-91d59a211e6d","Type":"ContainerStarted","Data":"6e2d110c5204125a9307e5f56578cd7a452cb32a4618f4a883f37eb8faa4c070"}
Oct 30 07:45:03.246384 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:03.246363    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889105-6dfr9" event={"ID":"afde60aa-712f-46da-a245-9df1d7e8326f","Type":"ContainerStarted","Data":"eced858fda9ba0a81bb9c554b767b67b6b466bb046fe76886d7c2e9c38cb1b83"}
Oct 30 07:45:03.248204 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:03.248182    6224 generic.go:356] "Generic (PLEG): container finished" podID="1641d01b-b971-4bf0-b3af-e1523a410dc5" containerID="267c2bf7110a17f67a0f301ba624e199bc37cfdcad6ec9f7a948866ebf634d51" exitCode=60
Oct 30 07:45:03.248307 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:03.248232    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f5081-29889105-4t64q" event={"ID":"1641d01b-b971-4bf0-b3af-e1523a410dc5","Type":"ContainerDied","Data":"267c2bf7110a17f67a0f301ba624e199bc37cfdcad6ec9f7a948866ebf634d51"}
Oct 30 07:45:03.248456 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:03.248432    6224 scope.go:122] "RemoveContainer" containerID="267c2bf7110a17f67a0f301ba624e199bc37cfdcad6ec9f7a948866ebf634d51"
Oct 30 07:45:03.256285 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:03.256256    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8e7948-29889105-m6vwb" event={"ID":"87432693-c1f2-42bb-9a66-7c148d63f2fa","Type":"ContainerStarted","Data":"c93716d798fa966e6a75877e924c34da70093e39f12ba58b7e2f55fd4ec60c8f"}
Oct 30 07:45:03.258577 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:03.258554    6224 generic.go:356] "Generic (PLEG): container finished" podID="f487dd5e-0777-4bfb-98c9-81b2602e63a3" containerID="831735b27cdbe870febc1af8a50772486818f1775d081dadc56a07e5c9d33043" exitCode=60
Oct 30 07:45:03.258752 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:03.258733    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889105-gmwl4" event={"ID":"f487dd5e-0777-4bfb-98c9-81b2602e63a3","Type":"ContainerDied","Data":"831735b27cdbe870febc1af8a50772486818f1775d081dadc56a07e5c9d33043"}
Oct 30 07:45:03.259050 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:03.259033    6224 scope.go:122] "RemoveContainer" containerID="831735b27cdbe870febc1af8a50772486818f1775d081dadc56a07e5c9d33043"
Oct 30 07:45:03.421454 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:03.421403    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889105-pkg77" podStartSLOduration=3.42138942 podStartE2EDuration="3.42138942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:45:03.419179088 +0000 UTC m=+2862.028586059" watchObservedRunningTime="2026-10-30 07:45:03.42138942 +0000 UTC m=+2862.030796389"
Oct 30 07:45:03.560624 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:03.560575    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8e7948-29889105-m6vwb" podStartSLOduration=3.560556364 podStartE2EDuration="3.560556364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:45:03.558976278 +0000 UTC m=+2862.168383249" watchObservedRunningTime="2026-10-30 07:45:03.560556364 +0000 UTC m=+2862.169963336"
Oct 30 07:45:03.606241 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:03.606190    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889105-6dfr9" podStartSLOduration=3.6061751060000002 podStartE2EDuration="3.606175106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:45:03.60501985 +0000 UTC m=+2862.214426821" watchObservedRunningTime="2026-10-30 07:45:03.606175106 +0000 UTC m=+2862.215582075"
Oct 30 07:45:04.271578 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:04.271491    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ad434e-29889105-fljfl" event={"ID":"7b63a195-1c69-4fdd-ba65-07fdc7996d37","Type":"ContainerStarted","Data":"eeef350ba9884ce9faac4e3652201f7778d6e0da186a02e65b66dcc604699759"}
Oct 30 07:45:04.273781 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:04.273754    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889105-4x5vf" event={"ID":"08398783-5da2-4d6b-b0d0-4d141510de4b","Type":"ContainerStarted","Data":"71d844727a7a2b9e6801908c13a54f62854b8f1d56fb590af58d17a07340d2b4"}
Oct 30 07:45:04.275995 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:04.275969    6224 generic.go:356] "Generic (PLEG): container finished" podID="abc98592-b397-43d3-8169-91d59a211e6d" containerID="6e2d110c5204125a9307e5f56578cd7a452cb32a4618f4a883f37eb8faa4c070" exitCode=60
Oct 30 07:45:04.276106 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:04.276044    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889105-pkg77" event={"ID":"abc98592-b397-43d3-8169-91d59a211e6d","Type":"ContainerDied","Data":"6e2d110c5204125a9307e5f56578cd7a452cb32a4618f4a883f37eb8faa4c070"}
Oct 30 07:45:04.276106 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:04.276102    6224 scope.go:122] "RemoveContainer" containerID="6a6d032dfd93909eb06c4710b589bfe21041fe987378a419bf37aa9c2dab2021"
Oct 30 07:45:04.276271 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:04.276252    6224 scope.go:122] "RemoveContainer" containerID="6e2d110c5204125a9307e5f56578cd7a452cb32a4618f4a883f37eb8faa4c070"
Oct 30 07:45:04.276559 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:45:04.276534    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889105-pkg77_odh-ai-gateway-infra(abc98592-b397-43d3-8169-91d59a211e6d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889105-pkg77" podUID="abc98592-b397-43d3-8169-91d59a211e6d"
Oct 30 07:45:04.278234 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:04.278215    6224 generic.go:356] "Generic (PLEG): container finished" podID="afde60aa-712f-46da-a245-9df1d7e8326f" containerID="eced858fda9ba0a81bb9c554b767b67b6b466bb046fe76886d7c2e9c38cb1b83" exitCode=60
Oct 30 07:45:04.278345 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:04.278293    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889105-6dfr9" event={"ID":"afde60aa-712f-46da-a245-9df1d7e8326f","Type":"ContainerDied","Data":"eced858fda9ba0a81bb9c554b767b67b6b466bb046fe76886d7c2e9c38cb1b83"}
Oct 30 07:45:04.278468 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:04.278454    6224 scope.go:122] "RemoveContainer" containerID="eced858fda9ba0a81bb9c554b767b67b6b466bb046fe76886d7c2e9c38cb1b83"
Oct 30 07:45:04.278719 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:45:04.278698    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889105-6dfr9_odh-ai-gateway-infra(afde60aa-712f-46da-a245-9df1d7e8326f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889105-6dfr9" podUID="afde60aa-712f-46da-a245-9df1d7e8326f"
Oct 30 07:45:04.280302 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:04.280222    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f5081-29889105-4t64q" event={"ID":"1641d01b-b971-4bf0-b3af-e1523a410dc5","Type":"ContainerStarted","Data":"9851aa0c35757edc344f70dec41a892073ab3ee7e8cc16f1fcadea22d43ba004"}
Oct 30 07:45:04.283202 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:04.283170    6224 generic.go:356] "Generic (PLEG): container finished" podID="87432693-c1f2-42bb-9a66-7c148d63f2fa" containerID="c93716d798fa966e6a75877e924c34da70093e39f12ba58b7e2f55fd4ec60c8f" exitCode=60
Oct 30 07:45:04.283202 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:04.283193    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8e7948-29889105-m6vwb" event={"ID":"87432693-c1f2-42bb-9a66-7c148d63f2fa","Type":"ContainerDied","Data":"c93716d798fa966e6a75877e924c34da70093e39f12ba58b7e2f55fd4ec60c8f"}
Oct 30 07:45:04.283501 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:04.283483    6224 scope.go:122] "RemoveContainer" containerID="c93716d798fa966e6a75877e924c34da70093e39f12ba58b7e2f55fd4ec60c8f"
Oct 30 07:45:04.283736 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:45:04.283718    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-8e7948-29889105-m6vwb_odh-ai-gateway-infra(87432693-c1f2-42bb-9a66-7c148d63f2fa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8e7948-29889105-m6vwb" podUID="87432693-c1f2-42bb-9a66-7c148d63f2fa"
Oct 30 07:45:04.285548 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:04.285524    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889105-gmwl4" event={"ID":"f487dd5e-0777-4bfb-98c9-81b2602e63a3","Type":"ContainerStarted","Data":"d16feff81d4643132356256d389644b97d2a52ae97405769dba259455c334c58"}
Oct 30 07:45:04.287703 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:04.287682    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889105-fmdqz" event={"ID":"b1966a9a-09be-440c-a430-8c79b8c89684","Type":"ContainerStarted","Data":"ef0a00b4bfee677009d2e6fb500622fc712347bc15f3fe87229dfcea4b992dcf"}
Oct 30 07:45:04.296022 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:04.295905    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8fef73-29889105-xmpv9" event={"ID":"49a8dd13-feae-465b-a031-aea87e1d8a59","Type":"ContainerStarted","Data":"d88cd01b6952181a7a0772c8daf86e4e6439a245ab3a2483a0e899c581965c5c"}
Oct 30 07:45:04.296097 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:04.296025    6224 scope.go:122] "RemoveContainer" containerID="61362905f5fe85dbb211de98a50b0f0ade19492401a48cf5017fa9fa1cee3af2"
Oct 30 07:45:04.311601 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:04.311576    6224 scope.go:122] "RemoveContainer" containerID="393e6611b4459fd58feee08d599c0f1edbdea38796274dc2b277f127222a3304"
Oct 30 07:45:04.501580 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:04.501532    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889105-gmwl4" podStartSLOduration=4.501518005 podStartE2EDuration="4.501518005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:45:04.499749078 +0000 UTC m=+2863.109156048" watchObservedRunningTime="2026-10-30 07:45:04.501518005 +0000 UTC m=+2863.110924975"
Oct 30 07:45:04.639328 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:04.639101    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8fef73-29889105-xmpv9" podStartSLOduration=4.639083125 podStartE2EDuration="4.639083125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:45:04.632614927 +0000 UTC m=+2863.242021896" watchObservedRunningTime="2026-10-30 07:45:04.639083125 +0000 UTC m=+2863.248490096"
Oct 30 07:45:05.305986 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:05.305952    6224 generic.go:356] "Generic (PLEG): container finished" podID="b1966a9a-09be-440c-a430-8c79b8c89684" containerID="ef0a00b4bfee677009d2e6fb500622fc712347bc15f3fe87229dfcea4b992dcf" exitCode=60
Oct 30 07:45:05.306435 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:05.306041    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889105-fmdqz" event={"ID":"b1966a9a-09be-440c-a430-8c79b8c89684","Type":"ContainerDied","Data":"ef0a00b4bfee677009d2e6fb500622fc712347bc15f3fe87229dfcea4b992dcf"}
Oct 30 07:45:05.306435 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:05.306149    6224 scope.go:122] "RemoveContainer" containerID="3750f8b9203ea9f73bbb0b0ac74f786d520b6018bfd624f078a2aaa21ea69469"
Oct 30 07:45:05.306435 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:05.306360    6224 scope.go:122] "RemoveContainer" containerID="ef0a00b4bfee677009d2e6fb500622fc712347bc15f3fe87229dfcea4b992dcf"
Oct 30 07:45:05.306781 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:45:05.306748    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-137122-29889105-fmdqz_odh-ai-gateway-infra(b1966a9a-09be-440c-a430-8c79b8c89684)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889105-fmdqz" podUID="b1966a9a-09be-440c-a430-8c79b8c89684"
Oct 30 07:45:05.308179 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:05.308157    6224 generic.go:356] "Generic (PLEG): container finished" podID="49a8dd13-feae-465b-a031-aea87e1d8a59" containerID="d88cd01b6952181a7a0772c8daf86e4e6439a245ab3a2483a0e899c581965c5c" exitCode=60
Oct 30 07:45:05.308303 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:05.308229    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8fef73-29889105-xmpv9" event={"ID":"49a8dd13-feae-465b-a031-aea87e1d8a59","Type":"ContainerDied","Data":"d88cd01b6952181a7a0772c8daf86e4e6439a245ab3a2483a0e899c581965c5c"}
Oct 30 07:45:05.308573 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:05.308558    6224 scope.go:122] "RemoveContainer" containerID="d88cd01b6952181a7a0772c8daf86e4e6439a245ab3a2483a0e899c581965c5c"
Oct 30 07:45:05.308995 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:45:05.308972    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-8fef73-29889105-xmpv9_odh-ai-gateway-infra(49a8dd13-feae-465b-a031-aea87e1d8a59)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8fef73-29889105-xmpv9" podUID="49a8dd13-feae-465b-a031-aea87e1d8a59"
Oct 30 07:45:05.310255 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:05.310234    6224 generic.go:356] "Generic (PLEG): container finished" podID="7b63a195-1c69-4fdd-ba65-07fdc7996d37" containerID="eeef350ba9884ce9faac4e3652201f7778d6e0da186a02e65b66dcc604699759" exitCode=60
Oct 30 07:45:05.310341 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:05.310299    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ad434e-29889105-fljfl" event={"ID":"7b63a195-1c69-4fdd-ba65-07fdc7996d37","Type":"ContainerDied","Data":"eeef350ba9884ce9faac4e3652201f7778d6e0da186a02e65b66dcc604699759"}
Oct 30 07:45:05.310618 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:05.310603    6224 scope.go:122] "RemoveContainer" containerID="eeef350ba9884ce9faac4e3652201f7778d6e0da186a02e65b66dcc604699759"
Oct 30 07:45:05.311003 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:45:05.310984    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-ad434e-29889105-fljfl_odh-ai-gateway-infra(7b63a195-1c69-4fdd-ba65-07fdc7996d37)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ad434e-29889105-fljfl" podUID="7b63a195-1c69-4fdd-ba65-07fdc7996d37"
Oct 30 07:45:05.312403 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:05.312378    6224 generic.go:356] "Generic (PLEG): container finished" podID="08398783-5da2-4d6b-b0d0-4d141510de4b" containerID="71d844727a7a2b9e6801908c13a54f62854b8f1d56fb590af58d17a07340d2b4" exitCode=60
Oct 30 07:45:05.312491 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:05.312455    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889105-4x5vf" event={"ID":"08398783-5da2-4d6b-b0d0-4d141510de4b","Type":"ContainerDied","Data":"71d844727a7a2b9e6801908c13a54f62854b8f1d56fb590af58d17a07340d2b4"}
Oct 30 07:45:05.312711 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:05.312694    6224 scope.go:122] "RemoveContainer" containerID="71d844727a7a2b9e6801908c13a54f62854b8f1d56fb590af58d17a07340d2b4"
Oct 30 07:45:05.313078 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:45:05.313052    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889105-4x5vf_odh-ai-gateway-infra(08398783-5da2-4d6b-b0d0-4d141510de4b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889105-4x5vf" podUID="08398783-5da2-4d6b-b0d0-4d141510de4b"
Oct 30 07:45:05.314411 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:05.314393    6224 scope.go:122] "RemoveContainer" containerID="6e2d110c5204125a9307e5f56578cd7a452cb32a4618f4a883f37eb8faa4c070"
Oct 30 07:45:05.314711 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:45:05.314688    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889105-pkg77_odh-ai-gateway-infra(abc98592-b397-43d3-8169-91d59a211e6d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889105-pkg77" podUID="abc98592-b397-43d3-8169-91d59a211e6d"
Oct 30 07:45:05.316129 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:05.316111    6224 scope.go:122] "RemoveContainer" containerID="eced858fda9ba0a81bb9c554b767b67b6b466bb046fe76886d7c2e9c38cb1b83"
Oct 30 07:45:05.316388 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:45:05.316362    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889105-6dfr9_odh-ai-gateway-infra(afde60aa-712f-46da-a245-9df1d7e8326f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889105-6dfr9" podUID="afde60aa-712f-46da-a245-9df1d7e8326f"
Oct 30 07:45:05.318034 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:05.318014    6224 generic.go:356] "Generic (PLEG): container finished" podID="1641d01b-b971-4bf0-b3af-e1523a410dc5" containerID="9851aa0c35757edc344f70dec41a892073ab3ee7e8cc16f1fcadea22d43ba004" exitCode=60
Oct 30 07:45:05.318137 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:05.318101    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f5081-29889105-4t64q" event={"ID":"1641d01b-b971-4bf0-b3af-e1523a410dc5","Type":"ContainerDied","Data":"9851aa0c35757edc344f70dec41a892073ab3ee7e8cc16f1fcadea22d43ba004"}
Oct 30 07:45:05.318360 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:05.318336    6224 scope.go:122] "RemoveContainer" containerID="9851aa0c35757edc344f70dec41a892073ab3ee7e8cc16f1fcadea22d43ba004"
Oct 30 07:45:05.318668 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:45:05.318642    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-0f5081-29889105-4t64q_odh-ai-gateway-infra(1641d01b-b971-4bf0-b3af-e1523a410dc5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f5081-29889105-4t64q" podUID="1641d01b-b971-4bf0-b3af-e1523a410dc5"
Oct 30 07:45:05.319984 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:05.319969    6224 scope.go:122] "RemoveContainer" containerID="c93716d798fa966e6a75877e924c34da70093e39f12ba58b7e2f55fd4ec60c8f"
Oct 30 07:45:05.320237 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:45:05.320219    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-8e7948-29889105-m6vwb_odh-ai-gateway-infra(87432693-c1f2-42bb-9a66-7c148d63f2fa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8e7948-29889105-m6vwb" podUID="87432693-c1f2-42bb-9a66-7c148d63f2fa"
Oct 30 07:45:05.321765 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:05.321707    6224 generic.go:356] "Generic (PLEG): container finished" podID="f487dd5e-0777-4bfb-98c9-81b2602e63a3" containerID="d16feff81d4643132356256d389644b97d2a52ae97405769dba259455c334c58" exitCode=60
Oct 30 07:45:05.321765 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:05.321759    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889105-gmwl4" event={"ID":"f487dd5e-0777-4bfb-98c9-81b2602e63a3","Type":"ContainerDied","Data":"d16feff81d4643132356256d389644b97d2a52ae97405769dba259455c334c58"}
Oct 30 07:45:05.321907 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:05.321792    6224 scope.go:122] "RemoveContainer" containerID="afc85b428f5ba608cf74790ef5315b750f541a795c11e4f1509a9eb3ea8d4dc6"
Oct 30 07:45:05.322029 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:05.322015    6224 scope.go:122] "RemoveContainer" containerID="d16feff81d4643132356256d389644b97d2a52ae97405769dba259455c334c58"
Oct 30 07:45:05.322262 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:45:05.322242    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889105-gmwl4_odh-ai-gateway-infra(f487dd5e-0777-4bfb-98c9-81b2602e63a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889105-gmwl4" podUID="f487dd5e-0777-4bfb-98c9-81b2602e63a3"
Oct 30 07:45:05.336587 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:05.336566    6224 scope.go:122] "RemoveContainer" containerID="6d75ed5da272510c436400896dde321df92ebe45aff1ef19c4af31688ee16d11"
Oct 30 07:45:05.351193 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:05.351170    6224 scope.go:122] "RemoveContainer" containerID="d63778381dcd0de2ffdd39b0f2b1b2f450aaad2fc36cff4cfafe7fb0dbafad15"
Oct 30 07:45:05.366959 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:05.366939    6224 scope.go:122] "RemoveContainer" containerID="267c2bf7110a17f67a0f301ba624e199bc37cfdcad6ec9f7a948866ebf634d51"
Oct 30 07:45:05.382528 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:05.382505    6224 scope.go:122] "RemoveContainer" containerID="831735b27cdbe870febc1af8a50772486818f1775d081dadc56a07e5c9d33043"
Oct 30 07:45:06.336874 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:06.336843    6224 scope.go:122] "RemoveContainer" containerID="eeef350ba9884ce9faac4e3652201f7778d6e0da186a02e65b66dcc604699759"
Oct 30 07:45:06.337401 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:45:06.337297    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-ad434e-29889105-fljfl_odh-ai-gateway-infra(7b63a195-1c69-4fdd-ba65-07fdc7996d37)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ad434e-29889105-fljfl" podUID="7b63a195-1c69-4fdd-ba65-07fdc7996d37"
Oct 30 07:45:06.338687 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:06.338663    6224 scope.go:122] "RemoveContainer" containerID="71d844727a7a2b9e6801908c13a54f62854b8f1d56fb590af58d17a07340d2b4"
Oct 30 07:45:06.339032 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:45:06.339008    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889105-4x5vf_odh-ai-gateway-infra(08398783-5da2-4d6b-b0d0-4d141510de4b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889105-4x5vf" podUID="08398783-5da2-4d6b-b0d0-4d141510de4b"
Oct 30 07:45:06.340238 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:06.340219    6224 scope.go:122] "RemoveContainer" containerID="9851aa0c35757edc344f70dec41a892073ab3ee7e8cc16f1fcadea22d43ba004"
Oct 30 07:45:06.340523 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:45:06.340505    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-0f5081-29889105-4t64q_odh-ai-gateway-infra(1641d01b-b971-4bf0-b3af-e1523a410dc5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f5081-29889105-4t64q" podUID="1641d01b-b971-4bf0-b3af-e1523a410dc5"
Oct 30 07:45:06.341772 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:06.341755    6224 scope.go:122] "RemoveContainer" containerID="d16feff81d4643132356256d389644b97d2a52ae97405769dba259455c334c58"
Oct 30 07:45:06.342060 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:45:06.342042    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889105-gmwl4_odh-ai-gateway-infra(f487dd5e-0777-4bfb-98c9-81b2602e63a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889105-gmwl4" podUID="f487dd5e-0777-4bfb-98c9-81b2602e63a3"
Oct 30 07:45:06.343323 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:06.343304    6224 scope.go:122] "RemoveContainer" containerID="ef0a00b4bfee677009d2e6fb500622fc712347bc15f3fe87229dfcea4b992dcf"
Oct 30 07:45:06.343524 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:45:06.343503    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-137122-29889105-fmdqz_odh-ai-gateway-infra(b1966a9a-09be-440c-a430-8c79b8c89684)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889105-fmdqz" podUID="b1966a9a-09be-440c-a430-8c79b8c89684"
Oct 30 07:45:06.344787 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:06.344771    6224 scope.go:122] "RemoveContainer" containerID="d88cd01b6952181a7a0772c8daf86e4e6439a245ab3a2483a0e899c581965c5c"
Oct 30 07:45:06.344971 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:45:06.344954    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-8fef73-29889105-xmpv9_odh-ai-gateway-infra(49a8dd13-feae-465b-a031-aea87e1d8a59)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8fef73-29889105-xmpv9" podUID="49a8dd13-feae-465b-a031-aea87e1d8a59"
Oct 30 07:45:12.720389 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:12.720352    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-4ab37e/test-model-4ab37e-kserve-75448f6fc5-ln8dd"]
Oct 30 07:45:12.728095 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:12.728068    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4ab37e/test-model-4ab37e-kserve-75448f6fc5-ln8dd"
Oct 30 07:45:12.734046 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:12.734003    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf2c853f-e673-498f-9135-2e963890f12e-tls-certs\") pod \"test-model-4ab37e-kserve-75448f6fc5-ln8dd\" (UID: \"bf2c853f-e673-498f-9135-2e963890f12e\") " pod="ai-tenant-e2e-inf-b-4ab37e/test-model-4ab37e-kserve-75448f6fc5-ln8dd"
Oct 30 07:45:12.734189 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:12.734090    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf2c853f-e673-498f-9135-2e963890f12e-dshm\") pod \"test-model-4ab37e-kserve-75448f6fc5-ln8dd\" (UID: \"bf2c853f-e673-498f-9135-2e963890f12e\") " pod="ai-tenant-e2e-inf-b-4ab37e/test-model-4ab37e-kserve-75448f6fc5-ln8dd"
Oct 30 07:45:12.734189 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:12.734140    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf2c853f-e673-498f-9135-2e963890f12e-home\") pod \"test-model-4ab37e-kserve-75448f6fc5-ln8dd\" (UID: \"bf2c853f-e673-498f-9135-2e963890f12e\") " pod="ai-tenant-e2e-inf-b-4ab37e/test-model-4ab37e-kserve-75448f6fc5-ln8dd"
Oct 30 07:45:12.734315 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:12.734275    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcv5x\" (UniqueName: \"kubernetes.io/projected/bf2c853f-e673-498f-9135-2e963890f12e-kube-api-access-hcv5x\") pod \"test-model-4ab37e-kserve-75448f6fc5-ln8dd\" (UID: \"bf2c853f-e673-498f-9135-2e963890f12e\") " pod="ai-tenant-e2e-inf-b-4ab37e/test-model-4ab37e-kserve-75448f6fc5-ln8dd"
Oct 30 07:45:12.734448 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:12.734418    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf2c853f-e673-498f-9135-2e963890f12e-tmp-dir\") pod \"test-model-4ab37e-kserve-75448f6fc5-ln8dd\" (UID: \"bf2c853f-e673-498f-9135-2e963890f12e\") " pod="ai-tenant-e2e-inf-b-4ab37e/test-model-4ab37e-kserve-75448f6fc5-ln8dd"
Oct 30 07:45:12.734569 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:12.734541    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf2c853f-e673-498f-9135-2e963890f12e-model-cache\") pod \"test-model-4ab37e-kserve-75448f6fc5-ln8dd\" (UID: \"bf2c853f-e673-498f-9135-2e963890f12e\") " pod="ai-tenant-e2e-inf-b-4ab37e/test-model-4ab37e-kserve-75448f6fc5-ln8dd"
Oct 30 07:45:12.752274 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:12.752246    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4ab37e/test-model-4ab37e-kserve-75448f6fc5-ln8dd"]
Oct 30 07:45:12.755381 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:12.755342    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4ab37e\"/\"default-dockercfg-dt6w4\""
Oct 30 07:45:12.755575 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:12.755555    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4ab37e\"/\"test-model-4ab37e-kserve-self-signed-certs\""
Oct 30 07:45:12.758925 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:12.758905    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4ab37e\"/\"openshift-service-ca.crt\""
Oct 30 07:45:12.759038 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:12.758955    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4ab37e\"/\"kube-root-ca.crt\""
Oct 30 07:45:12.836276 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:12.836237    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf2c853f-e673-498f-9135-2e963890f12e-tls-certs\") pod \"test-model-4ab37e-kserve-75448f6fc5-ln8dd\" (UID: \"bf2c853f-e673-498f-9135-2e963890f12e\") " pod="ai-tenant-e2e-inf-b-4ab37e/test-model-4ab37e-kserve-75448f6fc5-ln8dd"
Oct 30 07:45:12.836428 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:12.836283    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf2c853f-e673-498f-9135-2e963890f12e-dshm\") pod \"test-model-4ab37e-kserve-75448f6fc5-ln8dd\" (UID: \"bf2c853f-e673-498f-9135-2e963890f12e\") " pod="ai-tenant-e2e-inf-b-4ab37e/test-model-4ab37e-kserve-75448f6fc5-ln8dd"
Oct 30 07:45:12.836428 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:12.836317    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf2c853f-e673-498f-9135-2e963890f12e-home\") pod \"test-model-4ab37e-kserve-75448f6fc5-ln8dd\" (UID: \"bf2c853f-e673-498f-9135-2e963890f12e\") " pod="ai-tenant-e2e-inf-b-4ab37e/test-model-4ab37e-kserve-75448f6fc5-ln8dd"
Oct 30 07:45:12.836428 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:12.836400    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcv5x\" (UniqueName: \"kubernetes.io/projected/bf2c853f-e673-498f-9135-2e963890f12e-kube-api-access-hcv5x\") pod \"test-model-4ab37e-kserve-75448f6fc5-ln8dd\" (UID: \"bf2c853f-e673-498f-9135-2e963890f12e\") " pod="ai-tenant-e2e-inf-b-4ab37e/test-model-4ab37e-kserve-75448f6fc5-ln8dd"
Oct 30 07:45:12.836624 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:12.836475    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf2c853f-e673-498f-9135-2e963890f12e-tmp-dir\") pod \"test-model-4ab37e-kserve-75448f6fc5-ln8dd\" (UID: \"bf2c853f-e673-498f-9135-2e963890f12e\") " pod="ai-tenant-e2e-inf-b-4ab37e/test-model-4ab37e-kserve-75448f6fc5-ln8dd"
Oct 30 07:45:12.836624 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:12.836524    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf2c853f-e673-498f-9135-2e963890f12e-model-cache\") pod \"test-model-4ab37e-kserve-75448f6fc5-ln8dd\" (UID: \"bf2c853f-e673-498f-9135-2e963890f12e\") " pod="ai-tenant-e2e-inf-b-4ab37e/test-model-4ab37e-kserve-75448f6fc5-ln8dd"
Oct 30 07:45:12.836862 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:12.836839    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf2c853f-e673-498f-9135-2e963890f12e-home\") pod \"test-model-4ab37e-kserve-75448f6fc5-ln8dd\" (UID: \"bf2c853f-e673-498f-9135-2e963890f12e\") " pod="ai-tenant-e2e-inf-b-4ab37e/test-model-4ab37e-kserve-75448f6fc5-ln8dd"
Oct 30 07:45:12.836978 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:12.836883    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf2c853f-e673-498f-9135-2e963890f12e-tmp-dir\") pod \"test-model-4ab37e-kserve-75448f6fc5-ln8dd\" (UID: \"bf2c853f-e673-498f-9135-2e963890f12e\") " pod="ai-tenant-e2e-inf-b-4ab37e/test-model-4ab37e-kserve-75448f6fc5-ln8dd"
Oct 30 07:45:12.836978 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:12.836926    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf2c853f-e673-498f-9135-2e963890f12e-model-cache\") pod \"test-model-4ab37e-kserve-75448f6fc5-ln8dd\" (UID: \"bf2c853f-e673-498f-9135-2e963890f12e\") " pod="ai-tenant-e2e-inf-b-4ab37e/test-model-4ab37e-kserve-75448f6fc5-ln8dd"
Oct 30 07:45:12.838978 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:12.838953    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf2c853f-e673-498f-9135-2e963890f12e-tls-certs\") pod \"test-model-4ab37e-kserve-75448f6fc5-ln8dd\" (UID: \"bf2c853f-e673-498f-9135-2e963890f12e\") " pod="ai-tenant-e2e-inf-b-4ab37e/test-model-4ab37e-kserve-75448f6fc5-ln8dd"
Oct 30 07:45:12.839079 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:12.839026    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf2c853f-e673-498f-9135-2e963890f12e-dshm\") pod \"test-model-4ab37e-kserve-75448f6fc5-ln8dd\" (UID: \"bf2c853f-e673-498f-9135-2e963890f12e\") " pod="ai-tenant-e2e-inf-b-4ab37e/test-model-4ab37e-kserve-75448f6fc5-ln8dd"
Oct 30 07:45:12.865230 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:12.865139    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcv5x\" (UniqueName: \"kubernetes.io/projected/bf2c853f-e673-498f-9135-2e963890f12e-kube-api-access-hcv5x\") pod \"test-model-4ab37e-kserve-75448f6fc5-ln8dd\" (UID: \"bf2c853f-e673-498f-9135-2e963890f12e\") " pod="ai-tenant-e2e-inf-b-4ab37e/test-model-4ab37e-kserve-75448f6fc5-ln8dd"
Oct 30 07:45:13.053840 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:13.051018    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4ab37e/test-model-4ab37e-kserve-75448f6fc5-ln8dd"
Oct 30 07:45:13.373764 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:45:13.373732    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf2c853f_e673_498f_9135_2e963890f12e.slice/crio-e60f3f0b22485a4777a8d4461b7113c0d8a2455d72711dae7aec68fec812ca0d WatchSource:0}: Error finding container e60f3f0b22485a4777a8d4461b7113c0d8a2455d72711dae7aec68fec812ca0d: Status 404 returned error can't find the container with id e60f3f0b22485a4777a8d4461b7113c0d8a2455d72711dae7aec68fec812ca0d
Oct 30 07:45:13.375263 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:13.375234    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4ab37e/test-model-4ab37e-kserve-75448f6fc5-ln8dd"]
Oct 30 07:45:13.402333 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:13.402306    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4ab37e/test-model-4ab37e-kserve-75448f6fc5-ln8dd" event={"ID":"bf2c853f-e673-498f-9135-2e963890f12e","Type":"ContainerStarted","Data":"e60f3f0b22485a4777a8d4461b7113c0d8a2455d72711dae7aec68fec812ca0d"}
Oct 30 07:45:14.411975 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:14.411932    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4ab37e/test-model-4ab37e-kserve-75448f6fc5-ln8dd" event={"ID":"bf2c853f-e673-498f-9135-2e963890f12e","Type":"ContainerStarted","Data":"f01d5ab0da57da3f5e565c274402192bb3e3011b203c6537a3e137d14763a726"}
Oct 30 07:45:14.462392 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:14.462343    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-4ab37e/test-model-4ab37e-kserve-75448f6fc5-ln8dd" podStartSLOduration=2.4623287019999998 podStartE2EDuration="2.462328702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:45:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:45:14.457246106 +0000 UTC m=+2873.066653075" watchObservedRunningTime="2026-10-30 07:45:14.462328702 +0000 UTC m=+2873.071735671"
Oct 30 07:45:14.496602 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:14.496570    6224 scope.go:122] "RemoveContainer" containerID="c93716d798fa966e6a75877e924c34da70093e39f12ba58b7e2f55fd4ec60c8f"
Oct 30 07:45:15.054492 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:15.054442    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-4ab37e/test-model-4ab37e-kserve-75448f6fc5-ln8dd"
Oct 30 07:45:15.421495 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:15.421413    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8e7948-29889105-m6vwb" event={"ID":"87432693-c1f2-42bb-9a66-7c148d63f2fa","Type":"ContainerStarted","Data":"42008f4c6578dbae1453ce0ce045ffcb269ac0cc8b8983cc9c02bac1e17703e9"}
Oct 30 07:45:15.496278 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:15.496254    6224 scope.go:122] "RemoveContainer" containerID="71d844727a7a2b9e6801908c13a54f62854b8f1d56fb590af58d17a07340d2b4"
Oct 30 07:45:15.496543 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:15.496518    6224 scope.go:122] "RemoveContainer" containerID="eced858fda9ba0a81bb9c554b767b67b6b466bb046fe76886d7c2e9c38cb1b83"
Oct 30 07:45:15.498388 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:15.498145    6224 scope.go:122] "RemoveContainer" containerID="eeef350ba9884ce9faac4e3652201f7778d6e0da186a02e65b66dcc604699759"
Oct 30 07:45:16.433962 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:16.433923    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889105-4x5vf" event={"ID":"08398783-5da2-4d6b-b0d0-4d141510de4b","Type":"ContainerStarted","Data":"ea0ac896f16098b65d72d3ac821f735754c2a576f5d9487b1e6af868c5d710ba"}
Oct 30 07:45:16.436364 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:16.436332    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889105-6dfr9" event={"ID":"afde60aa-712f-46da-a245-9df1d7e8326f","Type":"ContainerStarted","Data":"b170fdd7d3059ed8717e8be9c5897da53c57f1227ce195d9de76467bf64b77d7"}
Oct 30 07:45:16.438767 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:16.438738    6224 generic.go:356] "Generic (PLEG): container finished" podID="87432693-c1f2-42bb-9a66-7c148d63f2fa" containerID="42008f4c6578dbae1453ce0ce045ffcb269ac0cc8b8983cc9c02bac1e17703e9" exitCode=60
Oct 30 07:45:16.439023 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:16.439005    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8e7948-29889105-m6vwb" event={"ID":"87432693-c1f2-42bb-9a66-7c148d63f2fa","Type":"ContainerDied","Data":"42008f4c6578dbae1453ce0ce045ffcb269ac0cc8b8983cc9c02bac1e17703e9"}
Oct 30 07:45:16.439068 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:16.439056    6224 scope.go:122] "RemoveContainer" containerID="c93716d798fa966e6a75877e924c34da70093e39f12ba58b7e2f55fd4ec60c8f"
Oct 30 07:45:16.439472 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:16.439454    6224 scope.go:122] "RemoveContainer" containerID="42008f4c6578dbae1453ce0ce045ffcb269ac0cc8b8983cc9c02bac1e17703e9"
Oct 30 07:45:16.439799 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:45:16.439777    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-8e7948-29889105-m6vwb_odh-ai-gateway-infra(87432693-c1f2-42bb-9a66-7c148d63f2fa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8e7948-29889105-m6vwb" podUID="87432693-c1f2-42bb-9a66-7c148d63f2fa"
Oct 30 07:45:16.441540 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:16.441513    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ad434e-29889105-fljfl" event={"ID":"7b63a195-1c69-4fdd-ba65-07fdc7996d37","Type":"ContainerStarted","Data":"f31e2a901f3c29868e24e269e396ecad678964f69054d02429d7e46cd9da91d4"}
Oct 30 07:45:16.478974 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:16.478940    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8e7948-29889105-m6vwb"]
Oct 30 07:45:16.495790 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:16.495764    6224 scope.go:122] "RemoveContainer" containerID="9851aa0c35757edc344f70dec41a892073ab3ee7e8cc16f1fcadea22d43ba004"
Oct 30 07:45:16.496520 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:16.496438    6224 scope.go:122] "RemoveContainer" containerID="d16feff81d4643132356256d389644b97d2a52ae97405769dba259455c334c58"
Oct 30 07:45:17.452598 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:17.452563    6224 generic.go:356] "Generic (PLEG): container finished" podID="7b63a195-1c69-4fdd-ba65-07fdc7996d37" containerID="f31e2a901f3c29868e24e269e396ecad678964f69054d02429d7e46cd9da91d4" exitCode=60
Oct 30 07:45:17.453096 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:17.452634    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ad434e-29889105-fljfl" event={"ID":"7b63a195-1c69-4fdd-ba65-07fdc7996d37","Type":"ContainerDied","Data":"f31e2a901f3c29868e24e269e396ecad678964f69054d02429d7e46cd9da91d4"}
Oct 30 07:45:17.453096 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:17.452691    6224 scope.go:122] "RemoveContainer" containerID="eeef350ba9884ce9faac4e3652201f7778d6e0da186a02e65b66dcc604699759"
Oct 30 07:45:17.453218 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:17.453151    6224 scope.go:122] "RemoveContainer" containerID="f31e2a901f3c29868e24e269e396ecad678964f69054d02429d7e46cd9da91d4"
Oct 30 07:45:17.453534 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:45:17.453512    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-ad434e-29889105-fljfl_odh-ai-gateway-infra(7b63a195-1c69-4fdd-ba65-07fdc7996d37)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ad434e-29889105-fljfl" podUID="7b63a195-1c69-4fdd-ba65-07fdc7996d37"
Oct 30 07:45:17.455032 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:17.455011    6224 generic.go:356] "Generic (PLEG): container finished" podID="08398783-5da2-4d6b-b0d0-4d141510de4b" containerID="ea0ac896f16098b65d72d3ac821f735754c2a576f5d9487b1e6af868c5d710ba" exitCode=60
Oct 30 07:45:17.455138 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:17.455086    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889105-4x5vf" event={"ID":"08398783-5da2-4d6b-b0d0-4d141510de4b","Type":"ContainerDied","Data":"ea0ac896f16098b65d72d3ac821f735754c2a576f5d9487b1e6af868c5d710ba"}
Oct 30 07:45:17.455460 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:17.455444    6224 scope.go:122] "RemoveContainer" containerID="ea0ac896f16098b65d72d3ac821f735754c2a576f5d9487b1e6af868c5d710ba"
Oct 30 07:45:17.455768 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:45:17.455746    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889105-4x5vf_odh-ai-gateway-infra(08398783-5da2-4d6b-b0d0-4d141510de4b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889105-4x5vf" podUID="08398783-5da2-4d6b-b0d0-4d141510de4b"
Oct 30 07:45:17.457010 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:17.456990    6224 generic.go:356] "Generic (PLEG): container finished" podID="afde60aa-712f-46da-a245-9df1d7e8326f" containerID="b170fdd7d3059ed8717e8be9c5897da53c57f1227ce195d9de76467bf64b77d7" exitCode=60
Oct 30 07:45:17.457104 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:17.457057    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889105-6dfr9" event={"ID":"afde60aa-712f-46da-a245-9df1d7e8326f","Type":"ContainerDied","Data":"b170fdd7d3059ed8717e8be9c5897da53c57f1227ce195d9de76467bf64b77d7"}
Oct 30 07:45:17.457413 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:17.457395    6224 scope.go:122] "RemoveContainer" containerID="b170fdd7d3059ed8717e8be9c5897da53c57f1227ce195d9de76467bf64b77d7"
Oct 30 07:45:17.457737 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:45:17.457714    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889105-6dfr9_odh-ai-gateway-infra(afde60aa-712f-46da-a245-9df1d7e8326f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889105-6dfr9" podUID="afde60aa-712f-46da-a245-9df1d7e8326f"
Oct 30 07:45:17.459430 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:17.459404    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f5081-29889105-4t64q" event={"ID":"1641d01b-b971-4bf0-b3af-e1523a410dc5","Type":"ContainerStarted","Data":"7771d658d8f7f3291648a87f3873370eb00298c4fe2d2a1d69d67a6a8a8f0125"}
Oct 30 07:45:17.463585 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:17.463556    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889105-gmwl4" event={"ID":"f487dd5e-0777-4bfb-98c9-81b2602e63a3","Type":"ContainerStarted","Data":"1afc095722ab92af79f7df0933e5846cab77c32f1b2a75bb449c4786e49966e3"}
Oct 30 07:45:17.468376 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:17.468341    6224 scope.go:122] "RemoveContainer" containerID="71d844727a7a2b9e6801908c13a54f62854b8f1d56fb590af58d17a07340d2b4"
Oct 30 07:45:17.491928 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:17.491894    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889105-4x5vf"]
Oct 30 07:45:17.508182 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:17.507969    6224 scope.go:122] "RemoveContainer" containerID="eced858fda9ba0a81bb9c554b767b67b6b466bb046fe76886d7c2e9c38cb1b83"
Oct 30 07:45:17.510282 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:17.509931    6224 scope.go:122] "RemoveContainer" containerID="d88cd01b6952181a7a0772c8daf86e4e6439a245ab3a2483a0e899c581965c5c"
Oct 30 07:45:17.510282 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:17.509980    6224 scope.go:122] "RemoveContainer" containerID="ef0a00b4bfee677009d2e6fb500622fc712347bc15f3fe87229dfcea4b992dcf"
Oct 30 07:45:17.510883 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:17.510830    6224 scope.go:122] "RemoveContainer" containerID="6e2d110c5204125a9307e5f56578cd7a452cb32a4618f4a883f37eb8faa4c070"
Oct 30 07:45:17.609470 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:17.609370    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889105-6dfr9"]
Oct 30 07:45:17.634235 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:17.634205    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ad434e-29889105-fljfl"]
Oct 30 07:45:17.725148 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:17.725117    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8e7948-29889105-m6vwb"
Oct 30 07:45:17.810851 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:17.810782    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87432693-c1f2-42bb-9a66-7c148d63f2fa-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87432693-c1f2-42bb-9a66-7c148d63f2fa-maas-api-service-ca\") pod \"87432693-c1f2-42bb-9a66-7c148d63f2fa\" (UID: \"87432693-c1f2-42bb-9a66-7c148d63f2fa\") "
Oct 30 07:45:17.811007 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:17.810943    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87432693-c1f2-42bb-9a66-7c148d63f2fa-kube-api-access-ph9nd\" (UniqueName: \"kubernetes.io/projected/87432693-c1f2-42bb-9a66-7c148d63f2fa-kube-api-access-ph9nd\") pod \"87432693-c1f2-42bb-9a66-7c148d63f2fa\" (UID: \"87432693-c1f2-42bb-9a66-7c148d63f2fa\") "
Oct 30 07:45:17.811200 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:17.811177    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87432693-c1f2-42bb-9a66-7c148d63f2fa-maas-api-service-ca" pod "87432693-c1f2-42bb-9a66-7c148d63f2fa" (UID: "87432693-c1f2-42bb-9a66-7c148d63f2fa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:45:17.811911 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:17.811567    6224 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87432693-c1f2-42bb-9a66-7c148d63f2fa-maas-api-service-ca\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:45:17.864144 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:17.864110    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87432693-c1f2-42bb-9a66-7c148d63f2fa-kube-api-access-ph9nd" pod "87432693-c1f2-42bb-9a66-7c148d63f2fa" (UID: "87432693-c1f2-42bb-9a66-7c148d63f2fa"). InnerVolumeSpecName "kube-api-access-ph9nd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:45:17.912522 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:17.912494    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph9nd\" (UniqueName: \"kubernetes.io/projected/87432693-c1f2-42bb-9a66-7c148d63f2fa-kube-api-access-ph9nd\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:45:18.475014 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:18.474973    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8e7948-29889105-m6vwb" event={"ID":"87432693-c1f2-42bb-9a66-7c148d63f2fa","Type":"ContainerDied","Data":"47a6a9df9e78a6bca65e139cfe3bafc1f0d155c91767c32c2beabc98e809842b"}
Oct 30 07:45:18.475606 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:18.475046    6224 scope.go:122] "RemoveContainer" containerID="42008f4c6578dbae1453ce0ce045ffcb269ac0cc8b8983cc9c02bac1e17703e9"
Oct 30 07:45:18.475606 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:18.474983    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8e7948-29889105-m6vwb"
Oct 30 07:45:18.477339 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:18.477318    6224 generic.go:356] "Generic (PLEG): container finished" podID="f487dd5e-0777-4bfb-98c9-81b2602e63a3" containerID="1afc095722ab92af79f7df0933e5846cab77c32f1b2a75bb449c4786e49966e3" exitCode=60
Oct 30 07:45:18.477474 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:18.477390    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889105-gmwl4" event={"ID":"f487dd5e-0777-4bfb-98c9-81b2602e63a3","Type":"ContainerDied","Data":"1afc095722ab92af79f7df0933e5846cab77c32f1b2a75bb449c4786e49966e3"}
Oct 30 07:45:18.477907 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:18.477884    6224 scope.go:122] "RemoveContainer" containerID="1afc095722ab92af79f7df0933e5846cab77c32f1b2a75bb449c4786e49966e3"
Oct 30 07:45:18.478270 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:45:18.478244    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889105-gmwl4_odh-ai-gateway-infra(f487dd5e-0777-4bfb-98c9-81b2602e63a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889105-gmwl4" podUID="f487dd5e-0777-4bfb-98c9-81b2602e63a3"
Oct 30 07:45:18.479745 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:18.479721    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889105-fmdqz" event={"ID":"b1966a9a-09be-440c-a430-8c79b8c89684","Type":"ContainerStarted","Data":"0b4808f4f0919eb60449e766bd158487d6c56480b46958128690d51104433554"}
Oct 30 07:45:18.481950 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:18.481927    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8fef73-29889105-xmpv9" event={"ID":"49a8dd13-feae-465b-a031-aea87e1d8a59","Type":"ContainerStarted","Data":"be5bc9e49b196b7c78628846dae9ef05101d9e197b68add863294cc3f7f7b58b"}
Oct 30 07:45:18.488128 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:18.488106    6224 scope.go:122] "RemoveContainer" containerID="d16feff81d4643132356256d389644b97d2a52ae97405769dba259455c334c58"
Oct 30 07:45:18.488538 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:18.488517    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889105-pkg77" event={"ID":"abc98592-b397-43d3-8169-91d59a211e6d","Type":"ContainerStarted","Data":"ebdea22fa9cdbae41b7a41e004e52715d95ac827f015d892e922dfaec94483fb"}
Oct 30 07:45:18.492474 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:18.492444    6224 generic.go:356] "Generic (PLEG): container finished" podID="1641d01b-b971-4bf0-b3af-e1523a410dc5" containerID="7771d658d8f7f3291648a87f3873370eb00298c4fe2d2a1d69d67a6a8a8f0125" exitCode=60
Oct 30 07:45:18.492556 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:18.492489    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f5081-29889105-4t64q" event={"ID":"1641d01b-b971-4bf0-b3af-e1523a410dc5","Type":"ContainerDied","Data":"7771d658d8f7f3291648a87f3873370eb00298c4fe2d2a1d69d67a6a8a8f0125"}
Oct 30 07:45:18.492949 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:18.492928    6224 scope.go:122] "RemoveContainer" containerID="7771d658d8f7f3291648a87f3873370eb00298c4fe2d2a1d69d67a6a8a8f0125"
Oct 30 07:45:18.493256 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:45:18.493232    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-0f5081-29889105-4t64q_odh-ai-gateway-infra(1641d01b-b971-4bf0-b3af-e1523a410dc5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f5081-29889105-4t64q" podUID="1641d01b-b971-4bf0-b3af-e1523a410dc5"
Oct 30 07:45:18.503901 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:18.503881    6224 scope.go:122] "RemoveContainer" containerID="9851aa0c35757edc344f70dec41a892073ab3ee7e8cc16f1fcadea22d43ba004"
Oct 30 07:45:18.715710 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:18.715677    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889105-gmwl4"]
Oct 30 07:45:18.746760 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:18.746688    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8e7948-29889105-m6vwb"]
Oct 30 07:45:18.752938 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:18.752912    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8e7948-29889105-m6vwb"]
Oct 30 07:45:18.767383 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:18.767357    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f5081-29889105-4t64q"]
Oct 30 07:45:18.865994 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:18.865966    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ad434e-29889105-fljfl"
Oct 30 07:45:18.871357 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:18.871331    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889105-4x5vf"
Oct 30 07:45:18.876190 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:18.876169    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889105-6dfr9"
Oct 30 07:45:18.929669 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:18.929617    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b63a195-1c69-4fdd-ba65-07fdc7996d37-kube-api-access-p76t4\" (UniqueName: \"kubernetes.io/projected/7b63a195-1c69-4fdd-ba65-07fdc7996d37-kube-api-access-p76t4\") pod \"7b63a195-1c69-4fdd-ba65-07fdc7996d37\" (UID: \"7b63a195-1c69-4fdd-ba65-07fdc7996d37\") "
Oct 30 07:45:18.930218 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:18.929710    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afde60aa-712f-46da-a245-9df1d7e8326f-kube-api-access-kb2jw\" (UniqueName: \"kubernetes.io/projected/afde60aa-712f-46da-a245-9df1d7e8326f-kube-api-access-kb2jw\") pod \"afde60aa-712f-46da-a245-9df1d7e8326f\" (UID: \"afde60aa-712f-46da-a245-9df1d7e8326f\") "
Oct 30 07:45:18.930218 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:18.929784    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08398783-5da2-4d6b-b0d0-4d141510de4b-kube-api-access-jbt6r\" (UniqueName: \"kubernetes.io/projected/08398783-5da2-4d6b-b0d0-4d141510de4b-kube-api-access-jbt6r\") pod \"08398783-5da2-4d6b-b0d0-4d141510de4b\" (UID: \"08398783-5da2-4d6b-b0d0-4d141510de4b\") "
Oct 30 07:45:18.930218 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:18.929982    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b63a195-1c69-4fdd-ba65-07fdc7996d37-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b63a195-1c69-4fdd-ba65-07fdc7996d37-maas-api-service-ca\") pod \"7b63a195-1c69-4fdd-ba65-07fdc7996d37\" (UID: \"7b63a195-1c69-4fdd-ba65-07fdc7996d37\") "
Oct 30 07:45:18.930218 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:18.930047    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afde60aa-712f-46da-a245-9df1d7e8326f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afde60aa-712f-46da-a245-9df1d7e8326f-maas-api-service-ca\") pod \"afde60aa-712f-46da-a245-9df1d7e8326f\" (UID: \"afde60aa-712f-46da-a245-9df1d7e8326f\") "
Oct 30 07:45:18.930218 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:18.930188    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08398783-5da2-4d6b-b0d0-4d141510de4b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08398783-5da2-4d6b-b0d0-4d141510de4b-maas-api-service-ca\") pod \"08398783-5da2-4d6b-b0d0-4d141510de4b\" (UID: \"08398783-5da2-4d6b-b0d0-4d141510de4b\") "
Oct 30 07:45:18.931422 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:18.931154    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afde60aa-712f-46da-a245-9df1d7e8326f-maas-api-service-ca" pod "afde60aa-712f-46da-a245-9df1d7e8326f" (UID: "afde60aa-712f-46da-a245-9df1d7e8326f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:45:18.931422 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:18.931313    6224 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afde60aa-712f-46da-a245-9df1d7e8326f-maas-api-service-ca\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:45:18.931760 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:18.931649    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08398783-5da2-4d6b-b0d0-4d141510de4b-maas-api-service-ca" pod "08398783-5da2-4d6b-b0d0-4d141510de4b" (UID: "08398783-5da2-4d6b-b0d0-4d141510de4b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:45:18.932284 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:18.932262    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b63a195-1c69-4fdd-ba65-07fdc7996d37-maas-api-service-ca" pod "7b63a195-1c69-4fdd-ba65-07fdc7996d37" (UID: "7b63a195-1c69-4fdd-ba65-07fdc7996d37"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:45:18.939979 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:18.939953    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08398783-5da2-4d6b-b0d0-4d141510de4b-kube-api-access-jbt6r" pod "08398783-5da2-4d6b-b0d0-4d141510de4b" (UID: "08398783-5da2-4d6b-b0d0-4d141510de4b"). InnerVolumeSpecName "kube-api-access-jbt6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:45:18.940122 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:18.939992    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afde60aa-712f-46da-a245-9df1d7e8326f-kube-api-access-kb2jw" pod "afde60aa-712f-46da-a245-9df1d7e8326f" (UID: "afde60aa-712f-46da-a245-9df1d7e8326f"). InnerVolumeSpecName "kube-api-access-kb2jw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:45:18.940122 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:18.940009    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b63a195-1c69-4fdd-ba65-07fdc7996d37-kube-api-access-p76t4" pod "7b63a195-1c69-4fdd-ba65-07fdc7996d37" (UID: "7b63a195-1c69-4fdd-ba65-07fdc7996d37"). InnerVolumeSpecName "kube-api-access-p76t4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:45:19.033165 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:19.033120    6224 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b63a195-1c69-4fdd-ba65-07fdc7996d37-maas-api-service-ca\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:45:19.033336 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:19.033169    6224 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08398783-5da2-4d6b-b0d0-4d141510de4b-maas-api-service-ca\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:45:19.033336 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:19.033184    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p76t4\" (UniqueName: \"kubernetes.io/projected/7b63a195-1c69-4fdd-ba65-07fdc7996d37-kube-api-access-p76t4\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:45:19.033336 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:19.033198    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kb2jw\" (UniqueName: \"kubernetes.io/projected/afde60aa-712f-46da-a245-9df1d7e8326f-kube-api-access-kb2jw\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:45:19.033336 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:19.033212    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbt6r\" (UniqueName: \"kubernetes.io/projected/08398783-5da2-4d6b-b0d0-4d141510de4b-kube-api-access-jbt6r\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:45:19.502133 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:19.502048    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ad434e-29889105-fljfl"
Oct 30 07:45:19.503576 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:19.503547    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889105-4x5vf"
Oct 30 07:45:19.505597 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:19.505571    6224 generic.go:356] "Generic (PLEG): container finished" podID="abc98592-b397-43d3-8169-91d59a211e6d" containerID="ebdea22fa9cdbae41b7a41e004e52715d95ac827f015d892e922dfaec94483fb" exitCode=60
Oct 30 07:45:19.506015 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:19.505987    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87432693-c1f2-42bb-9a66-7c148d63f2fa" path="/var/lib/kubelet/pods/87432693-c1f2-42bb-9a66-7c148d63f2fa/volumes"
Oct 30 07:45:19.507240 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:19.507202    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889105-6dfr9"
Oct 30 07:45:19.507240 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:19.507222    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ad434e-29889105-fljfl" event={"ID":"7b63a195-1c69-4fdd-ba65-07fdc7996d37","Type":"ContainerDied","Data":"7919f07cd39529c024bb7c472258a909c45032e3876aad37b3d4c302f0239739"}
Oct 30 07:45:19.507436 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:19.507259    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889105-4x5vf" event={"ID":"08398783-5da2-4d6b-b0d0-4d141510de4b","Type":"ContainerDied","Data":"52602708aac77f774fc197ea5e33b8c34d001a9aebf8260fda8aab6808cdcc32"}
Oct 30 07:45:19.507436 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:19.507283    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889105-pkg77" event={"ID":"abc98592-b397-43d3-8169-91d59a211e6d","Type":"ContainerDied","Data":"ebdea22fa9cdbae41b7a41e004e52715d95ac827f015d892e922dfaec94483fb"}
Oct 30 07:45:19.507436 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:19.507313    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889105-6dfr9" event={"ID":"afde60aa-712f-46da-a245-9df1d7e8326f","Type":"ContainerDied","Data":"132101cce56bcafcf4f0bcccbeb750bd7efe143009b480beda4119b39273242b"}
Oct 30 07:45:19.507701 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:19.507679    6224 scope.go:122] "RemoveContainer" containerID="ebdea22fa9cdbae41b7a41e004e52715d95ac827f015d892e922dfaec94483fb"
Oct 30 07:45:19.508206 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:45:19.508180    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889105-pkg77_odh-ai-gateway-infra(abc98592-b397-43d3-8169-91d59a211e6d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889105-pkg77" podUID="abc98592-b397-43d3-8169-91d59a211e6d"
Oct 30 07:45:19.508984 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:19.508512    6224 scope.go:122] "RemoveContainer" containerID="f31e2a901f3c29868e24e269e396ecad678964f69054d02429d7e46cd9da91d4"
Oct 30 07:45:19.520680 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:19.520654    6224 generic.go:356] "Generic (PLEG): container finished" podID="b1966a9a-09be-440c-a430-8c79b8c89684" containerID="0b4808f4f0919eb60449e766bd158487d6c56480b46958128690d51104433554" exitCode=60
Oct 30 07:45:19.520797 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:19.520726    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889105-fmdqz" event={"ID":"b1966a9a-09be-440c-a430-8c79b8c89684","Type":"ContainerDied","Data":"0b4808f4f0919eb60449e766bd158487d6c56480b46958128690d51104433554"}
Oct 30 07:45:19.521164 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:19.521145    6224 scope.go:122] "RemoveContainer" containerID="0b4808f4f0919eb60449e766bd158487d6c56480b46958128690d51104433554"
Oct 30 07:45:19.521467 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:45:19.521443    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-137122-29889105-fmdqz_odh-ai-gateway-infra(b1966a9a-09be-440c-a430-8c79b8c89684)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889105-fmdqz" podUID="b1966a9a-09be-440c-a430-8c79b8c89684"
Oct 30 07:45:19.522852 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:19.522831    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8fef73-29889105-xmpv9" event={"ID":"49a8dd13-feae-465b-a031-aea87e1d8a59","Type":"ContainerDied","Data":"be5bc9e49b196b7c78628846dae9ef05101d9e197b68add863294cc3f7f7b58b"}
Oct 30 07:45:19.522956 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:19.522801    6224 generic.go:356] "Generic (PLEG): container finished" podID="49a8dd13-feae-465b-a031-aea87e1d8a59" containerID="be5bc9e49b196b7c78628846dae9ef05101d9e197b68add863294cc3f7f7b58b" exitCode=60
Oct 30 07:45:19.523192 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:19.523176    6224 scope.go:122] "RemoveContainer" containerID="be5bc9e49b196b7c78628846dae9ef05101d9e197b68add863294cc3f7f7b58b"
Oct 30 07:45:19.523434 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:45:19.523415    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-8fef73-29889105-xmpv9_odh-ai-gateway-infra(49a8dd13-feae-465b-a031-aea87e1d8a59)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8fef73-29889105-xmpv9" podUID="49a8dd13-feae-465b-a031-aea87e1d8a59"
Oct 30 07:45:19.548995 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:19.548966    6224 scope.go:122] "RemoveContainer" containerID="ea0ac896f16098b65d72d3ac821f735754c2a576f5d9487b1e6af868c5d710ba"
Oct 30 07:45:19.563507 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:19.563473    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8fef73-29889105-xmpv9"]
Oct 30 07:45:19.582598 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:19.582564    6224 scope.go:122] "RemoveContainer" containerID="6e2d110c5204125a9307e5f56578cd7a452cb32a4618f4a883f37eb8faa4c070"
Oct 30 07:45:19.619062 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:19.619030    6224 scope.go:122] "RemoveContainer" containerID="b170fdd7d3059ed8717e8be9c5897da53c57f1227ce195d9de76467bf64b77d7"
Oct 30 07:45:19.635827 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:19.635783    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889105-pkg77"]
Oct 30 07:45:19.644980 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:19.644957    6224 scope.go:122] "RemoveContainer" containerID="ef0a00b4bfee677009d2e6fb500622fc712347bc15f3fe87229dfcea4b992dcf"
Oct 30 07:45:19.664745 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:19.664725    6224 scope.go:122] "RemoveContainer" containerID="d88cd01b6952181a7a0772c8daf86e4e6439a245ab3a2483a0e899c581965c5c"
Oct 30 07:45:19.816189 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:19.816160    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889105-gmwl4"
Oct 30 07:45:19.818767 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:19.818742    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f5081-29889105-4t64q"
Oct 30 07:45:19.859432 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:19.859395    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ad434e-29889105-fljfl"]
Oct 30 07:45:19.866564 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:19.866525    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ad434e-29889105-fljfl"]
Oct 30 07:45:19.904053 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:19.904023    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889105-4x5vf"]
Oct 30 07:45:19.907781 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:19.907756    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889105-fmdqz"]
Oct 30 07:45:19.912863 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:19.912839    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb6be4-29889105-4x5vf"]
Oct 30 07:45:19.946961 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:19.946935    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f487dd5e-0777-4bfb-98c9-81b2602e63a3-kube-api-access-j27d8\" (UniqueName: \"kubernetes.io/projected/f487dd5e-0777-4bfb-98c9-81b2602e63a3-kube-api-access-j27d8\") pod \"f487dd5e-0777-4bfb-98c9-81b2602e63a3\" (UID: \"f487dd5e-0777-4bfb-98c9-81b2602e63a3\") "
Oct 30 07:45:19.947091 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:19.947077    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f487dd5e-0777-4bfb-98c9-81b2602e63a3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f487dd5e-0777-4bfb-98c9-81b2602e63a3-maas-api-service-ca\") pod \"f487dd5e-0777-4bfb-98c9-81b2602e63a3\" (UID: \"f487dd5e-0777-4bfb-98c9-81b2602e63a3\") "
Oct 30 07:45:19.947249 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:19.947234    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1641d01b-b971-4bf0-b3af-e1523a410dc5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1641d01b-b971-4bf0-b3af-e1523a410dc5-maas-api-service-ca\") pod \"1641d01b-b971-4bf0-b3af-e1523a410dc5\" (UID: \"1641d01b-b971-4bf0-b3af-e1523a410dc5\") "
Oct 30 07:45:19.947297 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:19.947272    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1641d01b-b971-4bf0-b3af-e1523a410dc5-kube-api-access-sdhqc\" (UniqueName: \"kubernetes.io/projected/1641d01b-b971-4bf0-b3af-e1523a410dc5-kube-api-access-sdhqc\") pod \"1641d01b-b971-4bf0-b3af-e1523a410dc5\" (UID: \"1641d01b-b971-4bf0-b3af-e1523a410dc5\") "
Oct 30 07:45:19.947475 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:19.947442    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f487dd5e-0777-4bfb-98c9-81b2602e63a3-maas-api-service-ca" pod "f487dd5e-0777-4bfb-98c9-81b2602e63a3" (UID: "f487dd5e-0777-4bfb-98c9-81b2602e63a3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:45:19.947652 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:19.947631    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1641d01b-b971-4bf0-b3af-e1523a410dc5-maas-api-service-ca" pod "1641d01b-b971-4bf0-b3af-e1523a410dc5" (UID: "1641d01b-b971-4bf0-b3af-e1523a410dc5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:45:19.947887 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:19.947865    6224 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f487dd5e-0777-4bfb-98c9-81b2602e63a3-maas-api-service-ca\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:45:19.947975 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:19.947892    6224 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1641d01b-b971-4bf0-b3af-e1523a410dc5-maas-api-service-ca\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:45:19.949206 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:19.949182    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f487dd5e-0777-4bfb-98c9-81b2602e63a3-kube-api-access-j27d8" pod "f487dd5e-0777-4bfb-98c9-81b2602e63a3" (UID: "f487dd5e-0777-4bfb-98c9-81b2602e63a3"). InnerVolumeSpecName "kube-api-access-j27d8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:45:19.949350 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:19.949331    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1641d01b-b971-4bf0-b3af-e1523a410dc5-kube-api-access-sdhqc" pod "1641d01b-b971-4bf0-b3af-e1523a410dc5" (UID: "1641d01b-b971-4bf0-b3af-e1523a410dc5"). InnerVolumeSpecName "kube-api-access-sdhqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:45:19.956087 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:19.956064    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889105-6dfr9"]
Oct 30 07:45:19.964080 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:19.964058    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889105-6dfr9"]
Oct 30 07:45:20.049689 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:20.049655    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdhqc\" (UniqueName: \"kubernetes.io/projected/1641d01b-b971-4bf0-b3af-e1523a410dc5-kube-api-access-sdhqc\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:45:20.049920 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:20.049899    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j27d8\" (UniqueName: \"kubernetes.io/projected/f487dd5e-0777-4bfb-98c9-81b2602e63a3-kube-api-access-j27d8\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:45:20.537259 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:20.537232    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f5081-29889105-4t64q"
Oct 30 07:45:20.537259 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:20.537247    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f5081-29889105-4t64q" event={"ID":"1641d01b-b971-4bf0-b3af-e1523a410dc5","Type":"ContainerDied","Data":"b5a60519afc34082f130ccd32b985b7c7ae3584b90576aabc6c1b68e87ac0d05"}
Oct 30 07:45:20.537780 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:20.537310    6224 scope.go:122] "RemoveContainer" containerID="7771d658d8f7f3291648a87f3873370eb00298c4fe2d2a1d69d67a6a8a8f0125"
Oct 30 07:45:20.538760 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:20.538724    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889105-gmwl4"
Oct 30 07:45:20.538894 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:20.538798    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889105-gmwl4" event={"ID":"f487dd5e-0777-4bfb-98c9-81b2602e63a3","Type":"ContainerDied","Data":"23196585e56eb9b0961f3a0994c8d7dc89ef12e924caf3ef17d08061d561a0fe"}
Oct 30 07:45:20.558380 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:20.553997    6224 scope.go:122] "RemoveContainer" containerID="1afc095722ab92af79f7df0933e5846cab77c32f1b2a75bb449c4786e49966e3"
Oct 30 07:45:20.647556 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:20.640653    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889105-gmwl4"]
Oct 30 07:45:20.647772 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:20.647743    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e27e0c-29889105-gmwl4"]
Oct 30 07:45:20.735307 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:20.735264    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f5081-29889105-4t64q"]
Oct 30 07:45:20.752943 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:20.752910    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f5081-29889105-4t64q"]
Oct 30 07:45:20.803857 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:20.803823    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889105-fmdqz"
Oct 30 07:45:20.807746 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:20.807725    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889105-pkg77"
Oct 30 07:45:20.831885 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:20.831862    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8fef73-29889105-xmpv9"
Oct 30 07:45:20.871154 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:20.860785    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abc98592-b397-43d3-8169-91d59a211e6d-kube-api-access-wxwm5\" (UniqueName: \"kubernetes.io/projected/abc98592-b397-43d3-8169-91d59a211e6d-kube-api-access-wxwm5\") pod \"abc98592-b397-43d3-8169-91d59a211e6d\" (UID: \"abc98592-b397-43d3-8169-91d59a211e6d\") "
Oct 30 07:45:20.871154 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:20.860882    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1966a9a-09be-440c-a430-8c79b8c89684-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1966a9a-09be-440c-a430-8c79b8c89684-maas-api-service-ca\") pod \"b1966a9a-09be-440c-a430-8c79b8c89684\" (UID: \"b1966a9a-09be-440c-a430-8c79b8c89684\") "
Oct 30 07:45:20.871154 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:20.861230    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1966a9a-09be-440c-a430-8c79b8c89684-kube-api-access-lxp4q\" (UniqueName: \"kubernetes.io/projected/b1966a9a-09be-440c-a430-8c79b8c89684-kube-api-access-lxp4q\") pod \"b1966a9a-09be-440c-a430-8c79b8c89684\" (UID: \"b1966a9a-09be-440c-a430-8c79b8c89684\") "
Oct 30 07:45:20.871154 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:20.861316    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abc98592-b397-43d3-8169-91d59a211e6d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abc98592-b397-43d3-8169-91d59a211e6d-maas-api-service-ca\") pod \"abc98592-b397-43d3-8169-91d59a211e6d\" (UID: \"abc98592-b397-43d3-8169-91d59a211e6d\") "
Oct 30 07:45:20.871154 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:20.865725    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abc98592-b397-43d3-8169-91d59a211e6d-maas-api-service-ca" pod "abc98592-b397-43d3-8169-91d59a211e6d" (UID: "abc98592-b397-43d3-8169-91d59a211e6d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:45:20.871154 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:20.866008    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abc98592-b397-43d3-8169-91d59a211e6d-kube-api-access-wxwm5" pod "abc98592-b397-43d3-8169-91d59a211e6d" (UID: "abc98592-b397-43d3-8169-91d59a211e6d"). InnerVolumeSpecName "kube-api-access-wxwm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:45:20.871154 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:20.866189    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1966a9a-09be-440c-a430-8c79b8c89684-maas-api-service-ca" pod "b1966a9a-09be-440c-a430-8c79b8c89684" (UID: "b1966a9a-09be-440c-a430-8c79b8c89684"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:45:20.871154 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:20.868498    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1966a9a-09be-440c-a430-8c79b8c89684-kube-api-access-lxp4q" pod "b1966a9a-09be-440c-a430-8c79b8c89684" (UID: "b1966a9a-09be-440c-a430-8c79b8c89684"). InnerVolumeSpecName "kube-api-access-lxp4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:45:20.963343 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:20.963310    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49a8dd13-feae-465b-a031-aea87e1d8a59-kube-api-access-f4rnh\" (UniqueName: \"kubernetes.io/projected/49a8dd13-feae-465b-a031-aea87e1d8a59-kube-api-access-f4rnh\") pod \"49a8dd13-feae-465b-a031-aea87e1d8a59\" (UID: \"49a8dd13-feae-465b-a031-aea87e1d8a59\") "
Oct 30 07:45:20.963571 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:20.963554    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49a8dd13-feae-465b-a031-aea87e1d8a59-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49a8dd13-feae-465b-a031-aea87e1d8a59-maas-api-service-ca\") pod \"49a8dd13-feae-465b-a031-aea87e1d8a59\" (UID: \"49a8dd13-feae-465b-a031-aea87e1d8a59\") "
Oct 30 07:45:20.963994 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:20.963967    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49a8dd13-feae-465b-a031-aea87e1d8a59-maas-api-service-ca" pod "49a8dd13-feae-465b-a031-aea87e1d8a59" (UID: "49a8dd13-feae-465b-a031-aea87e1d8a59"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:45:20.964191 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:20.964173    6224 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49a8dd13-feae-465b-a031-aea87e1d8a59-maas-api-service-ca\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:45:20.964241 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:20.964199    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxwm5\" (UniqueName: \"kubernetes.io/projected/abc98592-b397-43d3-8169-91d59a211e6d-kube-api-access-wxwm5\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:45:20.964241 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:20.964214    6224 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1966a9a-09be-440c-a430-8c79b8c89684-maas-api-service-ca\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:45:20.964241 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:20.964227    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxp4q\" (UniqueName: \"kubernetes.io/projected/b1966a9a-09be-440c-a430-8c79b8c89684-kube-api-access-lxp4q\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:45:20.964342 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:20.964242    6224 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abc98592-b397-43d3-8169-91d59a211e6d-maas-api-service-ca\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:45:20.965644 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:20.965620    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49a8dd13-feae-465b-a031-aea87e1d8a59-kube-api-access-f4rnh" pod "49a8dd13-feae-465b-a031-aea87e1d8a59" (UID: "49a8dd13-feae-465b-a031-aea87e1d8a59"). InnerVolumeSpecName "kube-api-access-f4rnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:45:21.066160 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:21.066078    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4rnh\" (UniqueName: \"kubernetes.io/projected/49a8dd13-feae-465b-a031-aea87e1d8a59-kube-api-access-f4rnh\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:45:21.506302 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:21.506226    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08398783-5da2-4d6b-b0d0-4d141510de4b" path="/var/lib/kubelet/pods/08398783-5da2-4d6b-b0d0-4d141510de4b/volumes"
Oct 30 07:45:21.506650 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:21.506637    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1641d01b-b971-4bf0-b3af-e1523a410dc5" path="/var/lib/kubelet/pods/1641d01b-b971-4bf0-b3af-e1523a410dc5/volumes"
Oct 30 07:45:21.507089 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:21.507074    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b63a195-1c69-4fdd-ba65-07fdc7996d37" path="/var/lib/kubelet/pods/7b63a195-1c69-4fdd-ba65-07fdc7996d37/volumes"
Oct 30 07:45:21.507781 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:21.507766    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afde60aa-712f-46da-a245-9df1d7e8326f" path="/var/lib/kubelet/pods/afde60aa-712f-46da-a245-9df1d7e8326f/volumes"
Oct 30 07:45:21.508201 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:21.508186    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f487dd5e-0777-4bfb-98c9-81b2602e63a3" path="/var/lib/kubelet/pods/f487dd5e-0777-4bfb-98c9-81b2602e63a3/volumes"
Oct 30 07:45:21.553421 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:21.553391    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889105-pkg77"
Oct 30 07:45:21.553843 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:21.553395    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889105-pkg77" event={"ID":"abc98592-b397-43d3-8169-91d59a211e6d","Type":"ContainerDied","Data":"54650e0ad8ef0db03e087db37b65c86ce12a5a5817cf4d2788239804c7acaad6"}
Oct 30 07:45:21.553843 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:21.553529    6224 scope.go:122] "RemoveContainer" containerID="ebdea22fa9cdbae41b7a41e004e52715d95ac827f015d892e922dfaec94483fb"
Oct 30 07:45:21.557698 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:21.557678    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889105-fmdqz"
Oct 30 07:45:21.557698 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:21.557684    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889105-fmdqz" event={"ID":"b1966a9a-09be-440c-a430-8c79b8c89684","Type":"ContainerDied","Data":"be9ff56a94957b0c5324a245dbc9b1e30e0baa5f4c76768bb6814ae8ec3bb51c"}
Oct 30 07:45:21.559232 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:21.559207    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8fef73-29889105-xmpv9" event={"ID":"49a8dd13-feae-465b-a031-aea87e1d8a59","Type":"ContainerDied","Data":"02c3df0cb8ae75b003fb89b7cc015f3a152d2bc0d0d48e8ac88919e032b69a05"}
Oct 30 07:45:21.559350 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:21.559241    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8fef73-29889105-xmpv9"
Oct 30 07:45:21.568007 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:21.567848    6224 scope.go:122] "RemoveContainer" containerID="0b4808f4f0919eb60449e766bd158487d6c56480b46958128690d51104433554"
Oct 30 07:45:21.577723 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:21.577705    6224 scope.go:122] "RemoveContainer" containerID="be5bc9e49b196b7c78628846dae9ef05101d9e197b68add863294cc3f7f7b58b"
Oct 30 07:45:21.617431 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:21.617399    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889105-pkg77"]
Oct 30 07:45:21.623815 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:21.623784    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b5998f-29889105-pkg77"]
Oct 30 07:45:21.660063 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:21.660034    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889105-fmdqz"]
Oct 30 07:45:21.664517 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:21.664493    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889105-fmdqz"]
Oct 30 07:45:21.703817 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:21.703776    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8fef73-29889105-xmpv9"]
Oct 30 07:45:21.713246 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:21.713222    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8fef73-29889105-xmpv9"]
Oct 30 07:45:23.055740 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:23.054612    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-4ab37e/test-model-4ab37e-kserve-75448f6fc5-ln8dd"
Oct 30 07:45:23.068849 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:23.068799    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-4ab37e/test-model-4ab37e-kserve-75448f6fc5-ln8dd"
Oct 30 07:45:23.506327 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:23.506241    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49a8dd13-feae-465b-a031-aea87e1d8a59" path="/var/lib/kubelet/pods/49a8dd13-feae-465b-a031-aea87e1d8a59/volumes"
Oct 30 07:45:23.506730 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:23.506715    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abc98592-b397-43d3-8169-91d59a211e6d" path="/var/lib/kubelet/pods/abc98592-b397-43d3-8169-91d59a211e6d/volumes"
Oct 30 07:45:23.507138 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:23.507123    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1966a9a-09be-440c-a430-8c79b8c89684" path="/var/lib/kubelet/pods/b1966a9a-09be-440c-a430-8c79b8c89684/volumes"
Oct 30 07:45:23.590011 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:23.589975    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-4ab37e/test-model-4ab37e-kserve-75448f6fc5-ln8dd"
Oct 30 07:45:31.858829 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:45:31.858775    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:45:46.133953 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:46.133918    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:45:47.566972 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:47.566940    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-27d4cd/e2e-usage-sim-32bc7f-7cf9bb66c-mxc4b"]
Oct 30 07:45:47.567372 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:47.567157    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-extmod-shr-user-27d4cd/e2e-usage-sim-32bc7f-7cf9bb66c-mxc4b" podUID="5e4adbdd-072a-4cad-9a46-4bd3c51f04f8" containerName="sim" containerID="cri-o://b52e36d7232ccbf95c88453a8a2ec812645e65898940c4d535b4cff1773f6576" gracePeriod=30
Oct 30 07:45:47.777317 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:47.777280    6224 generic.go:356] "Generic (PLEG): container finished" podID="5e4adbdd-072a-4cad-9a46-4bd3c51f04f8" containerID="b52e36d7232ccbf95c88453a8a2ec812645e65898940c4d535b4cff1773f6576" exitCode=0
Oct 30 07:45:47.777702 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:47.777678    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-27d4cd/e2e-usage-sim-32bc7f-7cf9bb66c-mxc4b" event={"ID":"5e4adbdd-072a-4cad-9a46-4bd3c51f04f8","Type":"ContainerDied","Data":"b52e36d7232ccbf95c88453a8a2ec812645e65898940c4d535b4cff1773f6576"}
Oct 30 07:45:47.852215 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:47.852192    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-27d4cd/e2e-usage-sim-32bc7f-7cf9bb66c-mxc4b"
Oct 30 07:45:48.055593 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:48.055560    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e4adbdd-072a-4cad-9a46-4bd3c51f04f8-tls\" (UniqueName: \"kubernetes.io/secret/5e4adbdd-072a-4cad-9a46-4bd3c51f04f8-tls\") pod \"5e4adbdd-072a-4cad-9a46-4bd3c51f04f8\" (UID: \"5e4adbdd-072a-4cad-9a46-4bd3c51f04f8\") "
Oct 30 07:45:48.055802 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:48.055794    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e4adbdd-072a-4cad-9a46-4bd3c51f04f8-kube-api-access-4zvdb\" (UniqueName: \"kubernetes.io/projected/5e4adbdd-072a-4cad-9a46-4bd3c51f04f8-kube-api-access-4zvdb\") pod \"5e4adbdd-072a-4cad-9a46-4bd3c51f04f8\" (UID: \"5e4adbdd-072a-4cad-9a46-4bd3c51f04f8\") "
Oct 30 07:45:48.057846 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:48.057802    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e4adbdd-072a-4cad-9a46-4bd3c51f04f8-tls" pod "5e4adbdd-072a-4cad-9a46-4bd3c51f04f8" (UID: "5e4adbdd-072a-4cad-9a46-4bd3c51f04f8"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:45:48.057970 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:48.057937    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e4adbdd-072a-4cad-9a46-4bd3c51f04f8-kube-api-access-4zvdb" pod "5e4adbdd-072a-4cad-9a46-4bd3c51f04f8" (UID: "5e4adbdd-072a-4cad-9a46-4bd3c51f04f8"). InnerVolumeSpecName "kube-api-access-4zvdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:45:48.157620 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:48.157553    6224 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/5e4adbdd-072a-4cad-9a46-4bd3c51f04f8-tls\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:45:48.157620 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:48.157579    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zvdb\" (UniqueName: \"kubernetes.io/projected/5e4adbdd-072a-4cad-9a46-4bd3c51f04f8-kube-api-access-4zvdb\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:45:48.786477 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:48.786443    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-27d4cd/e2e-usage-sim-32bc7f-7cf9bb66c-mxc4b"
Oct 30 07:45:48.786981 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:48.786476    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-27d4cd/e2e-usage-sim-32bc7f-7cf9bb66c-mxc4b" event={"ID":"5e4adbdd-072a-4cad-9a46-4bd3c51f04f8","Type":"ContainerDied","Data":"5f47d08234da9ec1694a2ad2bf55486b6578fd2be98dcb16443b007648092f18"}
Oct 30 07:45:48.786981 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:48.786537    6224 scope.go:122] "RemoveContainer" containerID="b52e36d7232ccbf95c88453a8a2ec812645e65898940c4d535b4cff1773f6576"
Oct 30 07:45:48.851332 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:48.851306    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-27d4cd/e2e-usage-sim-32bc7f-7cf9bb66c-mxc4b"]
Oct 30 07:45:48.865126 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:48.865094    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-extmod-shr-user-27d4cd/e2e-usage-sim-32bc7f-7cf9bb66c-mxc4b"]
Oct 30 07:45:49.507690 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:45:49.507652    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e4adbdd-072a-4cad-9a46-4bd3c51f04f8" path="/var/lib/kubelet/pods/5e4adbdd-072a-4cad-9a46-4bd3c51f04f8/volumes"
Oct 30 07:46:01.151544 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.151511    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ccac6f-8567799696-99jx8"]
Oct 30 07:46:01.152508 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.152484    6224 cpu_manager.go:414] "containerMap: removing container" podUID="afde60aa-712f-46da-a245-9df1d7e8326f" containerName="cleanup"
Oct 30 07:46:01.152508 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.152508    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afde60aa-712f-46da-a245-9df1d7e8326f" containerName="cleanup"
Oct 30 07:46:01.152679 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.152528    6224 cpu_manager.go:414] "containerMap: removing container" podUID="b1966a9a-09be-440c-a430-8c79b8c89684" containerName="cleanup"
Oct 30 07:46:01.152679 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.152535    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1966a9a-09be-440c-a430-8c79b8c89684" containerName="cleanup"
Oct 30 07:46:01.152679 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.152548    6224 cpu_manager.go:414] "containerMap: removing container" podUID="49a8dd13-feae-465b-a031-aea87e1d8a59" containerName="cleanup"
Oct 30 07:46:01.152679 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.152556    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49a8dd13-feae-465b-a031-aea87e1d8a59" containerName="cleanup"
Oct 30 07:46:01.152679 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.152565    6224 cpu_manager.go:414] "containerMap: removing container" podUID="7b63a195-1c69-4fdd-ba65-07fdc7996d37" containerName="cleanup"
Oct 30 07:46:01.152679 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.152572    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b63a195-1c69-4fdd-ba65-07fdc7996d37" containerName="cleanup"
Oct 30 07:46:01.152679 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.152584    6224 cpu_manager.go:414] "containerMap: removing container" podUID="87432693-c1f2-42bb-9a66-7c148d63f2fa" containerName="cleanup"
Oct 30 07:46:01.152679 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.152591    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87432693-c1f2-42bb-9a66-7c148d63f2fa" containerName="cleanup"
Oct 30 07:46:01.152679 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.152599    6224 cpu_manager.go:414] "containerMap: removing container" podUID="08398783-5da2-4d6b-b0d0-4d141510de4b" containerName="cleanup"
Oct 30 07:46:01.152679 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.152605    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08398783-5da2-4d6b-b0d0-4d141510de4b" containerName="cleanup"
Oct 30 07:46:01.152679 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.152617    6224 cpu_manager.go:414] "containerMap: removing container" podUID="7b63a195-1c69-4fdd-ba65-07fdc7996d37" containerName="cleanup"
Oct 30 07:46:01.152679 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.152626    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b63a195-1c69-4fdd-ba65-07fdc7996d37" containerName="cleanup"
Oct 30 07:46:01.152679 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.152637    6224 cpu_manager.go:414] "containerMap: removing container" podUID="afde60aa-712f-46da-a245-9df1d7e8326f" containerName="cleanup"
Oct 30 07:46:01.152679 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.152645    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afde60aa-712f-46da-a245-9df1d7e8326f" containerName="cleanup"
Oct 30 07:46:01.152679 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.152656    6224 cpu_manager.go:414] "containerMap: removing container" podUID="b1966a9a-09be-440c-a430-8c79b8c89684" containerName="cleanup"
Oct 30 07:46:01.152679 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.152664    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1966a9a-09be-440c-a430-8c79b8c89684" containerName="cleanup"
Oct 30 07:46:01.152679 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.152685    6224 cpu_manager.go:414] "containerMap: removing container" podUID="49a8dd13-feae-465b-a031-aea87e1d8a59" containerName="cleanup"
Oct 30 07:46:01.153596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.152693    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49a8dd13-feae-465b-a031-aea87e1d8a59" containerName="cleanup"
Oct 30 07:46:01.153596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.152710    6224 cpu_manager.go:414] "containerMap: removing container" podUID="abc98592-b397-43d3-8169-91d59a211e6d" containerName="cleanup"
Oct 30 07:46:01.153596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.152717    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abc98592-b397-43d3-8169-91d59a211e6d" containerName="cleanup"
Oct 30 07:46:01.153596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.152730    6224 cpu_manager.go:414] "containerMap: removing container" podUID="5e4adbdd-072a-4cad-9a46-4bd3c51f04f8" containerName="sim"
Oct 30 07:46:01.153596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.152737    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e4adbdd-072a-4cad-9a46-4bd3c51f04f8" containerName="sim"
Oct 30 07:46:01.153596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.152746    6224 cpu_manager.go:414] "containerMap: removing container" podUID="08398783-5da2-4d6b-b0d0-4d141510de4b" containerName="cleanup"
Oct 30 07:46:01.153596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.152753    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08398783-5da2-4d6b-b0d0-4d141510de4b" containerName="cleanup"
Oct 30 07:46:01.153596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.152765    6224 cpu_manager.go:414] "containerMap: removing container" podUID="b1966a9a-09be-440c-a430-8c79b8c89684" containerName="cleanup"
Oct 30 07:46:01.153596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.152771    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1966a9a-09be-440c-a430-8c79b8c89684" containerName="cleanup"
Oct 30 07:46:01.153596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.152787    6224 cpu_manager.go:414] "containerMap: removing container" podUID="87432693-c1f2-42bb-9a66-7c148d63f2fa" containerName="cleanup"
Oct 30 07:46:01.153596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.152794    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87432693-c1f2-42bb-9a66-7c148d63f2fa" containerName="cleanup"
Oct 30 07:46:01.153596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.152837    6224 cpu_manager.go:414] "containerMap: removing container" podUID="7b63a195-1c69-4fdd-ba65-07fdc7996d37" containerName="cleanup"
Oct 30 07:46:01.153596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.152845    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b63a195-1c69-4fdd-ba65-07fdc7996d37" containerName="cleanup"
Oct 30 07:46:01.153596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.152863    6224 cpu_manager.go:414] "containerMap: removing container" podUID="1641d01b-b971-4bf0-b3af-e1523a410dc5" containerName="cleanup"
Oct 30 07:46:01.153596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.152870    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1641d01b-b971-4bf0-b3af-e1523a410dc5" containerName="cleanup"
Oct 30 07:46:01.153596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.152881    6224 cpu_manager.go:414] "containerMap: removing container" podUID="08398783-5da2-4d6b-b0d0-4d141510de4b" containerName="cleanup"
Oct 30 07:46:01.153596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.152887    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08398783-5da2-4d6b-b0d0-4d141510de4b" containerName="cleanup"
Oct 30 07:46:01.153596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.152902    6224 cpu_manager.go:414] "containerMap: removing container" podUID="abc98592-b397-43d3-8169-91d59a211e6d" containerName="cleanup"
Oct 30 07:46:01.153596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.152908    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abc98592-b397-43d3-8169-91d59a211e6d" containerName="cleanup"
Oct 30 07:46:01.153596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.152917    6224 cpu_manager.go:414] "containerMap: removing container" podUID="1641d01b-b971-4bf0-b3af-e1523a410dc5" containerName="cleanup"
Oct 30 07:46:01.153596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.152924    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1641d01b-b971-4bf0-b3af-e1523a410dc5" containerName="cleanup"
Oct 30 07:46:01.153596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.152933    6224 cpu_manager.go:414] "containerMap: removing container" podUID="f487dd5e-0777-4bfb-98c9-81b2602e63a3" containerName="cleanup"
Oct 30 07:46:01.153596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.152941    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f487dd5e-0777-4bfb-98c9-81b2602e63a3" containerName="cleanup"
Oct 30 07:46:01.153596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.152963    6224 cpu_manager.go:414] "containerMap: removing container" podUID="49a8dd13-feae-465b-a031-aea87e1d8a59" containerName="cleanup"
Oct 30 07:46:01.153596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.152970    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49a8dd13-feae-465b-a031-aea87e1d8a59" containerName="cleanup"
Oct 30 07:46:01.153596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.152979    6224 cpu_manager.go:414] "containerMap: removing container" podUID="f487dd5e-0777-4bfb-98c9-81b2602e63a3" containerName="cleanup"
Oct 30 07:46:01.153596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.152986    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f487dd5e-0777-4bfb-98c9-81b2602e63a3" containerName="cleanup"
Oct 30 07:46:01.153596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.153000    6224 cpu_manager.go:414] "containerMap: removing container" podUID="f487dd5e-0777-4bfb-98c9-81b2602e63a3" containerName="cleanup"
Oct 30 07:46:01.153596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.153007    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f487dd5e-0777-4bfb-98c9-81b2602e63a3" containerName="cleanup"
Oct 30 07:46:01.153596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.153018    6224 cpu_manager.go:414] "containerMap: removing container" podUID="87432693-c1f2-42bb-9a66-7c148d63f2fa" containerName="cleanup"
Oct 30 07:46:01.153596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.153026    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87432693-c1f2-42bb-9a66-7c148d63f2fa" containerName="cleanup"
Oct 30 07:46:01.153596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.153050    6224 cpu_manager.go:414] "containerMap: removing container" podUID="1641d01b-b971-4bf0-b3af-e1523a410dc5" containerName="cleanup"
Oct 30 07:46:01.153596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.153057    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1641d01b-b971-4bf0-b3af-e1523a410dc5" containerName="cleanup"
Oct 30 07:46:01.153596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.153078    6224 cpu_manager.go:414] "containerMap: removing container" podUID="abc98592-b397-43d3-8169-91d59a211e6d" containerName="cleanup"
Oct 30 07:46:01.153596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.153087    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abc98592-b397-43d3-8169-91d59a211e6d" containerName="cleanup"
Oct 30 07:46:01.153596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.153299    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="afde60aa-712f-46da-a245-9df1d7e8326f" containerName="cleanup"
Oct 30 07:46:01.153596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.153314    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="1641d01b-b971-4bf0-b3af-e1523a410dc5" containerName="cleanup"
Oct 30 07:46:01.153596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.153329    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="abc98592-b397-43d3-8169-91d59a211e6d" containerName="cleanup"
Oct 30 07:46:01.153596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.153340    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="afde60aa-712f-46da-a245-9df1d7e8326f" containerName="cleanup"
Oct 30 07:46:01.153596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.153349    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1966a9a-09be-440c-a430-8c79b8c89684" containerName="cleanup"
Oct 30 07:46:01.153596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.153357    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="49a8dd13-feae-465b-a031-aea87e1d8a59" containerName="cleanup"
Oct 30 07:46:01.153596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.153365    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b63a195-1c69-4fdd-ba65-07fdc7996d37" containerName="cleanup"
Oct 30 07:46:01.153596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.153374    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="87432693-c1f2-42bb-9a66-7c148d63f2fa" containerName="cleanup"
Oct 30 07:46:01.153596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.153382    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="08398783-5da2-4d6b-b0d0-4d141510de4b" containerName="cleanup"
Oct 30 07:46:01.153596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.153389    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b63a195-1c69-4fdd-ba65-07fdc7996d37" containerName="cleanup"
Oct 30 07:46:01.153596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.153399    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="afde60aa-712f-46da-a245-9df1d7e8326f" containerName="cleanup"
Oct 30 07:46:01.153596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.153406    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1966a9a-09be-440c-a430-8c79b8c89684" containerName="cleanup"
Oct 30 07:46:01.153596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.153415    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="49a8dd13-feae-465b-a031-aea87e1d8a59" containerName="cleanup"
Oct 30 07:46:01.153596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.153426    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e4adbdd-072a-4cad-9a46-4bd3c51f04f8" containerName="sim"
Oct 30 07:46:01.153596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.153436    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="08398783-5da2-4d6b-b0d0-4d141510de4b" containerName="cleanup"
Oct 30 07:46:01.153596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.153445    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="87432693-c1f2-42bb-9a66-7c148d63f2fa" containerName="cleanup"
Oct 30 07:46:01.153596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.153456    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="abc98592-b397-43d3-8169-91d59a211e6d" containerName="cleanup"
Oct 30 07:46:01.153596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.153465    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="1641d01b-b971-4bf0-b3af-e1523a410dc5" containerName="cleanup"
Oct 30 07:46:01.153596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.153474    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="f487dd5e-0777-4bfb-98c9-81b2602e63a3" containerName="cleanup"
Oct 30 07:46:01.153596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.153484    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="f487dd5e-0777-4bfb-98c9-81b2602e63a3" containerName="cleanup"
Oct 30 07:46:01.153596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.153493    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="f487dd5e-0777-4bfb-98c9-81b2602e63a3" containerName="cleanup"
Oct 30 07:46:01.153596 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.153503    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="87432693-c1f2-42bb-9a66-7c148d63f2fa" containerName="cleanup"
Oct 30 07:46:01.157915 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.157891    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ccac6f-8567799696-99jx8"
Oct 30 07:46:01.178200 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.178172    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ccac6f-8567799696-99jx8"]
Oct 30 07:46:01.188209 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.188185    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-ccac6f\""
Oct 30 07:46:01.254203 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.254169    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5qb8\" (UniqueName: \"kubernetes.io/projected/8be122aa-a9d8-45f4-b4c7-738496b41217-kube-api-access-q5qb8\") pod \"payload-processing-e2e-inf-a-ccac6f-8567799696-99jx8\" (UID: \"8be122aa-a9d8-45f4-b4c7-738496b41217\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ccac6f-8567799696-99jx8"
Oct 30 07:46:01.254376 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.254320    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8be122aa-a9d8-45f4-b4c7-738496b41217-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ccac6f-8567799696-99jx8\" (UID: \"8be122aa-a9d8-45f4-b4c7-738496b41217\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ccac6f-8567799696-99jx8"
Oct 30 07:46:01.363016 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.362973    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5qb8\" (UniqueName: \"kubernetes.io/projected/8be122aa-a9d8-45f4-b4c7-738496b41217-kube-api-access-q5qb8\") pod \"payload-processing-e2e-inf-a-ccac6f-8567799696-99jx8\" (UID: \"8be122aa-a9d8-45f4-b4c7-738496b41217\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ccac6f-8567799696-99jx8"
Oct 30 07:46:01.363329 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.363304    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8be122aa-a9d8-45f4-b4c7-738496b41217-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ccac6f-8567799696-99jx8\" (UID: \"8be122aa-a9d8-45f4-b4c7-738496b41217\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ccac6f-8567799696-99jx8"
Oct 30 07:46:01.367829 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.364474    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8be122aa-a9d8-45f4-b4c7-738496b41217-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ccac6f-8567799696-99jx8\" (UID: \"8be122aa-a9d8-45f4-b4c7-738496b41217\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ccac6f-8567799696-99jx8"
Oct 30 07:46:01.421899 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.421638    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5qb8\" (UniqueName: \"kubernetes.io/projected/8be122aa-a9d8-45f4-b4c7-738496b41217-kube-api-access-q5qb8\") pod \"payload-processing-e2e-inf-a-ccac6f-8567799696-99jx8\" (UID: \"8be122aa-a9d8-45f4-b4c7-738496b41217\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ccac6f-8567799696-99jx8"
Oct 30 07:46:01.472748 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.472717    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ccac6f-8567799696-99jx8"
Oct 30 07:46:01.780507 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:46:01.780476    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8be122aa_a9d8_45f4_b4c7_738496b41217.slice/crio-edc4760c7d6238fba7f10e2288e6dad91b1430dd276a4d1b5b093a8ffdd8438f WatchSource:0}: Error finding container edc4760c7d6238fba7f10e2288e6dad91b1430dd276a4d1b5b093a8ffdd8438f: Status 404 returned error can't find the container with id edc4760c7d6238fba7f10e2288e6dad91b1430dd276a4d1b5b093a8ffdd8438f
Oct 30 07:46:01.781965 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.781940    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ccac6f-8567799696-99jx8"]
Oct 30 07:46:01.866950 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:46:01.866915    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:46:01.896437 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.896400    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ccac6f-8567799696-99jx8" event={"ID":"8be122aa-a9d8-45f4-b4c7-738496b41217","Type":"ContainerStarted","Data":"d0280e3571343e26a6dfb9daa90d7b298304dec5689bc2f05d42f455989e03cf"}
Oct 30 07:46:01.896590 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.896462    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ccac6f-8567799696-99jx8" event={"ID":"8be122aa-a9d8-45f4-b4c7-738496b41217","Type":"ContainerStarted","Data":"edc4760c7d6238fba7f10e2288e6dad91b1430dd276a4d1b5b093a8ffdd8438f"}
Oct 30 07:46:01.896590 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.896495    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ccac6f-8567799696-99jx8"
Oct 30 07:46:01.988385 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:01.988328    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-ccac6f-8567799696-99jx8" podStartSLOduration=0.988315041 podStartE2EDuration="988.315041ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:46:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:46:01.987533333 +0000 UTC m=+2920.596940303" watchObservedRunningTime="2026-10-30 07:46:01.988315041 +0000 UTC m=+2920.597722010"
Oct 30 07:46:12.907324 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:12.907296    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ccac6f-8567799696-99jx8"
Oct 30 07:46:16.203591 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:16.203558    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4"]
Oct 30 07:46:16.204315 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:16.204301    6224 cpu_manager.go:414] "containerMap: removing container" podUID="afde60aa-712f-46da-a245-9df1d7e8326f" containerName="cleanup"
Oct 30 07:46:16.204369 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:16.204317    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afde60aa-712f-46da-a245-9df1d7e8326f" containerName="cleanup"
Oct 30 07:46:16.204424 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:16.204415    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="1641d01b-b971-4bf0-b3af-e1523a410dc5" containerName="cleanup"
Oct 30 07:46:16.204461 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:16.204429    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="abc98592-b397-43d3-8169-91d59a211e6d" containerName="cleanup"
Oct 30 07:46:16.204461 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:16.204439    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b63a195-1c69-4fdd-ba65-07fdc7996d37" containerName="cleanup"
Oct 30 07:46:16.204461 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:16.204445    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1966a9a-09be-440c-a430-8c79b8c89684" containerName="cleanup"
Oct 30 07:46:16.204461 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:16.204452    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="49a8dd13-feae-465b-a031-aea87e1d8a59" containerName="cleanup"
Oct 30 07:46:16.204461 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:16.204460    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="08398783-5da2-4d6b-b0d0-4d141510de4b" containerName="cleanup"
Oct 30 07:46:16.208201 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:16.208182    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4"
Oct 30 07:46:16.227923 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:16.227894    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-4ab37e-openshift-default-dockercfg-4l69h\""
Oct 30 07:46:16.246046 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:16.246021    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4"]
Oct 30 07:46:16.298963 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:16.298926    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f2070a8-8f2b-4673-a431-b3640f77d332-workload-socket\") pod \"e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4\" (UID: \"0f2070a8-8f2b-4673-a431-b3640f77d332\") " pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4"
Oct 30 07:46:16.299100 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:16.298998    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f2070a8-8f2b-4673-a431-b3640f77d332-istio-data\") pod \"e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4\" (UID: \"0f2070a8-8f2b-4673-a431-b3640f77d332\") " pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4"
Oct 30 07:46:16.299159 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:16.299090    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f2070a8-8f2b-4673-a431-b3640f77d332-workload-certs\") pod \"e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4\" (UID: \"0f2070a8-8f2b-4673-a431-b3640f77d332\") " pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4"
Oct 30 07:46:16.299159 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:16.299123    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f2070a8-8f2b-4673-a431-b3640f77d332-istio-podinfo\") pod \"e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4\" (UID: \"0f2070a8-8f2b-4673-a431-b3640f77d332\") " pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4"
Oct 30 07:46:16.299238 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:16.299181    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f2070a8-8f2b-4673-a431-b3640f77d332-credential-socket\") pod \"e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4\" (UID: \"0f2070a8-8f2b-4673-a431-b3640f77d332\") " pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4"
Oct 30 07:46:16.299238 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:16.299222    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68qk5\" (UniqueName: \"kubernetes.io/projected/0f2070a8-8f2b-4673-a431-b3640f77d332-kube-api-access-68qk5\") pod \"e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4\" (UID: \"0f2070a8-8f2b-4673-a431-b3640f77d332\") " pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4"
Oct 30 07:46:16.299377 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:16.299357    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f2070a8-8f2b-4673-a431-b3640f77d332-istio-token\") pod \"e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4\" (UID: \"0f2070a8-8f2b-4673-a431-b3640f77d332\") " pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4"
Oct 30 07:46:16.299421 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:16.299388    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f2070a8-8f2b-4673-a431-b3640f77d332-istio-envoy\") pod \"e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4\" (UID: \"0f2070a8-8f2b-4673-a431-b3640f77d332\") " pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4"
Oct 30 07:46:16.299421 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:16.299410    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f2070a8-8f2b-4673-a431-b3640f77d332-istiod-ca-cert\") pod \"e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4\" (UID: \"0f2070a8-8f2b-4673-a431-b3640f77d332\") " pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4"
Oct 30 07:46:16.400371 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:16.400339    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f2070a8-8f2b-4673-a431-b3640f77d332-istio-envoy\") pod \"e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4\" (UID: \"0f2070a8-8f2b-4673-a431-b3640f77d332\") " pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4"
Oct 30 07:46:16.400542 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:16.400381    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f2070a8-8f2b-4673-a431-b3640f77d332-istiod-ca-cert\") pod \"e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4\" (UID: \"0f2070a8-8f2b-4673-a431-b3640f77d332\") " pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4"
Oct 30 07:46:16.400542 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:16.400437    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f2070a8-8f2b-4673-a431-b3640f77d332-workload-socket\") pod \"e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4\" (UID: \"0f2070a8-8f2b-4673-a431-b3640f77d332\") " pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4"
Oct 30 07:46:16.400542 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:16.400483    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f2070a8-8f2b-4673-a431-b3640f77d332-istio-data\") pod \"e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4\" (UID: \"0f2070a8-8f2b-4673-a431-b3640f77d332\") " pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4"
Oct 30 07:46:16.400542 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:16.400540    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f2070a8-8f2b-4673-a431-b3640f77d332-workload-certs\") pod \"e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4\" (UID: \"0f2070a8-8f2b-4673-a431-b3640f77d332\") " pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4"
Oct 30 07:46:16.400782 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:16.400563    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f2070a8-8f2b-4673-a431-b3640f77d332-istio-podinfo\") pod \"e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4\" (UID: \"0f2070a8-8f2b-4673-a431-b3640f77d332\") " pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4"
Oct 30 07:46:16.400782 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:16.400631    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f2070a8-8f2b-4673-a431-b3640f77d332-credential-socket\") pod \"e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4\" (UID: \"0f2070a8-8f2b-4673-a431-b3640f77d332\") " pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4"
Oct 30 07:46:16.400977 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:16.400842    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68qk5\" (UniqueName: \"kubernetes.io/projected/0f2070a8-8f2b-4673-a431-b3640f77d332-kube-api-access-68qk5\") pod \"e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4\" (UID: \"0f2070a8-8f2b-4673-a431-b3640f77d332\") " pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4"
Oct 30 07:46:16.400977 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:16.400968    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f2070a8-8f2b-4673-a431-b3640f77d332-istio-token\") pod \"e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4\" (UID: \"0f2070a8-8f2b-4673-a431-b3640f77d332\") " pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4"
Oct 30 07:46:16.401179 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:16.401115    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f2070a8-8f2b-4673-a431-b3640f77d332-workload-socket\") pod \"e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4\" (UID: \"0f2070a8-8f2b-4673-a431-b3640f77d332\") " pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4"
Oct 30 07:46:16.401289 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:16.401193    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f2070a8-8f2b-4673-a431-b3640f77d332-istio-data\") pod \"e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4\" (UID: \"0f2070a8-8f2b-4673-a431-b3640f77d332\") " pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4"
Oct 30 07:46:16.401289 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:16.401201    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f2070a8-8f2b-4673-a431-b3640f77d332-workload-certs\") pod \"e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4\" (UID: \"0f2070a8-8f2b-4673-a431-b3640f77d332\") " pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4"
Oct 30 07:46:16.401409 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:16.401339    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f2070a8-8f2b-4673-a431-b3640f77d332-credential-socket\") pod \"e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4\" (UID: \"0f2070a8-8f2b-4673-a431-b3640f77d332\") " pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4"
Oct 30 07:46:16.401690 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:16.401671    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f2070a8-8f2b-4673-a431-b3640f77d332-istiod-ca-cert\") pod \"e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4\" (UID: \"0f2070a8-8f2b-4673-a431-b3640f77d332\") " pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4"
Oct 30 07:46:16.403005 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:16.402977    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f2070a8-8f2b-4673-a431-b3640f77d332-istio-envoy\") pod \"e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4\" (UID: \"0f2070a8-8f2b-4673-a431-b3640f77d332\") " pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4"
Oct 30 07:46:16.403324 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:16.403298    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f2070a8-8f2b-4673-a431-b3640f77d332-istio-podinfo\") pod \"e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4\" (UID: \"0f2070a8-8f2b-4673-a431-b3640f77d332\") " pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4"
Oct 30 07:46:16.426510 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:16.426486    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68qk5\" (UniqueName: \"kubernetes.io/projected/0f2070a8-8f2b-4673-a431-b3640f77d332-kube-api-access-68qk5\") pod \"e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4\" (UID: \"0f2070a8-8f2b-4673-a431-b3640f77d332\") " pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4"
Oct 30 07:46:16.426685 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:16.426661    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f2070a8-8f2b-4673-a431-b3640f77d332-istio-token\") pod \"e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4\" (UID: \"0f2070a8-8f2b-4673-a431-b3640f77d332\") " pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4"
Oct 30 07:46:16.521766 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:16.521685    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4"
Oct 30 07:46:16.836727 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:46:16.836692    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f2070a8_8f2b_4673_a431_b3640f77d332.slice/crio-bca8de4366804f503894537b316f7d36882854664eb7d39b33c93d692c5e1647 WatchSource:0}: Error finding container bca8de4366804f503894537b316f7d36882854664eb7d39b33c93d692c5e1647: Status 404 returned error can't find the container with id bca8de4366804f503894537b316f7d36882854664eb7d39b33c93d692c5e1647
Oct 30 07:46:16.837719 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:16.837691    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4"]
Oct 30 07:46:16.839699 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:16.839669    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:46:16.839781 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:16.839734    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:46:16.839781 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:16.839761    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:46:17.023122 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:17.023076    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4" event={"ID":"0f2070a8-8f2b-4673-a431-b3640f77d332","Type":"ContainerStarted","Data":"652a781cfc9622e2da68ceb37967d8d7cf520aa7d538faf1c26dc2ac46291958"}
Oct 30 07:46:17.023288 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:17.023182    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4" event={"ID":"0f2070a8-8f2b-4673-a431-b3640f77d332","Type":"ContainerStarted","Data":"bca8de4366804f503894537b316f7d36882854664eb7d39b33c93d692c5e1647"}
Oct 30 07:46:17.090738 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:17.090624    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4" podStartSLOduration=1.090606442 podStartE2EDuration="1.090606442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:46:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:46:17.086242789 +0000 UTC m=+2935.695649762" watchObservedRunningTime="2026-10-30 07:46:17.090606442 +0000 UTC m=+2935.700013413"
Oct 30 07:46:17.522438 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:17.522356    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4"
Oct 30 07:46:17.524289 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:17.524262    6224 patch_prober.go:29] interesting pod/e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Oct 30 07:46:17.524411 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:17.524314    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4" podUID="0f2070a8-8f2b-4673-a431-b3640f77d332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Oct 30 07:46:18.522322 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:18.522289    6224 patch_prober.go:29] interesting pod/e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Oct 30 07:46:18.522524 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:18.522339    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4" podUID="0f2070a8-8f2b-4673-a431-b3640f77d332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Oct 30 07:46:19.522323 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:19.522287    6224 patch_prober.go:29] interesting pod/e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Oct 30 07:46:19.522536 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:19.522343    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4" podUID="0f2070a8-8f2b-4673-a431-b3640f77d332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Oct 30 07:46:20.523053 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:20.523016    6224 patch_prober.go:29] interesting pod/e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Oct 30 07:46:20.523505 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:20.523082    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4" podUID="0f2070a8-8f2b-4673-a431-b3640f77d332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Oct 30 07:46:21.530456 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:21.525625    6224 patch_prober.go:29] interesting pod/e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Oct 30 07:46:21.530456 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:21.525686    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4" podUID="0f2070a8-8f2b-4673-a431-b3640f77d332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Oct 30 07:46:22.522551 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:22.522508    6224 patch_prober.go:29] interesting pod/e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Oct 30 07:46:22.522760 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:22.522563    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4" podUID="0f2070a8-8f2b-4673-a431-b3640f77d332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Oct 30 07:46:23.347218 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:23.347185    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj"]
Oct 30 07:46:23.347654 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:23.347372    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj" podUID="6cf36810-0393-4f8d-97f5-3a191f94020a" containerName="istio-proxy" containerID="cri-o://7b31518820437b54397e8530d818e34cceadcb7d7908119f1224a87440dd30c0" gracePeriod=30
Oct 30 07:46:23.522524 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:23.522485    6224 patch_prober.go:29] interesting pod/e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Oct 30 07:46:23.522721 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:23.522548    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4" podUID="0f2070a8-8f2b-4673-a431-b3640f77d332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Oct 30 07:46:24.528147 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:24.528118    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4"
Oct 30 07:46:24.528593 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:24.528572    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4"
Oct 30 07:46:24.529202 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:24.529182    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4"
Oct 30 07:46:28.631751 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:28.631674    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj"
Oct 30 07:46:28.726724 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:28.726685    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cf36810-0393-4f8d-97f5-3a191f94020a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cf36810-0393-4f8d-97f5-3a191f94020a-workload-certs\") pod \"6cf36810-0393-4f8d-97f5-3a191f94020a\" (UID: \"6cf36810-0393-4f8d-97f5-3a191f94020a\") "
Oct 30 07:46:28.726975 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:28.726824    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cf36810-0393-4f8d-97f5-3a191f94020a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cf36810-0393-4f8d-97f5-3a191f94020a-istio-envoy\") pod \"6cf36810-0393-4f8d-97f5-3a191f94020a\" (UID: \"6cf36810-0393-4f8d-97f5-3a191f94020a\") "
Oct 30 07:46:28.726975 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:28.726860    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cf36810-0393-4f8d-97f5-3a191f94020a-istio-token\" (UniqueName: \"kubernetes.io/projected/6cf36810-0393-4f8d-97f5-3a191f94020a-istio-token\") pod \"6cf36810-0393-4f8d-97f5-3a191f94020a\" (UID: \"6cf36810-0393-4f8d-97f5-3a191f94020a\") "
Oct 30 07:46:28.727111 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:28.726971    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cf36810-0393-4f8d-97f5-3a191f94020a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cf36810-0393-4f8d-97f5-3a191f94020a-credential-socket\") pod \"6cf36810-0393-4f8d-97f5-3a191f94020a\" (UID: \"6cf36810-0393-4f8d-97f5-3a191f94020a\") "
Oct 30 07:46:28.727111 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:28.727023    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cf36810-0393-4f8d-97f5-3a191f94020a-workload-certs" pod "6cf36810-0393-4f8d-97f5-3a191f94020a" (UID: "6cf36810-0393-4f8d-97f5-3a191f94020a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:46:28.727111 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:28.727049    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cf36810-0393-4f8d-97f5-3a191f94020a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cf36810-0393-4f8d-97f5-3a191f94020a-workload-socket\") pod \"6cf36810-0393-4f8d-97f5-3a191f94020a\" (UID: \"6cf36810-0393-4f8d-97f5-3a191f94020a\") "
Oct 30 07:46:28.727111 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:28.727076    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cf36810-0393-4f8d-97f5-3a191f94020a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cf36810-0393-4f8d-97f5-3a191f94020a-istiod-ca-cert\") pod \"6cf36810-0393-4f8d-97f5-3a191f94020a\" (UID: \"6cf36810-0393-4f8d-97f5-3a191f94020a\") "
Oct 30 07:46:28.727111 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:28.727110    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cf36810-0393-4f8d-97f5-3a191f94020a-kube-api-access-cp5dw\" (UniqueName: \"kubernetes.io/projected/6cf36810-0393-4f8d-97f5-3a191f94020a-kube-api-access-cp5dw\") pod \"6cf36810-0393-4f8d-97f5-3a191f94020a\" (UID: \"6cf36810-0393-4f8d-97f5-3a191f94020a\") "
Oct 30 07:46:28.727395 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:28.727148    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6cf36810-0393-4f8d-97f5-3a191f94020a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cf36810-0393-4f8d-97f5-3a191f94020a-istio-podinfo\") pod \"6cf36810-0393-4f8d-97f5-3a191f94020a\" (UID: \"6cf36810-0393-4f8d-97f5-3a191f94020a\") "
Oct 30 07:46:28.727395 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:28.727244    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cf36810-0393-4f8d-97f5-3a191f94020a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cf36810-0393-4f8d-97f5-3a191f94020a-istio-data\") pod \"6cf36810-0393-4f8d-97f5-3a191f94020a\" (UID: \"6cf36810-0393-4f8d-97f5-3a191f94020a\") "
Oct 30 07:46:28.727499 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:28.727393    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cf36810-0393-4f8d-97f5-3a191f94020a-workload-socket" pod "6cf36810-0393-4f8d-97f5-3a191f94020a" (UID: "6cf36810-0393-4f8d-97f5-3a191f94020a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:46:28.727588 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:28.727568    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cf36810-0393-4f8d-97f5-3a191f94020a-credential-socket" pod "6cf36810-0393-4f8d-97f5-3a191f94020a" (UID: "6cf36810-0393-4f8d-97f5-3a191f94020a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:46:28.727968 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:28.727942    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cf36810-0393-4f8d-97f5-3a191f94020a-istio-data" pod "6cf36810-0393-4f8d-97f5-3a191f94020a" (UID: "6cf36810-0393-4f8d-97f5-3a191f94020a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:46:28.728066 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:28.728026    6224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cf36810-0393-4f8d-97f5-3a191f94020a-credential-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:46:28.728066 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:28.728051    6224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cf36810-0393-4f8d-97f5-3a191f94020a-workload-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:46:28.728178 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:28.728067    6224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cf36810-0393-4f8d-97f5-3a191f94020a-workload-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:46:28.728241 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:28.728229    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cf36810-0393-4f8d-97f5-3a191f94020a-istiod-ca-cert" pod "6cf36810-0393-4f8d-97f5-3a191f94020a" (UID: "6cf36810-0393-4f8d-97f5-3a191f94020a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:46:28.730187 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:28.730160    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cf36810-0393-4f8d-97f5-3a191f94020a-istio-token" pod "6cf36810-0393-4f8d-97f5-3a191f94020a" (UID: "6cf36810-0393-4f8d-97f5-3a191f94020a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:46:28.730301 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:28.730197    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cf36810-0393-4f8d-97f5-3a191f94020a-istio-envoy" pod "6cf36810-0393-4f8d-97f5-3a191f94020a" (UID: "6cf36810-0393-4f8d-97f5-3a191f94020a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:46:28.730470 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:28.730449    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6cf36810-0393-4f8d-97f5-3a191f94020a-istio-podinfo" pod "6cf36810-0393-4f8d-97f5-3a191f94020a" (UID: "6cf36810-0393-4f8d-97f5-3a191f94020a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:46:28.730470 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:28.730464    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cf36810-0393-4f8d-97f5-3a191f94020a-kube-api-access-cp5dw" pod "6cf36810-0393-4f8d-97f5-3a191f94020a" (UID: "6cf36810-0393-4f8d-97f5-3a191f94020a"). InnerVolumeSpecName "kube-api-access-cp5dw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:46:28.835418 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:28.835381    6224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cf36810-0393-4f8d-97f5-3a191f94020a-istio-envoy\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:46:28.835418 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:28.835413    6224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cf36810-0393-4f8d-97f5-3a191f94020a-istio-token\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:46:28.835418 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:28.835424    6224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cf36810-0393-4f8d-97f5-3a191f94020a-istiod-ca-cert\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:46:28.835652 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:28.835432    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cp5dw\" (UniqueName: \"kubernetes.io/projected/6cf36810-0393-4f8d-97f5-3a191f94020a-kube-api-access-cp5dw\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:46:28.835652 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:28.835442    6224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cf36810-0393-4f8d-97f5-3a191f94020a-istio-podinfo\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:46:28.835652 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:28.835452    6224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cf36810-0393-4f8d-97f5-3a191f94020a-istio-data\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:46:29.181765 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:29.181732    6224 generic.go:356] "Generic (PLEG): container finished" podID="6cf36810-0393-4f8d-97f5-3a191f94020a" containerID="7b31518820437b54397e8530d818e34cceadcb7d7908119f1224a87440dd30c0" exitCode=0
Oct 30 07:46:29.181996 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:29.181786    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj"
Oct 30 07:46:29.181996 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:29.181843    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj" event={"ID":"6cf36810-0393-4f8d-97f5-3a191f94020a","Type":"ContainerDied","Data":"7b31518820437b54397e8530d818e34cceadcb7d7908119f1224a87440dd30c0"}
Oct 30 07:46:29.181996 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:29.181906    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj" event={"ID":"6cf36810-0393-4f8d-97f5-3a191f94020a","Type":"ContainerDied","Data":"b14a51a6f072cd00e1db38da7ad4e9819f76b61d23fe6d19d6f701e16f9de860"}
Oct 30 07:46:29.181996 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:29.181928    6224 scope.go:122] "RemoveContainer" containerID="7b31518820437b54397e8530d818e34cceadcb7d7908119f1224a87440dd30c0"
Oct 30 07:46:29.205465 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:29.205406    6224 scope.go:122] "RemoveContainer" containerID="7b31518820437b54397e8530d818e34cceadcb7d7908119f1224a87440dd30c0"
Oct 30 07:46:29.206011 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:46:29.205973    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b31518820437b54397e8530d818e34cceadcb7d7908119f1224a87440dd30c0\": container with ID starting with 7b31518820437b54397e8530d818e34cceadcb7d7908119f1224a87440dd30c0 not found: ID does not exist" containerID="7b31518820437b54397e8530d818e34cceadcb7d7908119f1224a87440dd30c0"
Oct 30 07:46:29.206301 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:29.206018    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b31518820437b54397e8530d818e34cceadcb7d7908119f1224a87440dd30c0"} err="failed to get container status \"7b31518820437b54397e8530d818e34cceadcb7d7908119f1224a87440dd30c0\": rpc error: code = NotFound desc = could not find container \"7b31518820437b54397e8530d818e34cceadcb7d7908119f1224a87440dd30c0\": container with ID starting with 7b31518820437b54397e8530d818e34cceadcb7d7908119f1224a87440dd30c0 not found: ID does not exist"
Oct 30 07:46:29.251837 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:29.251787    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj"]
Oct 30 07:46:29.260367 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:29.260337    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-0f5081-openshift-default-85554fc58f-gpgvj"]
Oct 30 07:46:29.506464 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:29.506385    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cf36810-0393-4f8d-97f5-3a191f94020a" path="/var/lib/kubelet/pods/6cf36810-0393-4f8d-97f5-3a191f94020a/volumes"
Oct 30 07:46:31.859593 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:46:31.859551    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:46:49.853766 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:49.853716    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-8fef73/test-model-8fef73-kserve-54ff956b49-429dm"]
Oct 30 07:46:49.854931 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:49.854907    6224 cpu_manager.go:414] "containerMap: removing container" podUID="6cf36810-0393-4f8d-97f5-3a191f94020a" containerName="istio-proxy"
Oct 30 07:46:49.855104 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:49.855084    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cf36810-0393-4f8d-97f5-3a191f94020a" containerName="istio-proxy"
Oct 30 07:46:49.855354 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:49.855339    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cf36810-0393-4f8d-97f5-3a191f94020a" containerName="istio-proxy"
Oct 30 07:46:49.860734 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:49.860684    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8fef73/test-model-8fef73-kserve-54ff956b49-429dm"
Oct 30 07:46:49.877673 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:49.877646    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-8fef73/test-model-8fef73-kserve-54ff956b49-429dm"]
Oct 30 07:46:49.887615 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:49.887578    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-8fef73\"/\"default-dockercfg-9xdrz\""
Oct 30 07:46:49.901850 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:49.901793    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-8fef73\"/\"openshift-service-ca.crt\""
Oct 30 07:46:49.901986 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:49.901878    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-8fef73\"/\"test-model-8fef73-kserve-self-signed-certs\""
Oct 30 07:46:49.901986 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:49.901920    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-8fef73\"/\"kube-root-ca.crt\""
Oct 30 07:46:49.961235 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:49.961202    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27700938-db52-4225-90cd-244060765bbc-home\") pod \"test-model-8fef73-kserve-54ff956b49-429dm\" (UID: \"27700938-db52-4225-90cd-244060765bbc\") " pod="ai-tenant-e2e-inf-a-8fef73/test-model-8fef73-kserve-54ff956b49-429dm"
Oct 30 07:46:49.961405 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:49.961251    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwk7p\" (UniqueName: \"kubernetes.io/projected/27700938-db52-4225-90cd-244060765bbc-kube-api-access-wwk7p\") pod \"test-model-8fef73-kserve-54ff956b49-429dm\" (UID: \"27700938-db52-4225-90cd-244060765bbc\") " pod="ai-tenant-e2e-inf-a-8fef73/test-model-8fef73-kserve-54ff956b49-429dm"
Oct 30 07:46:49.961405 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:49.961287    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27700938-db52-4225-90cd-244060765bbc-tmp-dir\") pod \"test-model-8fef73-kserve-54ff956b49-429dm\" (UID: \"27700938-db52-4225-90cd-244060765bbc\") " pod="ai-tenant-e2e-inf-a-8fef73/test-model-8fef73-kserve-54ff956b49-429dm"
Oct 30 07:46:49.961530 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:49.961478    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27700938-db52-4225-90cd-244060765bbc-model-cache\") pod \"test-model-8fef73-kserve-54ff956b49-429dm\" (UID: \"27700938-db52-4225-90cd-244060765bbc\") " pod="ai-tenant-e2e-inf-a-8fef73/test-model-8fef73-kserve-54ff956b49-429dm"
Oct 30 07:46:49.961590 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:49.961540    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27700938-db52-4225-90cd-244060765bbc-tls-certs\") pod \"test-model-8fef73-kserve-54ff956b49-429dm\" (UID: \"27700938-db52-4225-90cd-244060765bbc\") " pod="ai-tenant-e2e-inf-a-8fef73/test-model-8fef73-kserve-54ff956b49-429dm"
Oct 30 07:46:49.961648 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:49.961591    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27700938-db52-4225-90cd-244060765bbc-dshm\") pod \"test-model-8fef73-kserve-54ff956b49-429dm\" (UID: \"27700938-db52-4225-90cd-244060765bbc\") " pod="ai-tenant-e2e-inf-a-8fef73/test-model-8fef73-kserve-54ff956b49-429dm"
Oct 30 07:46:50.063458 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:50.063407    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27700938-db52-4225-90cd-244060765bbc-tls-certs\") pod \"test-model-8fef73-kserve-54ff956b49-429dm\" (UID: \"27700938-db52-4225-90cd-244060765bbc\") " pod="ai-tenant-e2e-inf-a-8fef73/test-model-8fef73-kserve-54ff956b49-429dm"
Oct 30 07:46:50.063458 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:50.063451    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27700938-db52-4225-90cd-244060765bbc-dshm\") pod \"test-model-8fef73-kserve-54ff956b49-429dm\" (UID: \"27700938-db52-4225-90cd-244060765bbc\") " pod="ai-tenant-e2e-inf-a-8fef73/test-model-8fef73-kserve-54ff956b49-429dm"
Oct 30 07:46:50.063715 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:50.063510    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27700938-db52-4225-90cd-244060765bbc-home\") pod \"test-model-8fef73-kserve-54ff956b49-429dm\" (UID: \"27700938-db52-4225-90cd-244060765bbc\") " pod="ai-tenant-e2e-inf-a-8fef73/test-model-8fef73-kserve-54ff956b49-429dm"
Oct 30 07:46:50.063715 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:50.063534    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwk7p\" (UniqueName: \"kubernetes.io/projected/27700938-db52-4225-90cd-244060765bbc-kube-api-access-wwk7p\") pod \"test-model-8fef73-kserve-54ff956b49-429dm\" (UID: \"27700938-db52-4225-90cd-244060765bbc\") " pod="ai-tenant-e2e-inf-a-8fef73/test-model-8fef73-kserve-54ff956b49-429dm"
Oct 30 07:46:50.063715 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:50.063565    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27700938-db52-4225-90cd-244060765bbc-tmp-dir\") pod \"test-model-8fef73-kserve-54ff956b49-429dm\" (UID: \"27700938-db52-4225-90cd-244060765bbc\") " pod="ai-tenant-e2e-inf-a-8fef73/test-model-8fef73-kserve-54ff956b49-429dm"
Oct 30 07:46:50.064115 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:50.063726    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27700938-db52-4225-90cd-244060765bbc-model-cache\") pod \"test-model-8fef73-kserve-54ff956b49-429dm\" (UID: \"27700938-db52-4225-90cd-244060765bbc\") " pod="ai-tenant-e2e-inf-a-8fef73/test-model-8fef73-kserve-54ff956b49-429dm"
Oct 30 07:46:50.064184 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:50.064121    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27700938-db52-4225-90cd-244060765bbc-tmp-dir\") pod \"test-model-8fef73-kserve-54ff956b49-429dm\" (UID: \"27700938-db52-4225-90cd-244060765bbc\") " pod="ai-tenant-e2e-inf-a-8fef73/test-model-8fef73-kserve-54ff956b49-429dm"
Oct 30 07:46:50.064184 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:50.064152    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27700938-db52-4225-90cd-244060765bbc-home\") pod \"test-model-8fef73-kserve-54ff956b49-429dm\" (UID: \"27700938-db52-4225-90cd-244060765bbc\") " pod="ai-tenant-e2e-inf-a-8fef73/test-model-8fef73-kserve-54ff956b49-429dm"
Oct 30 07:46:50.064184 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:50.064177    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27700938-db52-4225-90cd-244060765bbc-model-cache\") pod \"test-model-8fef73-kserve-54ff956b49-429dm\" (UID: \"27700938-db52-4225-90cd-244060765bbc\") " pod="ai-tenant-e2e-inf-a-8fef73/test-model-8fef73-kserve-54ff956b49-429dm"
Oct 30 07:46:50.066192 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:50.066167    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27700938-db52-4225-90cd-244060765bbc-tls-certs\") pod \"test-model-8fef73-kserve-54ff956b49-429dm\" (UID: \"27700938-db52-4225-90cd-244060765bbc\") " pod="ai-tenant-e2e-inf-a-8fef73/test-model-8fef73-kserve-54ff956b49-429dm"
Oct 30 07:46:50.066373 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:50.066351    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27700938-db52-4225-90cd-244060765bbc-dshm\") pod \"test-model-8fef73-kserve-54ff956b49-429dm\" (UID: \"27700938-db52-4225-90cd-244060765bbc\") " pod="ai-tenant-e2e-inf-a-8fef73/test-model-8fef73-kserve-54ff956b49-429dm"
Oct 30 07:46:50.088970 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:50.088942    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwk7p\" (UniqueName: \"kubernetes.io/projected/27700938-db52-4225-90cd-244060765bbc-kube-api-access-wwk7p\") pod \"test-model-8fef73-kserve-54ff956b49-429dm\" (UID: \"27700938-db52-4225-90cd-244060765bbc\") " pod="ai-tenant-e2e-inf-a-8fef73/test-model-8fef73-kserve-54ff956b49-429dm"
Oct 30 07:46:50.185882 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:50.185105    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8fef73/test-model-8fef73-kserve-54ff956b49-429dm"
Oct 30 07:46:50.240334 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:50.240168    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:46:50.536245 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:46:50.536213    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod27700938_db52_4225_90cd_244060765bbc.slice/crio-465613d3ba8239d5cf53cbd416d94cac8de8f86289dd8ae636fb87bb82228caf WatchSource:0}: Error finding container 465613d3ba8239d5cf53cbd416d94cac8de8f86289dd8ae636fb87bb82228caf: Status 404 returned error can't find the container with id 465613d3ba8239d5cf53cbd416d94cac8de8f86289dd8ae636fb87bb82228caf
Oct 30 07:46:50.538313 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:50.538288    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-8fef73/test-model-8fef73-kserve-54ff956b49-429dm"]
Oct 30 07:46:51.451320 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:51.451282    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8fef73/test-model-8fef73-kserve-54ff956b49-429dm" event={"ID":"27700938-db52-4225-90cd-244060765bbc","Type":"ContainerStarted","Data":"437cec0b042f64cfaf3cca470402569407e84494291aa5cfc4913d42f1a1cdc2"}
Oct 30 07:46:51.451846 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:51.451335    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8fef73/test-model-8fef73-kserve-54ff956b49-429dm" event={"ID":"27700938-db52-4225-90cd-244060765bbc","Type":"ContainerStarted","Data":"465613d3ba8239d5cf53cbd416d94cac8de8f86289dd8ae636fb87bb82228caf"}
Oct 30 07:46:51.500697 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:51.500642    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-8fef73/test-model-8fef73-kserve-54ff956b49-429dm" podStartSLOduration=2.500624741 podStartE2EDuration="2.500624741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:46:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:46:51.496547309 +0000 UTC m=+2970.105954279" watchObservedRunningTime="2026-10-30 07:46:51.500624741 +0000 UTC m=+2970.110031711"
Oct 30 07:46:52.185851 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:52.185816    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-8fef73/test-model-8fef73-kserve-54ff956b49-429dm"
Oct 30 07:46:52.469435 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:52.469345    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8c0e05-697c896c55-l22m2"]
Oct 30 07:46:52.474157 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:52.474132    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8c0e05-697c896c55-l22m2"
Oct 30 07:46:52.499223 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:52.497694    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-8c0e05-dockercfg-9q2g2\""
Oct 30 07:46:52.499457 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:52.497966    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8c0e05-697c896c55-l22m2"]
Oct 30 07:46:52.509545 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:52.506724    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-8c0e05\""
Oct 30 07:46:52.623749 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:52.613352    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f78gh\" (UniqueName: \"kubernetes.io/projected/eb72e294-0efe-4aef-80e7-240321871b92-kube-api-access-f78gh\") pod \"payload-pre-processing-e2e-inf-b-8c0e05-697c896c55-l22m2\" (UID: \"eb72e294-0efe-4aef-80e7-240321871b92\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8c0e05-697c896c55-l22m2"
Oct 30 07:46:52.623749 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:52.613583    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb72e294-0efe-4aef-80e7-240321871b92-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8c0e05-697c896c55-l22m2\" (UID: \"eb72e294-0efe-4aef-80e7-240321871b92\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8c0e05-697c896c55-l22m2"
Oct 30 07:46:52.655194 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:52.655157    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8c0e05-594d698444-8sp4q"]
Oct 30 07:46:52.691134 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:52.669603    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8c0e05-594d698444-8sp4q"
Oct 30 07:46:52.691134 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:52.689153    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8c0e05-594d698444-8sp4q"]
Oct 30 07:46:52.717430 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:52.717189    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb72e294-0efe-4aef-80e7-240321871b92-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8c0e05-697c896c55-l22m2\" (UID: \"eb72e294-0efe-4aef-80e7-240321871b92\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8c0e05-697c896c55-l22m2"
Oct 30 07:46:52.717605 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:52.717454    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b4d7b70-3af7-41dd-97f4-b62acb3215d8-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8c0e05-594d698444-8sp4q\" (UID: \"5b4d7b70-3af7-41dd-97f4-b62acb3215d8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8c0e05-594d698444-8sp4q"
Oct 30 07:46:52.717605 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:52.717513    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f78gh\" (UniqueName: \"kubernetes.io/projected/eb72e294-0efe-4aef-80e7-240321871b92-kube-api-access-f78gh\") pod \"payload-pre-processing-e2e-inf-b-8c0e05-697c896c55-l22m2\" (UID: \"eb72e294-0efe-4aef-80e7-240321871b92\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8c0e05-697c896c55-l22m2"
Oct 30 07:46:52.717743 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:52.717622    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzn7r\" (UniqueName: \"kubernetes.io/projected/5b4d7b70-3af7-41dd-97f4-b62acb3215d8-kube-api-access-jzn7r\") pod \"payload-processing-e2e-inf-b-8c0e05-594d698444-8sp4q\" (UID: \"5b4d7b70-3af7-41dd-97f4-b62acb3215d8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8c0e05-594d698444-8sp4q"
Oct 30 07:46:52.718613 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:52.718580    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb72e294-0efe-4aef-80e7-240321871b92-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8c0e05-697c896c55-l22m2\" (UID: \"eb72e294-0efe-4aef-80e7-240321871b92\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8c0e05-697c896c55-l22m2"
Oct 30 07:46:52.746069 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:52.746007    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f78gh\" (UniqueName: \"kubernetes.io/projected/eb72e294-0efe-4aef-80e7-240321871b92-kube-api-access-f78gh\") pod \"payload-pre-processing-e2e-inf-b-8c0e05-697c896c55-l22m2\" (UID: \"eb72e294-0efe-4aef-80e7-240321871b92\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8c0e05-697c896c55-l22m2"
Oct 30 07:46:52.818557 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:52.818523    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b4d7b70-3af7-41dd-97f4-b62acb3215d8-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8c0e05-594d698444-8sp4q\" (UID: \"5b4d7b70-3af7-41dd-97f4-b62acb3215d8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8c0e05-594d698444-8sp4q"
Oct 30 07:46:52.818764 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:52.818628    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzn7r\" (UniqueName: \"kubernetes.io/projected/5b4d7b70-3af7-41dd-97f4-b62acb3215d8-kube-api-access-jzn7r\") pod \"payload-processing-e2e-inf-b-8c0e05-594d698444-8sp4q\" (UID: \"5b4d7b70-3af7-41dd-97f4-b62acb3215d8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8c0e05-594d698444-8sp4q"
Oct 30 07:46:52.819353 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:52.819329    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b4d7b70-3af7-41dd-97f4-b62acb3215d8-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8c0e05-594d698444-8sp4q\" (UID: \"5b4d7b70-3af7-41dd-97f4-b62acb3215d8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8c0e05-594d698444-8sp4q"
Oct 30 07:46:52.819537 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:52.819521    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8c0e05-697c896c55-l22m2"
Oct 30 07:46:52.843781 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:52.843756    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzn7r\" (UniqueName: \"kubernetes.io/projected/5b4d7b70-3af7-41dd-97f4-b62acb3215d8-kube-api-access-jzn7r\") pod \"payload-processing-e2e-inf-b-8c0e05-594d698444-8sp4q\" (UID: \"5b4d7b70-3af7-41dd-97f4-b62acb3215d8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8c0e05-594d698444-8sp4q"
Oct 30 07:46:53.014213 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:53.014127    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8c0e05-594d698444-8sp4q"
Oct 30 07:46:53.145384 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:46:53.145340    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb72e294_0efe_4aef_80e7_240321871b92.slice/crio-bd4d3c38a0f964b4c0222b70a0e27b204f061c6ccd726bae9343b7043db9084d WatchSource:0}: Error finding container bd4d3c38a0f964b4c0222b70a0e27b204f061c6ccd726bae9343b7043db9084d: Status 404 returned error can't find the container with id bd4d3c38a0f964b4c0222b70a0e27b204f061c6ccd726bae9343b7043db9084d
Oct 30 07:46:53.149363 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:53.148117    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8c0e05-697c896c55-l22m2"]
Oct 30 07:46:53.335458 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:46:53.335419    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b4d7b70_3af7_41dd_97f4_b62acb3215d8.slice/crio-a09bf6f48893251287cdfb91f542879d4fd29407d0a0f91a90f0019279d348f6 WatchSource:0}: Error finding container a09bf6f48893251287cdfb91f542879d4fd29407d0a0f91a90f0019279d348f6: Status 404 returned error can't find the container with id a09bf6f48893251287cdfb91f542879d4fd29407d0a0f91a90f0019279d348f6
Oct 30 07:46:53.336884 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:53.336857    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8c0e05-594d698444-8sp4q"]
Oct 30 07:46:53.380098 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:53.380066    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0e9723-54cb7ddd4-jnzdg"]
Oct 30 07:46:53.386888 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:53.386864    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0e9723-54cb7ddd4-jnzdg"
Oct 30 07:46:53.405051 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:53.405023    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-0e9723\""
Oct 30 07:46:53.406015 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:53.405943    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0e9723-54cb7ddd4-jnzdg"]
Oct 30 07:46:53.483976 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:53.483934    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8c0e05-594d698444-8sp4q" event={"ID":"5b4d7b70-3af7-41dd-97f4-b62acb3215d8","Type":"ContainerStarted","Data":"ba3022eed8ed84acb0232dc553407ef2952194729841c61244e02ee2d234f29f"}
Oct 30 07:46:53.484374 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:53.484056    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8c0e05-594d698444-8sp4q" event={"ID":"5b4d7b70-3af7-41dd-97f4-b62acb3215d8","Type":"ContainerStarted","Data":"a09bf6f48893251287cdfb91f542879d4fd29407d0a0f91a90f0019279d348f6"}
Oct 30 07:46:53.484374 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:53.484172    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-8c0e05-594d698444-8sp4q"
Oct 30 07:46:53.486705 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:53.486678    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8c0e05-697c896c55-l22m2" event={"ID":"eb72e294-0efe-4aef-80e7-240321871b92","Type":"ContainerStarted","Data":"da82dbbb437e1c93722bd2a97d702aa47b4fad49fde2532e8c9fa41ecc1f32e6"}
Oct 30 07:46:53.486855 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:53.486728    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8c0e05-697c896c55-l22m2"
Oct 30 07:46:53.486855 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:53.486749    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8c0e05-697c896c55-l22m2" event={"ID":"eb72e294-0efe-4aef-80e7-240321871b92","Type":"ContainerStarted","Data":"bd4d3c38a0f964b4c0222b70a0e27b204f061c6ccd726bae9343b7043db9084d"}
Oct 30 07:46:53.531843 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:53.531753    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-8c0e05-594d698444-8sp4q" podStartSLOduration=1.531711721 podStartE2EDuration="1.531711721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:46:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:46:53.531058148 +0000 UTC m=+2972.140465118" watchObservedRunningTime="2026-10-30 07:46:53.531711721 +0000 UTC m=+2972.141118684"
Oct 30 07:46:53.538485 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:53.538450    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6br7\" (UniqueName: \"kubernetes.io/projected/46b5c1df-3984-4df3-8d60-8f4bca4228f4-kube-api-access-q6br7\") pod \"payload-processing-e2e-inf-a-0e9723-54cb7ddd4-jnzdg\" (UID: \"46b5c1df-3984-4df3-8d60-8f4bca4228f4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0e9723-54cb7ddd4-jnzdg"
Oct 30 07:46:53.538643 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:53.538504    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46b5c1df-3984-4df3-8d60-8f4bca4228f4-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0e9723-54cb7ddd4-jnzdg\" (UID: \"46b5c1df-3984-4df3-8d60-8f4bca4228f4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0e9723-54cb7ddd4-jnzdg"
Oct 30 07:46:53.599854 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:53.586277    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8c0e05-697c896c55-l22m2" podStartSLOduration=1.586259274 podStartE2EDuration="1.586259274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:46:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:46:53.577238633 +0000 UTC m=+2972.186645606" watchObservedRunningTime="2026-10-30 07:46:53.586259274 +0000 UTC m=+2972.195666245"
Oct 30 07:46:53.643026 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:53.642960    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6br7\" (UniqueName: \"kubernetes.io/projected/46b5c1df-3984-4df3-8d60-8f4bca4228f4-kube-api-access-q6br7\") pod \"payload-processing-e2e-inf-a-0e9723-54cb7ddd4-jnzdg\" (UID: \"46b5c1df-3984-4df3-8d60-8f4bca4228f4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0e9723-54cb7ddd4-jnzdg"
Oct 30 07:46:53.643338 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:53.643306    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46b5c1df-3984-4df3-8d60-8f4bca4228f4-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0e9723-54cb7ddd4-jnzdg\" (UID: \"46b5c1df-3984-4df3-8d60-8f4bca4228f4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0e9723-54cb7ddd4-jnzdg"
Oct 30 07:46:53.654500 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:53.654464    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46b5c1df-3984-4df3-8d60-8f4bca4228f4-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0e9723-54cb7ddd4-jnzdg\" (UID: \"46b5c1df-3984-4df3-8d60-8f4bca4228f4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0e9723-54cb7ddd4-jnzdg"
Oct 30 07:46:53.698465 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:53.692401    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6br7\" (UniqueName: \"kubernetes.io/projected/46b5c1df-3984-4df3-8d60-8f4bca4228f4-kube-api-access-q6br7\") pod \"payload-processing-e2e-inf-a-0e9723-54cb7ddd4-jnzdg\" (UID: \"46b5c1df-3984-4df3-8d60-8f4bca4228f4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0e9723-54cb7ddd4-jnzdg"
Oct 30 07:46:53.709694 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:53.706503    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0e9723-54cb7ddd4-jnzdg"
Oct 30 07:46:54.030363 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:46:54.030335    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46b5c1df_3984_4df3_8d60_8f4bca4228f4.slice/crio-f17ff19544e35106c031fdfbfb93123bd6b22c34e6c516159753fa18410878c7 WatchSource:0}: Error finding container f17ff19544e35106c031fdfbfb93123bd6b22c34e6c516159753fa18410878c7: Status 404 returned error can't find the container with id f17ff19544e35106c031fdfbfb93123bd6b22c34e6c516159753fa18410878c7
Oct 30 07:46:54.033647 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:54.033620    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0e9723-54cb7ddd4-jnzdg"]
Oct 30 07:46:54.496339 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:54.496243    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0e9723-54cb7ddd4-jnzdg" event={"ID":"46b5c1df-3984-4df3-8d60-8f4bca4228f4","Type":"ContainerStarted","Data":"b20f1aa38d12029b8acf62f9f5e3604167b20739971cec79b96bf8baae6de039"}
Oct 30 07:46:54.496339 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:54.496305    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0e9723-54cb7ddd4-jnzdg" event={"ID":"46b5c1df-3984-4df3-8d60-8f4bca4228f4","Type":"ContainerStarted","Data":"f17ff19544e35106c031fdfbfb93123bd6b22c34e6c516159753fa18410878c7"}
Oct 30 07:46:54.496959 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:54.496533    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-0e9723-54cb7ddd4-jnzdg"
Oct 30 07:46:54.542675 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:46:54.542626    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-0e9723-54cb7ddd4-jnzdg" podStartSLOduration=1.542612468 podStartE2EDuration="1.542612468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:46:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:46:54.539628218 +0000 UTC m=+2973.149035188" watchObservedRunningTime="2026-10-30 07:46:54.542612468 +0000 UTC m=+2973.152019462"
Oct 30 07:47:00.186609 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:00.186575    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-8fef73/test-model-8fef73-kserve-54ff956b49-429dm"
Oct 30 07:47:00.201423 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:00.201390    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-8fef73/test-model-8fef73-kserve-54ff956b49-429dm"
Oct 30 07:47:00.552906 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:00.552875    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-8fef73/test-model-8fef73-kserve-54ff956b49-429dm"
Oct 30 07:47:01.859638 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:47:01.859602    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:47:04.499594 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:04.499515    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8c0e05-697c896c55-l22m2"
Oct 30 07:47:04.500074 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:04.499606    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-8c0e05-594d698444-8sp4q"
Oct 30 07:47:05.507248 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:05.507216    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-0e9723-54cb7ddd4-jnzdg"
Oct 30 07:47:14.422726 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:14.422693    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-f480c6/test-model-f480c6-kserve-7bbd97ff6f-75xld"]
Oct 30 07:47:14.430099 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:14.430072    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f480c6/test-model-f480c6-kserve-7bbd97ff6f-75xld"
Oct 30 07:47:14.454280 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:14.454253    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f480c6/test-model-f480c6-kserve-7bbd97ff6f-75xld"]
Oct 30 07:47:14.463050 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:14.463018    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f480c6\"/\"kube-root-ca.crt\""
Oct 30 07:47:14.463185 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:14.463072    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f480c6\"/\"openshift-service-ca.crt\""
Oct 30 07:47:14.463185 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:14.463153    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f480c6\"/\"default-dockercfg-vswn7\""
Oct 30 07:47:14.463313 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:14.463236    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f480c6\"/\"test-model-f480c6-kserve-self-signed-certs\""
Oct 30 07:47:14.564055 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:14.564018    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gl9p\" (UniqueName: \"kubernetes.io/projected/2ffa4b03-7acf-4a5c-9045-59f171bb9bfa-kube-api-access-7gl9p\") pod \"test-model-f480c6-kserve-7bbd97ff6f-75xld\" (UID: \"2ffa4b03-7acf-4a5c-9045-59f171bb9bfa\") " pod="ai-tenant-e2e-inf-b-f480c6/test-model-f480c6-kserve-7bbd97ff6f-75xld"
Oct 30 07:47:14.564256 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:14.564065    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ffa4b03-7acf-4a5c-9045-59f171bb9bfa-model-cache\") pod \"test-model-f480c6-kserve-7bbd97ff6f-75xld\" (UID: \"2ffa4b03-7acf-4a5c-9045-59f171bb9bfa\") " pod="ai-tenant-e2e-inf-b-f480c6/test-model-f480c6-kserve-7bbd97ff6f-75xld"
Oct 30 07:47:14.564256 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:14.564180    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ffa4b03-7acf-4a5c-9045-59f171bb9bfa-home\") pod \"test-model-f480c6-kserve-7bbd97ff6f-75xld\" (UID: \"2ffa4b03-7acf-4a5c-9045-59f171bb9bfa\") " pod="ai-tenant-e2e-inf-b-f480c6/test-model-f480c6-kserve-7bbd97ff6f-75xld"
Oct 30 07:47:14.564398 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:14.564292    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ffa4b03-7acf-4a5c-9045-59f171bb9bfa-tmp-dir\") pod \"test-model-f480c6-kserve-7bbd97ff6f-75xld\" (UID: \"2ffa4b03-7acf-4a5c-9045-59f171bb9bfa\") " pod="ai-tenant-e2e-inf-b-f480c6/test-model-f480c6-kserve-7bbd97ff6f-75xld"
Oct 30 07:47:14.564398 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:14.564311    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ffa4b03-7acf-4a5c-9045-59f171bb9bfa-tls-certs\") pod \"test-model-f480c6-kserve-7bbd97ff6f-75xld\" (UID: \"2ffa4b03-7acf-4a5c-9045-59f171bb9bfa\") " pod="ai-tenant-e2e-inf-b-f480c6/test-model-f480c6-kserve-7bbd97ff6f-75xld"
Oct 30 07:47:14.564398 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:14.564339    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ffa4b03-7acf-4a5c-9045-59f171bb9bfa-dshm\") pod \"test-model-f480c6-kserve-7bbd97ff6f-75xld\" (UID: \"2ffa4b03-7acf-4a5c-9045-59f171bb9bfa\") " pod="ai-tenant-e2e-inf-b-f480c6/test-model-f480c6-kserve-7bbd97ff6f-75xld"
Oct 30 07:47:14.665746 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:14.665708    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gl9p\" (UniqueName: \"kubernetes.io/projected/2ffa4b03-7acf-4a5c-9045-59f171bb9bfa-kube-api-access-7gl9p\") pod \"test-model-f480c6-kserve-7bbd97ff6f-75xld\" (UID: \"2ffa4b03-7acf-4a5c-9045-59f171bb9bfa\") " pod="ai-tenant-e2e-inf-b-f480c6/test-model-f480c6-kserve-7bbd97ff6f-75xld"
Oct 30 07:47:14.668231 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:14.668202    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ffa4b03-7acf-4a5c-9045-59f171bb9bfa-model-cache\") pod \"test-model-f480c6-kserve-7bbd97ff6f-75xld\" (UID: \"2ffa4b03-7acf-4a5c-9045-59f171bb9bfa\") " pod="ai-tenant-e2e-inf-b-f480c6/test-model-f480c6-kserve-7bbd97ff6f-75xld"
Oct 30 07:47:14.668881 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:14.668859    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ffa4b03-7acf-4a5c-9045-59f171bb9bfa-home\") pod \"test-model-f480c6-kserve-7bbd97ff6f-75xld\" (UID: \"2ffa4b03-7acf-4a5c-9045-59f171bb9bfa\") " pod="ai-tenant-e2e-inf-b-f480c6/test-model-f480c6-kserve-7bbd97ff6f-75xld"
Oct 30 07:47:14.669498 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:14.669479    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ffa4b03-7acf-4a5c-9045-59f171bb9bfa-tmp-dir\") pod \"test-model-f480c6-kserve-7bbd97ff6f-75xld\" (UID: \"2ffa4b03-7acf-4a5c-9045-59f171bb9bfa\") " pod="ai-tenant-e2e-inf-b-f480c6/test-model-f480c6-kserve-7bbd97ff6f-75xld"
Oct 30 07:47:14.669955 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:14.669938    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ffa4b03-7acf-4a5c-9045-59f171bb9bfa-tls-certs\") pod \"test-model-f480c6-kserve-7bbd97ff6f-75xld\" (UID: \"2ffa4b03-7acf-4a5c-9045-59f171bb9bfa\") " pod="ai-tenant-e2e-inf-b-f480c6/test-model-f480c6-kserve-7bbd97ff6f-75xld"
Oct 30 07:47:14.670520 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:14.670499    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ffa4b03-7acf-4a5c-9045-59f171bb9bfa-dshm\") pod \"test-model-f480c6-kserve-7bbd97ff6f-75xld\" (UID: \"2ffa4b03-7acf-4a5c-9045-59f171bb9bfa\") " pod="ai-tenant-e2e-inf-b-f480c6/test-model-f480c6-kserve-7bbd97ff6f-75xld"
Oct 30 07:47:14.670911 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:14.669885    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ffa4b03-7acf-4a5c-9045-59f171bb9bfa-tmp-dir\") pod \"test-model-f480c6-kserve-7bbd97ff6f-75xld\" (UID: \"2ffa4b03-7acf-4a5c-9045-59f171bb9bfa\") " pod="ai-tenant-e2e-inf-b-f480c6/test-model-f480c6-kserve-7bbd97ff6f-75xld"
Oct 30 07:47:14.671036 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:14.669289    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ffa4b03-7acf-4a5c-9045-59f171bb9bfa-home\") pod \"test-model-f480c6-kserve-7bbd97ff6f-75xld\" (UID: \"2ffa4b03-7acf-4a5c-9045-59f171bb9bfa\") " pod="ai-tenant-e2e-inf-b-f480c6/test-model-f480c6-kserve-7bbd97ff6f-75xld"
Oct 30 07:47:14.671231 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:14.668769    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ffa4b03-7acf-4a5c-9045-59f171bb9bfa-model-cache\") pod \"test-model-f480c6-kserve-7bbd97ff6f-75xld\" (UID: \"2ffa4b03-7acf-4a5c-9045-59f171bb9bfa\") " pod="ai-tenant-e2e-inf-b-f480c6/test-model-f480c6-kserve-7bbd97ff6f-75xld"
Oct 30 07:47:14.677868 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:14.677794    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ffa4b03-7acf-4a5c-9045-59f171bb9bfa-tls-certs\") pod \"test-model-f480c6-kserve-7bbd97ff6f-75xld\" (UID: \"2ffa4b03-7acf-4a5c-9045-59f171bb9bfa\") " pod="ai-tenant-e2e-inf-b-f480c6/test-model-f480c6-kserve-7bbd97ff6f-75xld"
Oct 30 07:47:14.708116 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:14.708080    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ffa4b03-7acf-4a5c-9045-59f171bb9bfa-dshm\") pod \"test-model-f480c6-kserve-7bbd97ff6f-75xld\" (UID: \"2ffa4b03-7acf-4a5c-9045-59f171bb9bfa\") " pod="ai-tenant-e2e-inf-b-f480c6/test-model-f480c6-kserve-7bbd97ff6f-75xld"
Oct 30 07:47:14.724198 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:14.717461    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gl9p\" (UniqueName: \"kubernetes.io/projected/2ffa4b03-7acf-4a5c-9045-59f171bb9bfa-kube-api-access-7gl9p\") pod \"test-model-f480c6-kserve-7bbd97ff6f-75xld\" (UID: \"2ffa4b03-7acf-4a5c-9045-59f171bb9bfa\") " pod="ai-tenant-e2e-inf-b-f480c6/test-model-f480c6-kserve-7bbd97ff6f-75xld"
Oct 30 07:47:14.762346 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:14.757085    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f480c6/test-model-f480c6-kserve-7bbd97ff6f-75xld"
Oct 30 07:47:15.101084 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:15.101056    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f480c6/test-model-f480c6-kserve-7bbd97ff6f-75xld"]
Oct 30 07:47:15.725111 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:15.725076    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f480c6/test-model-f480c6-kserve-7bbd97ff6f-75xld" event={"ID":"2ffa4b03-7acf-4a5c-9045-59f171bb9bfa","Type":"ContainerStarted","Data":"3449da77d4786fc38c0f9ae2bd5ffe75c5c1a5133510244a7989b4c984ca8974"}
Oct 30 07:47:15.725527 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:15.725134    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f480c6/test-model-f480c6-kserve-7bbd97ff6f-75xld" event={"ID":"2ffa4b03-7acf-4a5c-9045-59f171bb9bfa","Type":"ContainerStarted","Data":"3c25a12514fa57986a6be01f53ce41305c9e761e57b77707e30a32dae69e7143"}
Oct 30 07:47:15.762864 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:15.762833    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-f480c6/test-model-f480c6-kserve-7bbd97ff6f-75xld"
Oct 30 07:47:15.782163 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:15.782114    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-f480c6/test-model-f480c6-kserve-7bbd97ff6f-75xld" podStartSLOduration=1.782100923 podStartE2EDuration="1.782100923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:47:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:47:15.778418728 +0000 UTC m=+2994.387825700" watchObservedRunningTime="2026-10-30 07:47:15.782100923 +0000 UTC m=+2994.391507887"
Oct 30 07:47:24.762496 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:24.762460    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-f480c6/test-model-f480c6-kserve-7bbd97ff6f-75xld"
Oct 30 07:47:24.780654 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:24.780622    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-f480c6/test-model-f480c6-kserve-7bbd97ff6f-75xld"
Oct 30 07:47:24.811962 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:24.811936    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-f480c6/test-model-f480c6-kserve-7bbd97ff6f-75xld"
Oct 30 07:47:26.538923 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:26.538887    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5"]
Oct 30 07:47:26.539378 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:26.539076    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5" podUID="d0c358f8-4625-4498-8322-c843e5eeb017" containerName="istio-proxy" containerID="cri-o://2da2762b04296e23f620839c5d9e9fa750a7720bbfac1ec922a59d0807d43909" gracePeriod=30
Oct 30 07:47:28.942253 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:28.942220    6224 patch_prober.go:29] interesting pod/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 07:47:28.942726 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:28.942292    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5" podUID="d0c358f8-4625-4498-8322-c843e5eeb017" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 07:47:31.821879 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:31.821798    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5"
Oct 30 07:47:31.861472 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:31.861436    6224 generic.go:356] "Generic (PLEG): container finished" podID="d0c358f8-4625-4498-8322-c843e5eeb017" containerID="2da2762b04296e23f620839c5d9e9fa750a7720bbfac1ec922a59d0807d43909" exitCode=0
Oct 30 07:47:31.861670 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:31.861496    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5"
Oct 30 07:47:31.861670 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:31.861510    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5" event={"ID":"d0c358f8-4625-4498-8322-c843e5eeb017","Type":"ContainerDied","Data":"2da2762b04296e23f620839c5d9e9fa750a7720bbfac1ec922a59d0807d43909"}
Oct 30 07:47:31.861670 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:31.861570    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5" event={"ID":"d0c358f8-4625-4498-8322-c843e5eeb017","Type":"ContainerDied","Data":"b7e9027551366c54782fb6711b13bac524e212650bf4dd5179c9b94a8c78f85c"}
Oct 30 07:47:31.861670 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:31.861594    6224 scope.go:122] "RemoveContainer" containerID="2da2762b04296e23f620839c5d9e9fa750a7720bbfac1ec922a59d0807d43909"
Oct 30 07:47:31.865966 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:47:31.865941    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:47:31.878753 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:31.878732    6224 scope.go:122] "RemoveContainer" containerID="2da2762b04296e23f620839c5d9e9fa750a7720bbfac1ec922a59d0807d43909"
Oct 30 07:47:31.879093 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:47:31.879064    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2da2762b04296e23f620839c5d9e9fa750a7720bbfac1ec922a59d0807d43909\": container with ID starting with 2da2762b04296e23f620839c5d9e9fa750a7720bbfac1ec922a59d0807d43909 not found: ID does not exist" containerID="2da2762b04296e23f620839c5d9e9fa750a7720bbfac1ec922a59d0807d43909"
Oct 30 07:47:31.879204 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:31.879100    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2da2762b04296e23f620839c5d9e9fa750a7720bbfac1ec922a59d0807d43909"} err="failed to get container status \"2da2762b04296e23f620839c5d9e9fa750a7720bbfac1ec922a59d0807d43909\": rpc error: code = NotFound desc = could not find container \"2da2762b04296e23f620839c5d9e9fa750a7720bbfac1ec922a59d0807d43909\": container with ID starting with 2da2762b04296e23f620839c5d9e9fa750a7720bbfac1ec922a59d0807d43909 not found: ID does not exist"
Oct 30 07:47:31.932084 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:31.932037    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0c358f8-4625-4498-8322-c843e5eeb017-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c358f8-4625-4498-8322-c843e5eeb017-workload-socket\") pod \"d0c358f8-4625-4498-8322-c843e5eeb017\" (UID: \"d0c358f8-4625-4498-8322-c843e5eeb017\") "
Oct 30 07:47:31.932286 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:31.932095    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0c358f8-4625-4498-8322-c843e5eeb017-istio-token\" (UniqueName: \"kubernetes.io/projected/d0c358f8-4625-4498-8322-c843e5eeb017-istio-token\") pod \"d0c358f8-4625-4498-8322-c843e5eeb017\" (UID: \"d0c358f8-4625-4498-8322-c843e5eeb017\") "
Oct 30 07:47:31.932286 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:31.932128    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d0c358f8-4625-4498-8322-c843e5eeb017-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0c358f8-4625-4498-8322-c843e5eeb017-istio-podinfo\") pod \"d0c358f8-4625-4498-8322-c843e5eeb017\" (UID: \"d0c358f8-4625-4498-8322-c843e5eeb017\") "
Oct 30 07:47:31.932286 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:31.932240    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0c358f8-4625-4498-8322-c843e5eeb017-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0c358f8-4625-4498-8322-c843e5eeb017-workload-certs\") pod \"d0c358f8-4625-4498-8322-c843e5eeb017\" (UID: \"d0c358f8-4625-4498-8322-c843e5eeb017\") "
Oct 30 07:47:31.932482 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:31.932327    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0c358f8-4625-4498-8322-c843e5eeb017-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0c358f8-4625-4498-8322-c843e5eeb017-istio-data\") pod \"d0c358f8-4625-4498-8322-c843e5eeb017\" (UID: \"d0c358f8-4625-4498-8322-c843e5eeb017\") "
Oct 30 07:47:31.932482 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:31.932386    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c358f8-4625-4498-8322-c843e5eeb017-workload-socket" pod "d0c358f8-4625-4498-8322-c843e5eeb017" (UID: "d0c358f8-4625-4498-8322-c843e5eeb017"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:47:31.932482 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:31.932424    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0c358f8-4625-4498-8322-c843e5eeb017-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0c358f8-4625-4498-8322-c843e5eeb017-istio-envoy\") pod \"d0c358f8-4625-4498-8322-c843e5eeb017\" (UID: \"d0c358f8-4625-4498-8322-c843e5eeb017\") "
Oct 30 07:47:31.932664 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:31.932518    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0c358f8-4625-4498-8322-c843e5eeb017-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0c358f8-4625-4498-8322-c843e5eeb017-istiod-ca-cert\") pod \"d0c358f8-4625-4498-8322-c843e5eeb017\" (UID: \"d0c358f8-4625-4498-8322-c843e5eeb017\") "
Oct 30 07:47:31.932664 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:31.932574    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0c358f8-4625-4498-8322-c843e5eeb017-kube-api-access-dpszx\" (UniqueName: \"kubernetes.io/projected/d0c358f8-4625-4498-8322-c843e5eeb017-kube-api-access-dpszx\") pod \"d0c358f8-4625-4498-8322-c843e5eeb017\" (UID: \"d0c358f8-4625-4498-8322-c843e5eeb017\") "
Oct 30 07:47:31.932664 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:31.932615    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c358f8-4625-4498-8322-c843e5eeb017-workload-certs" pod "d0c358f8-4625-4498-8322-c843e5eeb017" (UID: "d0c358f8-4625-4498-8322-c843e5eeb017"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:47:31.932855 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:31.932623    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0c358f8-4625-4498-8322-c843e5eeb017-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c358f8-4625-4498-8322-c843e5eeb017-credential-socket\") pod \"d0c358f8-4625-4498-8322-c843e5eeb017\" (UID: \"d0c358f8-4625-4498-8322-c843e5eeb017\") "
Oct 30 07:47:31.932855 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:31.932705    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c358f8-4625-4498-8322-c843e5eeb017-istio-data" pod "d0c358f8-4625-4498-8322-c843e5eeb017" (UID: "d0c358f8-4625-4498-8322-c843e5eeb017"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:47:31.932978 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:31.932873    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c358f8-4625-4498-8322-c843e5eeb017-credential-socket" pod "d0c358f8-4625-4498-8322-c843e5eeb017" (UID: "d0c358f8-4625-4498-8322-c843e5eeb017"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:47:31.933105 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:31.933078    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0c358f8-4625-4498-8322-c843e5eeb017-istiod-ca-cert" pod "d0c358f8-4625-4498-8322-c843e5eeb017" (UID: "d0c358f8-4625-4498-8322-c843e5eeb017"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:47:31.933471 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:31.933446    6224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0c358f8-4625-4498-8322-c843e5eeb017-istiod-ca-cert\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:47:31.933633 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:31.933615    6224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c358f8-4625-4498-8322-c843e5eeb017-credential-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:47:31.933745 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:31.933730    6224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c358f8-4625-4498-8322-c843e5eeb017-workload-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:47:31.933871 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:31.933856    6224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0c358f8-4625-4498-8322-c843e5eeb017-workload-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:47:31.933973 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:31.933962    6224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0c358f8-4625-4498-8322-c843e5eeb017-istio-data\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:47:31.935481 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:31.935435    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c358f8-4625-4498-8322-c843e5eeb017-istio-envoy" pod "d0c358f8-4625-4498-8322-c843e5eeb017" (UID: "d0c358f8-4625-4498-8322-c843e5eeb017"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:47:31.935592 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:31.935505    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0c358f8-4625-4498-8322-c843e5eeb017-istio-podinfo" pod "d0c358f8-4625-4498-8322-c843e5eeb017" (UID: "d0c358f8-4625-4498-8322-c843e5eeb017"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:47:31.935664 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:31.935604    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0c358f8-4625-4498-8322-c843e5eeb017-istio-token" pod "d0c358f8-4625-4498-8322-c843e5eeb017" (UID: "d0c358f8-4625-4498-8322-c843e5eeb017"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:47:31.935726 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:31.935693    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0c358f8-4625-4498-8322-c843e5eeb017-kube-api-access-dpszx" pod "d0c358f8-4625-4498-8322-c843e5eeb017" (UID: "d0c358f8-4625-4498-8322-c843e5eeb017"). InnerVolumeSpecName "kube-api-access-dpszx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:47:32.035672 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:32.035635    6224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0c358f8-4625-4498-8322-c843e5eeb017-istio-envoy\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:47:32.035672 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:32.035667    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpszx\" (UniqueName: \"kubernetes.io/projected/d0c358f8-4625-4498-8322-c843e5eeb017-kube-api-access-dpszx\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:47:32.035672 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:32.035677    6224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0c358f8-4625-4498-8322-c843e5eeb017-istio-token\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:47:32.035936 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:32.035685    6224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0c358f8-4625-4498-8322-c843e5eeb017-istio-podinfo\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:47:32.214987 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:32.214955    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5"]
Oct 30 07:47:32.228512 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:32.228483    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb6be4-openshift-default-8557dddb7-snzc5"]
Oct 30 07:47:33.505313 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:33.505272    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0c358f8-4625-4498-8322-c843e5eeb017" path="/var/lib/kubelet/pods/d0c358f8-4625-4498-8322-c843e5eeb017/volumes"
Oct 30 07:47:42.695887 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:42.695855    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28"]
Oct 30 07:47:42.696634 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:42.696618    6224 cpu_manager.go:414] "containerMap: removing container" podUID="d0c358f8-4625-4498-8322-c843e5eeb017" containerName="istio-proxy"
Oct 30 07:47:42.696703 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:42.696635    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0c358f8-4625-4498-8322-c843e5eeb017" containerName="istio-proxy"
Oct 30 07:47:42.696753 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:42.696744    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0c358f8-4625-4498-8322-c843e5eeb017" containerName="istio-proxy"
Oct 30 07:47:42.702061 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:42.702042    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28"
Oct 30 07:47:42.733055 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:42.733028    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-8fef73-openshift-default-dockercfg-dsd9s\""
Oct 30 07:47:42.739872 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:42.739841    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28"]
Oct 30 07:47:42.777298 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:42.777271    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpt7g\" (UniqueName: \"kubernetes.io/projected/aa7e606a-884a-4553-943a-15fd296e0a24-kube-api-access-zpt7g\") pod \"e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28\" (UID: \"aa7e606a-884a-4553-943a-15fd296e0a24\") " pod="openshift-ingress/e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28"
Oct 30 07:47:42.777452 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:42.777338    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa7e606a-884a-4553-943a-15fd296e0a24-istiod-ca-cert\") pod \"e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28\" (UID: \"aa7e606a-884a-4553-943a-15fd296e0a24\") " pod="openshift-ingress/e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28"
Oct 30 07:47:42.777532 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:42.777467    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa7e606a-884a-4553-943a-15fd296e0a24-workload-socket\") pod \"e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28\" (UID: \"aa7e606a-884a-4553-943a-15fd296e0a24\") " pod="openshift-ingress/e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28"
Oct 30 07:47:42.777532 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:42.777515    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa7e606a-884a-4553-943a-15fd296e0a24-istio-podinfo\") pod \"e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28\" (UID: \"aa7e606a-884a-4553-943a-15fd296e0a24\") " pod="openshift-ingress/e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28"
Oct 30 07:47:42.777672 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:42.777656    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa7e606a-884a-4553-943a-15fd296e0a24-istio-data\") pod \"e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28\" (UID: \"aa7e606a-884a-4553-943a-15fd296e0a24\") " pod="openshift-ingress/e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28"
Oct 30 07:47:42.777766 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:42.777752    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa7e606a-884a-4553-943a-15fd296e0a24-workload-certs\") pod \"e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28\" (UID: \"aa7e606a-884a-4553-943a-15fd296e0a24\") " pod="openshift-ingress/e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28"
Oct 30 07:47:42.777881 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:42.777866    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa7e606a-884a-4553-943a-15fd296e0a24-istio-envoy\") pod \"e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28\" (UID: \"aa7e606a-884a-4553-943a-15fd296e0a24\") " pod="openshift-ingress/e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28"
Oct 30 07:47:42.777954 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:42.777905    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa7e606a-884a-4553-943a-15fd296e0a24-credential-socket\") pod \"e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28\" (UID: \"aa7e606a-884a-4553-943a-15fd296e0a24\") " pod="openshift-ingress/e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28"
Oct 30 07:47:42.777954 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:42.777947    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa7e606a-884a-4553-943a-15fd296e0a24-istio-token\") pod \"e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28\" (UID: \"aa7e606a-884a-4553-943a-15fd296e0a24\") " pod="openshift-ingress/e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28"
Oct 30 07:47:42.879947 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:42.879914    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa7e606a-884a-4553-943a-15fd296e0a24-istio-envoy\") pod \"e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28\" (UID: \"aa7e606a-884a-4553-943a-15fd296e0a24\") " pod="openshift-ingress/e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28"
Oct 30 07:47:42.879947 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:42.879951    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa7e606a-884a-4553-943a-15fd296e0a24-credential-socket\") pod \"e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28\" (UID: \"aa7e606a-884a-4553-943a-15fd296e0a24\") " pod="openshift-ingress/e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28"
Oct 30 07:47:42.880184 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:42.879980    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa7e606a-884a-4553-943a-15fd296e0a24-istio-token\") pod \"e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28\" (UID: \"aa7e606a-884a-4553-943a-15fd296e0a24\") " pod="openshift-ingress/e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28"
Oct 30 07:47:42.880184 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:42.880026    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpt7g\" (UniqueName: \"kubernetes.io/projected/aa7e606a-884a-4553-943a-15fd296e0a24-kube-api-access-zpt7g\") pod \"e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28\" (UID: \"aa7e606a-884a-4553-943a-15fd296e0a24\") " pod="openshift-ingress/e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28"
Oct 30 07:47:42.880184 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:42.880088    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa7e606a-884a-4553-943a-15fd296e0a24-istiod-ca-cert\") pod \"e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28\" (UID: \"aa7e606a-884a-4553-943a-15fd296e0a24\") " pod="openshift-ingress/e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28"
Oct 30 07:47:42.880184 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:42.880116    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa7e606a-884a-4553-943a-15fd296e0a24-workload-socket\") pod \"e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28\" (UID: \"aa7e606a-884a-4553-943a-15fd296e0a24\") " pod="openshift-ingress/e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28"
Oct 30 07:47:42.880184 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:42.880148    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa7e606a-884a-4553-943a-15fd296e0a24-istio-podinfo\") pod \"e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28\" (UID: \"aa7e606a-884a-4553-943a-15fd296e0a24\") " pod="openshift-ingress/e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28"
Oct 30 07:47:42.880525 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:42.880252    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa7e606a-884a-4553-943a-15fd296e0a24-istio-data\") pod \"e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28\" (UID: \"aa7e606a-884a-4553-943a-15fd296e0a24\") " pod="openshift-ingress/e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28"
Oct 30 07:47:42.880525 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:42.880339    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa7e606a-884a-4553-943a-15fd296e0a24-workload-certs\") pod \"e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28\" (UID: \"aa7e606a-884a-4553-943a-15fd296e0a24\") " pod="openshift-ingress/e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28"
Oct 30 07:47:42.880525 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:42.880402    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa7e606a-884a-4553-943a-15fd296e0a24-credential-socket\") pod \"e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28\" (UID: \"aa7e606a-884a-4553-943a-15fd296e0a24\") " pod="openshift-ingress/e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28"
Oct 30 07:47:42.880694 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:42.880587    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa7e606a-884a-4553-943a-15fd296e0a24-workload-socket\") pod \"e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28\" (UID: \"aa7e606a-884a-4553-943a-15fd296e0a24\") " pod="openshift-ingress/e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28"
Oct 30 07:47:42.880694 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:42.880607    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa7e606a-884a-4553-943a-15fd296e0a24-istio-data\") pod \"e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28\" (UID: \"aa7e606a-884a-4553-943a-15fd296e0a24\") " pod="openshift-ingress/e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28"
Oct 30 07:47:42.880797 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:42.880779    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa7e606a-884a-4553-943a-15fd296e0a24-workload-certs\") pod \"e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28\" (UID: \"aa7e606a-884a-4553-943a-15fd296e0a24\") " pod="openshift-ingress/e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28"
Oct 30 07:47:42.880907 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:42.880881    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa7e606a-884a-4553-943a-15fd296e0a24-istiod-ca-cert\") pod \"e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28\" (UID: \"aa7e606a-884a-4553-943a-15fd296e0a24\") " pod="openshift-ingress/e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28"
Oct 30 07:47:42.882213 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:42.882189    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa7e606a-884a-4553-943a-15fd296e0a24-istio-envoy\") pod \"e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28\" (UID: \"aa7e606a-884a-4553-943a-15fd296e0a24\") " pod="openshift-ingress/e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28"
Oct 30 07:47:42.882615 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:42.882599    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa7e606a-884a-4553-943a-15fd296e0a24-istio-podinfo\") pod \"e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28\" (UID: \"aa7e606a-884a-4553-943a-15fd296e0a24\") " pod="openshift-ingress/e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28"
Oct 30 07:47:42.905131 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:42.905103    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa7e606a-884a-4553-943a-15fd296e0a24-istio-token\") pod \"e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28\" (UID: \"aa7e606a-884a-4553-943a-15fd296e0a24\") " pod="openshift-ingress/e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28"
Oct 30 07:47:42.905294 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:42.905270    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpt7g\" (UniqueName: \"kubernetes.io/projected/aa7e606a-884a-4553-943a-15fd296e0a24-kube-api-access-zpt7g\") pod \"e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28\" (UID: \"aa7e606a-884a-4553-943a-15fd296e0a24\") " pod="openshift-ingress/e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28"
Oct 30 07:47:43.016940 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:43.016848    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28"
Oct 30 07:47:43.335234 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:47:43.335201    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa7e606a_884a_4553_943a_15fd296e0a24.slice/crio-29938bd754a68c906bd84e1caca5d2f2ada0cb65d1327ed7d0cd52071491856a WatchSource:0}: Error finding container 29938bd754a68c906bd84e1caca5d2f2ada0cb65d1327ed7d0cd52071491856a: Status 404 returned error can't find the container with id 29938bd754a68c906bd84e1caca5d2f2ada0cb65d1327ed7d0cd52071491856a
Oct 30 07:47:43.336509 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:43.336488    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28"]
Oct 30 07:47:43.338397 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:43.338354    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:47:43.338471 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:43.338436    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:47:43.338521 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:43.338484    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:47:43.994237 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:43.994134    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28" event={"ID":"aa7e606a-884a-4553-943a-15fd296e0a24","Type":"ContainerStarted","Data":"f7ba6292111a6248b2dc76056f36da9806cfffe69d84620fab4a714f8ba7c3e7"}
Oct 30 07:47:43.994237 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:43.994199    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28" event={"ID":"aa7e606a-884a-4553-943a-15fd296e0a24","Type":"ContainerStarted","Data":"29938bd754a68c906bd84e1caca5d2f2ada0cb65d1327ed7d0cd52071491856a"}
Oct 30 07:47:44.017835 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:44.017784    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28"
Oct 30 07:47:44.019740 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:44.019717    6224 patch_prober.go:29] interesting pod/e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Oct 30 07:47:44.019820 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:44.019763    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28" podUID="aa7e606a-884a-4553-943a-15fd296e0a24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Oct 30 07:47:44.059588 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:44.059540    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28" podStartSLOduration=2.059527575 podStartE2EDuration="2.059527575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:47:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:47:44.056237777 +0000 UTC m=+3022.665644749" watchObservedRunningTime="2026-10-30 07:47:44.059527575 +0000 UTC m=+3022.668934544"
Oct 30 07:47:45.018052 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:45.018018    6224 patch_prober.go:29] interesting pod/e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Oct 30 07:47:45.018455 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:45.018069    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28" podUID="aa7e606a-884a-4553-943a-15fd296e0a24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Oct 30 07:47:46.017247 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:46.017204    6224 patch_prober.go:29] interesting pod/e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Oct 30 07:47:46.017545 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:46.017268    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28" podUID="aa7e606a-884a-4553-943a-15fd296e0a24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Oct 30 07:47:47.017416 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:47.017384    6224 patch_prober.go:29] interesting pod/e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Oct 30 07:47:47.017878 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:47.017436    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28" podUID="aa7e606a-884a-4553-943a-15fd296e0a24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Oct 30 07:47:48.017854 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:48.017800    6224 patch_prober.go:29] interesting pod/e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Oct 30 07:47:48.018264 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:48.017870    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28" podUID="aa7e606a-884a-4553-943a-15fd296e0a24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Oct 30 07:47:49.018075 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:49.018040    6224 patch_prober.go:29] interesting pod/e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Oct 30 07:47:49.018510 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:49.018116    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28" podUID="aa7e606a-884a-4553-943a-15fd296e0a24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Oct 30 07:47:50.022502 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:50.022469    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28"
Oct 30 07:47:50.022981 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:50.022913    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28"
Oct 30 07:47:50.023681 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:50.023659    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28"
Oct 30 07:47:57.662016 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:47:57.661985    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:48:01.860195 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:48:01.860155    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:48:13.599221 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:13.599183    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688"]
Oct 30 07:48:13.605330 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:13.605301    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688"
Oct 30 07:48:13.634314 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:13.634285    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688"]
Oct 30 07:48:13.681912 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:13.681881    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-896ca4-openshift-default-dockercfg-5jnsv\""
Oct 30 07:48:13.743832 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:13.743756    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbptl\" (UniqueName: \"kubernetes.io/projected/ecd9f1bd-d4f6-468f-9994-203a3620a1a3-kube-api-access-mbptl\") pod \"e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688\" (UID: \"ecd9f1bd-d4f6-468f-9994-203a3620a1a3\") " pod="openshift-ingress/e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688"
Oct 30 07:48:13.744044 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:13.743870    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecd9f1bd-d4f6-468f-9994-203a3620a1a3-istiod-ca-cert\") pod \"e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688\" (UID: \"ecd9f1bd-d4f6-468f-9994-203a3620a1a3\") " pod="openshift-ingress/e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688"
Oct 30 07:48:13.744044 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:13.743913    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecd9f1bd-d4f6-468f-9994-203a3620a1a3-workload-socket\") pod \"e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688\" (UID: \"ecd9f1bd-d4f6-468f-9994-203a3620a1a3\") " pod="openshift-ingress/e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688"
Oct 30 07:48:13.744044 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:13.743973    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecd9f1bd-d4f6-468f-9994-203a3620a1a3-istio-token\") pod \"e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688\" (UID: \"ecd9f1bd-d4f6-468f-9994-203a3620a1a3\") " pod="openshift-ingress/e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688"
Oct 30 07:48:13.744044 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:13.744041    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecd9f1bd-d4f6-468f-9994-203a3620a1a3-credential-socket\") pod \"e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688\" (UID: \"ecd9f1bd-d4f6-468f-9994-203a3620a1a3\") " pod="openshift-ingress/e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688"
Oct 30 07:48:13.744292 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:13.744101    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecd9f1bd-d4f6-468f-9994-203a3620a1a3-workload-certs\") pod \"e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688\" (UID: \"ecd9f1bd-d4f6-468f-9994-203a3620a1a3\") " pod="openshift-ingress/e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688"
Oct 30 07:48:13.744292 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:13.744172    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecd9f1bd-d4f6-468f-9994-203a3620a1a3-istio-envoy\") pod \"e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688\" (UID: \"ecd9f1bd-d4f6-468f-9994-203a3620a1a3\") " pod="openshift-ingress/e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688"
Oct 30 07:48:13.753824 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:13.745027    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecd9f1bd-d4f6-468f-9994-203a3620a1a3-istio-podinfo\") pod \"e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688\" (UID: \"ecd9f1bd-d4f6-468f-9994-203a3620a1a3\") " pod="openshift-ingress/e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688"
Oct 30 07:48:13.753824 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:13.745103    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecd9f1bd-d4f6-468f-9994-203a3620a1a3-istio-data\") pod \"e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688\" (UID: \"ecd9f1bd-d4f6-468f-9994-203a3620a1a3\") " pod="openshift-ingress/e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688"
Oct 30 07:48:13.852880 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:13.850621    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecd9f1bd-d4f6-468f-9994-203a3620a1a3-istio-data\") pod \"e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688\" (UID: \"ecd9f1bd-d4f6-468f-9994-203a3620a1a3\") " pod="openshift-ingress/e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688"
Oct 30 07:48:13.852880 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:13.850704    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbptl\" (UniqueName: \"kubernetes.io/projected/ecd9f1bd-d4f6-468f-9994-203a3620a1a3-kube-api-access-mbptl\") pod \"e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688\" (UID: \"ecd9f1bd-d4f6-468f-9994-203a3620a1a3\") " pod="openshift-ingress/e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688"
Oct 30 07:48:13.852880 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:13.850778    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecd9f1bd-d4f6-468f-9994-203a3620a1a3-istiod-ca-cert\") pod \"e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688\" (UID: \"ecd9f1bd-d4f6-468f-9994-203a3620a1a3\") " pod="openshift-ingress/e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688"
Oct 30 07:48:13.852880 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:13.850839    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecd9f1bd-d4f6-468f-9994-203a3620a1a3-workload-socket\") pod \"e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688\" (UID: \"ecd9f1bd-d4f6-468f-9994-203a3620a1a3\") " pod="openshift-ingress/e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688"
Oct 30 07:48:13.852880 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:13.850899    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecd9f1bd-d4f6-468f-9994-203a3620a1a3-istio-token\") pod \"e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688\" (UID: \"ecd9f1bd-d4f6-468f-9994-203a3620a1a3\") " pod="openshift-ingress/e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688"
Oct 30 07:48:13.852880 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:13.850973    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecd9f1bd-d4f6-468f-9994-203a3620a1a3-credential-socket\") pod \"e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688\" (UID: \"ecd9f1bd-d4f6-468f-9994-203a3620a1a3\") " pod="openshift-ingress/e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688"
Oct 30 07:48:13.852880 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:13.851033    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecd9f1bd-d4f6-468f-9994-203a3620a1a3-workload-certs\") pod \"e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688\" (UID: \"ecd9f1bd-d4f6-468f-9994-203a3620a1a3\") " pod="openshift-ingress/e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688"
Oct 30 07:48:13.852880 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:13.851102    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecd9f1bd-d4f6-468f-9994-203a3620a1a3-istio-envoy\") pod \"e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688\" (UID: \"ecd9f1bd-d4f6-468f-9994-203a3620a1a3\") " pod="openshift-ingress/e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688"
Oct 30 07:48:13.852880 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:13.851220    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecd9f1bd-d4f6-468f-9994-203a3620a1a3-istio-podinfo\") pod \"e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688\" (UID: \"ecd9f1bd-d4f6-468f-9994-203a3620a1a3\") " pod="openshift-ingress/e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688"
Oct 30 07:48:13.852880 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:13.852197    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecd9f1bd-d4f6-468f-9994-203a3620a1a3-workload-socket\") pod \"e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688\" (UID: \"ecd9f1bd-d4f6-468f-9994-203a3620a1a3\") " pod="openshift-ingress/e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688"
Oct 30 07:48:13.852880 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:13.852456    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecd9f1bd-d4f6-468f-9994-203a3620a1a3-istio-data\") pod \"e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688\" (UID: \"ecd9f1bd-d4f6-468f-9994-203a3620a1a3\") " pod="openshift-ingress/e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688"
Oct 30 07:48:13.853620 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:13.853438    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecd9f1bd-d4f6-468f-9994-203a3620a1a3-istiod-ca-cert\") pod \"e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688\" (UID: \"ecd9f1bd-d4f6-468f-9994-203a3620a1a3\") " pod="openshift-ingress/e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688"
Oct 30 07:48:13.853710 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:13.853683    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecd9f1bd-d4f6-468f-9994-203a3620a1a3-credential-socket\") pod \"e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688\" (UID: \"ecd9f1bd-d4f6-468f-9994-203a3620a1a3\") " pod="openshift-ingress/e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688"
Oct 30 07:48:13.857552 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:13.857528    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecd9f1bd-d4f6-468f-9994-203a3620a1a3-workload-certs\") pod \"e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688\" (UID: \"ecd9f1bd-d4f6-468f-9994-203a3620a1a3\") " pod="openshift-ingress/e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688"
Oct 30 07:48:13.863829 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:13.861162    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecd9f1bd-d4f6-468f-9994-203a3620a1a3-istio-podinfo\") pod \"e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688\" (UID: \"ecd9f1bd-d4f6-468f-9994-203a3620a1a3\") " pod="openshift-ingress/e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688"
Oct 30 07:48:13.882143 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:13.882097    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecd9f1bd-d4f6-468f-9994-203a3620a1a3-istio-envoy\") pod \"e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688\" (UID: \"ecd9f1bd-d4f6-468f-9994-203a3620a1a3\") " pod="openshift-ingress/e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688"
Oct 30 07:48:13.907461 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:13.907418    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbptl\" (UniqueName: \"kubernetes.io/projected/ecd9f1bd-d4f6-468f-9994-203a3620a1a3-kube-api-access-mbptl\") pod \"e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688\" (UID: \"ecd9f1bd-d4f6-468f-9994-203a3620a1a3\") " pod="openshift-ingress/e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688"
Oct 30 07:48:13.928206 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:13.928118    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecd9f1bd-d4f6-468f-9994-203a3620a1a3-istio-token\") pod \"e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688\" (UID: \"ecd9f1bd-d4f6-468f-9994-203a3620a1a3\") " pod="openshift-ingress/e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688"
Oct 30 07:48:13.947377 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:13.947147    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688"
Oct 30 07:48:14.276413 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:48:14.276378    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecd9f1bd_d4f6_468f_9994_203a3620a1a3.slice/crio-637803a9aea52b4852e32e3609d9a805362e92c58b52cce498bf62a2c2e52cd1 WatchSource:0}: Error finding container 637803a9aea52b4852e32e3609d9a805362e92c58b52cce498bf62a2c2e52cd1: Status 404 returned error can't find the container with id 637803a9aea52b4852e32e3609d9a805362e92c58b52cce498bf62a2c2e52cd1
Oct 30 07:48:14.279559 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:14.279528    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:48:14.279645 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:14.279601    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:48:14.279645 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:14.279629    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:48:14.279968 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:14.279944    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688"]
Oct 30 07:48:15.247319 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:15.247282    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688" event={"ID":"ecd9f1bd-d4f6-468f-9994-203a3620a1a3","Type":"ContainerStarted","Data":"55da16416b464b23e612a907bafbbed04a2c7eb62d44560a1fc60d393233730e"}
Oct 30 07:48:15.247735 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:15.247374    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688" event={"ID":"ecd9f1bd-d4f6-468f-9994-203a3620a1a3","Type":"ContainerStarted","Data":"637803a9aea52b4852e32e3609d9a805362e92c58b52cce498bf62a2c2e52cd1"}
Oct 30 07:48:15.327865 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:15.327765    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688" podStartSLOduration=2.327752049 podStartE2EDuration="2.327752049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:48:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:48:15.327407108 +0000 UTC m=+3053.936814078" watchObservedRunningTime="2026-10-30 07:48:15.327752049 +0000 UTC m=+3053.937159043"
Oct 30 07:48:15.948562 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:15.948529    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688"
Oct 30 07:48:15.955469 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:15.955441    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688"
Oct 30 07:48:16.255619 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:16.255544    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688"
Oct 30 07:48:16.256637 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:16.256617    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688"
Oct 30 07:48:19.814331 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:19.814297    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-896ca4-54fd8b66c5-52wzf"]
Oct 30 07:48:19.820257 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:19.820235    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-896ca4-54fd8b66c5-52wzf"
Oct 30 07:48:19.837006 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:19.836976    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-896ca4-54fd8b66c5-52wzf"]
Oct 30 07:48:19.837180 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:19.836994    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-896ca4\""
Oct 30 07:48:19.841940 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:19.841906    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f7c9bd0-cb40-4bae-b3b7-9301bc73534d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-896ca4-54fd8b66c5-52wzf\" (UID: \"3f7c9bd0-cb40-4bae-b3b7-9301bc73534d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-896ca4-54fd8b66c5-52wzf"
Oct 30 07:48:19.842115 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:19.842088    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsfjb\" (UniqueName: \"kubernetes.io/projected/3f7c9bd0-cb40-4bae-b3b7-9301bc73534d-kube-api-access-dsfjb\") pod \"payload-pre-processing-e2e-inf-b-896ca4-54fd8b66c5-52wzf\" (UID: \"3f7c9bd0-cb40-4bae-b3b7-9301bc73534d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-896ca4-54fd8b66c5-52wzf"
Oct 30 07:48:19.846023 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:19.845995    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-896ca4-dockercfg-c998g\""
Oct 30 07:48:19.908826 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:19.896190    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-896ca4-7c64c58848-895qc"]
Oct 30 07:48:19.908826 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:19.902380    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-896ca4-7c64c58848-895qc"
Oct 30 07:48:19.932831 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:19.922647    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-896ca4-7c64c58848-895qc"]
Oct 30 07:48:19.991727 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:19.991272    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f7c9bd0-cb40-4bae-b3b7-9301bc73534d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-896ca4-54fd8b66c5-52wzf\" (UID: \"3f7c9bd0-cb40-4bae-b3b7-9301bc73534d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-896ca4-54fd8b66c5-52wzf"
Oct 30 07:48:19.991727 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:19.991362    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5clhk\" (UniqueName: \"kubernetes.io/projected/78d46aa4-8d9e-4094-9e97-5ef71b817928-kube-api-access-5clhk\") pod \"payload-processing-e2e-inf-b-896ca4-7c64c58848-895qc\" (UID: \"78d46aa4-8d9e-4094-9e97-5ef71b817928\") " pod="openshift-ingress/payload-processing-e2e-inf-b-896ca4-7c64c58848-895qc"
Oct 30 07:48:19.991727 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:19.991430    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsfjb\" (UniqueName: \"kubernetes.io/projected/3f7c9bd0-cb40-4bae-b3b7-9301bc73534d-kube-api-access-dsfjb\") pod \"payload-pre-processing-e2e-inf-b-896ca4-54fd8b66c5-52wzf\" (UID: \"3f7c9bd0-cb40-4bae-b3b7-9301bc73534d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-896ca4-54fd8b66c5-52wzf"
Oct 30 07:48:19.991727 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:19.991507    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78d46aa4-8d9e-4094-9e97-5ef71b817928-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-896ca4-7c64c58848-895qc\" (UID: \"78d46aa4-8d9e-4094-9e97-5ef71b817928\") " pod="openshift-ingress/payload-processing-e2e-inf-b-896ca4-7c64c58848-895qc"
Oct 30 07:48:19.993586 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:19.993324    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f7c9bd0-cb40-4bae-b3b7-9301bc73534d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-896ca4-54fd8b66c5-52wzf\" (UID: \"3f7c9bd0-cb40-4bae-b3b7-9301bc73534d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-896ca4-54fd8b66c5-52wzf"
Oct 30 07:48:20.036827 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:20.036273    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsfjb\" (UniqueName: \"kubernetes.io/projected/3f7c9bd0-cb40-4bae-b3b7-9301bc73534d-kube-api-access-dsfjb\") pod \"payload-pre-processing-e2e-inf-b-896ca4-54fd8b66c5-52wzf\" (UID: \"3f7c9bd0-cb40-4bae-b3b7-9301bc73534d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-896ca4-54fd8b66c5-52wzf"
Oct 30 07:48:20.093401 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:20.093310    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78d46aa4-8d9e-4094-9e97-5ef71b817928-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-896ca4-7c64c58848-895qc\" (UID: \"78d46aa4-8d9e-4094-9e97-5ef71b817928\") " pod="openshift-ingress/payload-processing-e2e-inf-b-896ca4-7c64c58848-895qc"
Oct 30 07:48:20.093652 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:20.093628    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5clhk\" (UniqueName: \"kubernetes.io/projected/78d46aa4-8d9e-4094-9e97-5ef71b817928-kube-api-access-5clhk\") pod \"payload-processing-e2e-inf-b-896ca4-7c64c58848-895qc\" (UID: \"78d46aa4-8d9e-4094-9e97-5ef71b817928\") " pod="openshift-ingress/payload-processing-e2e-inf-b-896ca4-7c64c58848-895qc"
Oct 30 07:48:20.094167 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:20.094149    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78d46aa4-8d9e-4094-9e97-5ef71b817928-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-896ca4-7c64c58848-895qc\" (UID: \"78d46aa4-8d9e-4094-9e97-5ef71b817928\") " pod="openshift-ingress/payload-processing-e2e-inf-b-896ca4-7c64c58848-895qc"
Oct 30 07:48:20.119529 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:20.119496    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5clhk\" (UniqueName: \"kubernetes.io/projected/78d46aa4-8d9e-4094-9e97-5ef71b817928-kube-api-access-5clhk\") pod \"payload-processing-e2e-inf-b-896ca4-7c64c58848-895qc\" (UID: \"78d46aa4-8d9e-4094-9e97-5ef71b817928\") " pod="openshift-ingress/payload-processing-e2e-inf-b-896ca4-7c64c58848-895qc"
Oct 30 07:48:20.148819 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:20.148784    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-896ca4-54fd8b66c5-52wzf"
Oct 30 07:48:20.293878 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:20.293849    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-896ca4-7c64c58848-895qc"
Oct 30 07:48:20.460451 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:48:20.460422    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f7c9bd0_cb40_4bae_b3b7_9301bc73534d.slice/crio-478d0ea6362029dee7bac09a8cfcb135b84e98982ca70a932bf5226cc2589590 WatchSource:0}: Error finding container 478d0ea6362029dee7bac09a8cfcb135b84e98982ca70a932bf5226cc2589590: Status 404 returned error can't find the container with id 478d0ea6362029dee7bac09a8cfcb135b84e98982ca70a932bf5226cc2589590
Oct 30 07:48:20.461639 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:20.461610    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-896ca4-54fd8b66c5-52wzf"]
Oct 30 07:48:20.600065 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:48:20.600034    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78d46aa4_8d9e_4094_9e97_5ef71b817928.slice/crio-ae3a359df5380fce63a8f944f95e4185c535b2b2d996edfc85ca5a2f6f8948e4 WatchSource:0}: Error finding container ae3a359df5380fce63a8f944f95e4185c535b2b2d996edfc85ca5a2f6f8948e4: Status 404 returned error can't find the container with id ae3a359df5380fce63a8f944f95e4185c535b2b2d996edfc85ca5a2f6f8948e4
Oct 30 07:48:20.602066 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:20.602039    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-896ca4-7c64c58848-895qc"]
Oct 30 07:48:21.302832 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:21.302770    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-896ca4-54fd8b66c5-52wzf" event={"ID":"3f7c9bd0-cb40-4bae-b3b7-9301bc73534d","Type":"ContainerStarted","Data":"7ff7beb3018f0e2dbe12a6c2b6b323a61d2a14b3783a0a7fa0c620bb3b5cea8f"}
Oct 30 07:48:21.303290 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:21.302849    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-896ca4-54fd8b66c5-52wzf"
Oct 30 07:48:21.303290 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:21.302876    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-896ca4-54fd8b66c5-52wzf" event={"ID":"3f7c9bd0-cb40-4bae-b3b7-9301bc73534d","Type":"ContainerStarted","Data":"478d0ea6362029dee7bac09a8cfcb135b84e98982ca70a932bf5226cc2589590"}
Oct 30 07:48:21.304572 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:21.304542    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-896ca4-7c64c58848-895qc" event={"ID":"78d46aa4-8d9e-4094-9e97-5ef71b817928","Type":"ContainerStarted","Data":"9e8c5b845d5a00b692cd9c84a62668c3f0333422dee9f1603a3c6e98abd248f0"}
Oct 30 07:48:21.304701 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:21.304592    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-896ca4-7c64c58848-895qc" event={"ID":"78d46aa4-8d9e-4094-9e97-5ef71b817928","Type":"ContainerStarted","Data":"ae3a359df5380fce63a8f944f95e4185c535b2b2d996edfc85ca5a2f6f8948e4"}
Oct 30 07:48:21.304701 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:21.304622    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-896ca4-7c64c58848-895qc"
Oct 30 07:48:21.348929 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:21.348873    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-896ca4-54fd8b66c5-52wzf" podStartSLOduration=2.3488556210000002 podStartE2EDuration="2.348855621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:48:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:48:21.346644543 +0000 UTC m=+3059.956051514" watchObservedRunningTime="2026-10-30 07:48:21.348855621 +0000 UTC m=+3059.958262592"
Oct 30 07:48:21.393010 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:21.392965    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-896ca4-7c64c58848-895qc" podStartSLOduration=2.392952547 podStartE2EDuration="2.392952547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:48:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:48:21.390414619 +0000 UTC m=+3059.999821592" watchObservedRunningTime="2026-10-30 07:48:21.392952547 +0000 UTC m=+3060.002359526"
Oct 30 07:48:31.858826 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:48:31.858768    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:48:32.316167 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:32.316140    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-896ca4-7c64c58848-895qc"
Oct 30 07:48:32.316353 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:32.316217    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-896ca4-54fd8b66c5-52wzf"
Oct 30 07:48:47.732978 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:48:47.732940    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:49:01.859175 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:49:01.859136    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:49:04.785782 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:04.785743    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-91c8c2/test-model-91c8c2-kserve-78669df767-s5qb6"]
Oct 30 07:49:04.792096 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:04.792070    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-91c8c2/test-model-91c8c2-kserve-78669df767-s5qb6"
Oct 30 07:49:04.809444 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:04.809421    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-91c8c2\"/\"kube-root-ca.crt\""
Oct 30 07:49:04.810223 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:04.810196    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-91c8c2\"/\"test-model-91c8c2-kserve-self-signed-certs\""
Oct 30 07:49:04.810223 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:04.810199    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-91c8c2\"/\"default-dockercfg-xmsch\""
Oct 30 07:49:04.814925 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:04.814900    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-91c8c2/test-model-91c8c2-kserve-78669df767-s5qb6"]
Oct 30 07:49:04.822068 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:04.822044    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-91c8c2\"/\"openshift-service-ca.crt\""
Oct 30 07:49:04.963604 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:04.963561    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0632075e-3c5c-4ba9-b79d-541825da2a3e-home\") pod \"test-model-91c8c2-kserve-78669df767-s5qb6\" (UID: \"0632075e-3c5c-4ba9-b79d-541825da2a3e\") " pod="ai-tenant-e2e-inf-a-91c8c2/test-model-91c8c2-kserve-78669df767-s5qb6"
Oct 30 07:49:04.963778 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:04.963670    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0632075e-3c5c-4ba9-b79d-541825da2a3e-model-cache\") pod \"test-model-91c8c2-kserve-78669df767-s5qb6\" (UID: \"0632075e-3c5c-4ba9-b79d-541825da2a3e\") " pod="ai-tenant-e2e-inf-a-91c8c2/test-model-91c8c2-kserve-78669df767-s5qb6"
Oct 30 07:49:04.963778 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:04.963719    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0632075e-3c5c-4ba9-b79d-541825da2a3e-tmp-dir\") pod \"test-model-91c8c2-kserve-78669df767-s5qb6\" (UID: \"0632075e-3c5c-4ba9-b79d-541825da2a3e\") " pod="ai-tenant-e2e-inf-a-91c8c2/test-model-91c8c2-kserve-78669df767-s5qb6"
Oct 30 07:49:04.964058 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:04.964033    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcspp\" (UniqueName: \"kubernetes.io/projected/0632075e-3c5c-4ba9-b79d-541825da2a3e-kube-api-access-bcspp\") pod \"test-model-91c8c2-kserve-78669df767-s5qb6\" (UID: \"0632075e-3c5c-4ba9-b79d-541825da2a3e\") " pod="ai-tenant-e2e-inf-a-91c8c2/test-model-91c8c2-kserve-78669df767-s5qb6"
Oct 30 07:49:04.964107 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:04.964074    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0632075e-3c5c-4ba9-b79d-541825da2a3e-tls-certs\") pod \"test-model-91c8c2-kserve-78669df767-s5qb6\" (UID: \"0632075e-3c5c-4ba9-b79d-541825da2a3e\") " pod="ai-tenant-e2e-inf-a-91c8c2/test-model-91c8c2-kserve-78669df767-s5qb6"
Oct 30 07:49:04.964166 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:04.964149    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0632075e-3c5c-4ba9-b79d-541825da2a3e-dshm\") pod \"test-model-91c8c2-kserve-78669df767-s5qb6\" (UID: \"0632075e-3c5c-4ba9-b79d-541825da2a3e\") " pod="ai-tenant-e2e-inf-a-91c8c2/test-model-91c8c2-kserve-78669df767-s5qb6"
Oct 30 07:49:05.068487 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:05.068390    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0632075e-3c5c-4ba9-b79d-541825da2a3e-home\") pod \"test-model-91c8c2-kserve-78669df767-s5qb6\" (UID: \"0632075e-3c5c-4ba9-b79d-541825da2a3e\") " pod="ai-tenant-e2e-inf-a-91c8c2/test-model-91c8c2-kserve-78669df767-s5qb6"
Oct 30 07:49:05.068850 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:05.068821    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0632075e-3c5c-4ba9-b79d-541825da2a3e-model-cache\") pod \"test-model-91c8c2-kserve-78669df767-s5qb6\" (UID: \"0632075e-3c5c-4ba9-b79d-541825da2a3e\") " pod="ai-tenant-e2e-inf-a-91c8c2/test-model-91c8c2-kserve-78669df767-s5qb6"
Oct 30 07:49:05.069030 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:05.069014    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0632075e-3c5c-4ba9-b79d-541825da2a3e-tmp-dir\") pod \"test-model-91c8c2-kserve-78669df767-s5qb6\" (UID: \"0632075e-3c5c-4ba9-b79d-541825da2a3e\") " pod="ai-tenant-e2e-inf-a-91c8c2/test-model-91c8c2-kserve-78669df767-s5qb6"
Oct 30 07:49:05.069542 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:05.069131    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0632075e-3c5c-4ba9-b79d-541825da2a3e-model-cache\") pod \"test-model-91c8c2-kserve-78669df767-s5qb6\" (UID: \"0632075e-3c5c-4ba9-b79d-541825da2a3e\") " pod="ai-tenant-e2e-inf-a-91c8c2/test-model-91c8c2-kserve-78669df767-s5qb6"
Oct 30 07:49:05.069542 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:05.069367    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcspp\" (UniqueName: \"kubernetes.io/projected/0632075e-3c5c-4ba9-b79d-541825da2a3e-kube-api-access-bcspp\") pod \"test-model-91c8c2-kserve-78669df767-s5qb6\" (UID: \"0632075e-3c5c-4ba9-b79d-541825da2a3e\") " pod="ai-tenant-e2e-inf-a-91c8c2/test-model-91c8c2-kserve-78669df767-s5qb6"
Oct 30 07:49:05.069542 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:05.069405    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0632075e-3c5c-4ba9-b79d-541825da2a3e-tls-certs\") pod \"test-model-91c8c2-kserve-78669df767-s5qb6\" (UID: \"0632075e-3c5c-4ba9-b79d-541825da2a3e\") " pod="ai-tenant-e2e-inf-a-91c8c2/test-model-91c8c2-kserve-78669df767-s5qb6"
Oct 30 07:49:05.069542 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:05.069441    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0632075e-3c5c-4ba9-b79d-541825da2a3e-dshm\") pod \"test-model-91c8c2-kserve-78669df767-s5qb6\" (UID: \"0632075e-3c5c-4ba9-b79d-541825da2a3e\") " pod="ai-tenant-e2e-inf-a-91c8c2/test-model-91c8c2-kserve-78669df767-s5qb6"
Oct 30 07:49:05.069542 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:05.069468    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0632075e-3c5c-4ba9-b79d-541825da2a3e-tmp-dir\") pod \"test-model-91c8c2-kserve-78669df767-s5qb6\" (UID: \"0632075e-3c5c-4ba9-b79d-541825da2a3e\") " pod="ai-tenant-e2e-inf-a-91c8c2/test-model-91c8c2-kserve-78669df767-s5qb6"
Oct 30 07:49:05.070704 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:05.070679    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0632075e-3c5c-4ba9-b79d-541825da2a3e-home\") pod \"test-model-91c8c2-kserve-78669df767-s5qb6\" (UID: \"0632075e-3c5c-4ba9-b79d-541825da2a3e\") " pod="ai-tenant-e2e-inf-a-91c8c2/test-model-91c8c2-kserve-78669df767-s5qb6"
Oct 30 07:49:05.074462 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:05.074400    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0632075e-3c5c-4ba9-b79d-541825da2a3e-dshm\") pod \"test-model-91c8c2-kserve-78669df767-s5qb6\" (UID: \"0632075e-3c5c-4ba9-b79d-541825da2a3e\") " pod="ai-tenant-e2e-inf-a-91c8c2/test-model-91c8c2-kserve-78669df767-s5qb6"
Oct 30 07:49:05.077672 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:05.077401    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0632075e-3c5c-4ba9-b79d-541825da2a3e-tls-certs\") pod \"test-model-91c8c2-kserve-78669df767-s5qb6\" (UID: \"0632075e-3c5c-4ba9-b79d-541825da2a3e\") " pod="ai-tenant-e2e-inf-a-91c8c2/test-model-91c8c2-kserve-78669df767-s5qb6"
Oct 30 07:49:05.098730 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:05.098698    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcspp\" (UniqueName: \"kubernetes.io/projected/0632075e-3c5c-4ba9-b79d-541825da2a3e-kube-api-access-bcspp\") pod \"test-model-91c8c2-kserve-78669df767-s5qb6\" (UID: \"0632075e-3c5c-4ba9-b79d-541825da2a3e\") " pod="ai-tenant-e2e-inf-a-91c8c2/test-model-91c8c2-kserve-78669df767-s5qb6"
Oct 30 07:49:05.110516 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:05.110447    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-91c8c2/test-model-91c8c2-kserve-78669df767-s5qb6"
Oct 30 07:49:05.644320 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:05.643344    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-91c8c2/test-model-91c8c2-kserve-78669df767-s5qb6"]
Oct 30 07:49:05.699310 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:05.699267    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-91c8c2/test-model-91c8c2-kserve-78669df767-s5qb6" event={"ID":"0632075e-3c5c-4ba9-b79d-541825da2a3e","Type":"ContainerStarted","Data":"4b9ec3612f10275d1cda938e16648a28f6099bfb8f7c2186b186140415898427"}
Oct 30 07:49:06.710328 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:06.710288    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-91c8c2/test-model-91c8c2-kserve-78669df767-s5qb6" event={"ID":"0632075e-3c5c-4ba9-b79d-541825da2a3e","Type":"ContainerStarted","Data":"0faea9cb300cf84ef0c36fab6090ea8563b00bfebaf74ba3a50bee9c4e8fc05a"}
Oct 30 07:49:06.755707 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:06.755660    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-91c8c2/test-model-91c8c2-kserve-78669df767-s5qb6" podStartSLOduration=2.755647148 podStartE2EDuration="2.755647148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:49:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:49:06.755148781 +0000 UTC m=+3105.364555775" watchObservedRunningTime="2026-10-30 07:49:06.755647148 +0000 UTC m=+3105.365054117"
Oct 30 07:49:07.111777 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:07.111741    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-91c8c2/test-model-91c8c2-kserve-78669df767-s5qb6"
Oct 30 07:49:11.053275 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:11.053242    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8514a3-774fb4b48c-b547d"]
Oct 30 07:49:11.058385 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:11.058364    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8514a3-774fb4b48c-b547d"
Oct 30 07:49:11.076620 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:11.076584    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8514a3-774fb4b48c-b547d"]
Oct 30 07:49:11.085175 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:11.085146    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-8514a3-dockercfg-hbvq7\""
Oct 30 07:49:11.085175 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:11.085152    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-8514a3\""
Oct 30 07:49:11.124942 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:11.124905    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d6723d6-a9d5-4a9f-a6d3-edc4dca51a7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8514a3-774fb4b48c-b547d\" (UID: \"9d6723d6-a9d5-4a9f-a6d3-edc4dca51a7f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8514a3-774fb4b48c-b547d"
Oct 30 07:49:11.125202 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:11.125180    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bpc2\" (UniqueName: \"kubernetes.io/projected/9d6723d6-a9d5-4a9f-a6d3-edc4dca51a7f-kube-api-access-4bpc2\") pod \"payload-pre-processing-e2e-inf-b-8514a3-774fb4b48c-b547d\" (UID: \"9d6723d6-a9d5-4a9f-a6d3-edc4dca51a7f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8514a3-774fb4b48c-b547d"
Oct 30 07:49:11.227902 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:11.227846    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d6723d6-a9d5-4a9f-a6d3-edc4dca51a7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8514a3-774fb4b48c-b547d\" (UID: \"9d6723d6-a9d5-4a9f-a6d3-edc4dca51a7f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8514a3-774fb4b48c-b547d"
Oct 30 07:49:11.228115 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:11.228069    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bpc2\" (UniqueName: \"kubernetes.io/projected/9d6723d6-a9d5-4a9f-a6d3-edc4dca51a7f-kube-api-access-4bpc2\") pod \"payload-pre-processing-e2e-inf-b-8514a3-774fb4b48c-b547d\" (UID: \"9d6723d6-a9d5-4a9f-a6d3-edc4dca51a7f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8514a3-774fb4b48c-b547d"
Oct 30 07:49:11.229560 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:11.229485    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d6723d6-a9d5-4a9f-a6d3-edc4dca51a7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8514a3-774fb4b48c-b547d\" (UID: \"9d6723d6-a9d5-4a9f-a6d3-edc4dca51a7f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8514a3-774fb4b48c-b547d"
Oct 30 07:49:11.292075 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:11.284986    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bpc2\" (UniqueName: \"kubernetes.io/projected/9d6723d6-a9d5-4a9f-a6d3-edc4dca51a7f-kube-api-access-4bpc2\") pod \"payload-pre-processing-e2e-inf-b-8514a3-774fb4b48c-b547d\" (UID: \"9d6723d6-a9d5-4a9f-a6d3-edc4dca51a7f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8514a3-774fb4b48c-b547d"
Oct 30 07:49:11.373919 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:11.373836    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8514a3-774fb4b48c-b547d"
Oct 30 07:49:11.681154 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:49:11.681124    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d6723d6_a9d5_4a9f_a6d3_edc4dca51a7f.slice/crio-24abd65a4245b86ec8615fdea218e200ad7c6002578405884bbf477096ce515e WatchSource:0}: Error finding container 24abd65a4245b86ec8615fdea218e200ad7c6002578405884bbf477096ce515e: Status 404 returned error can't find the container with id 24abd65a4245b86ec8615fdea218e200ad7c6002578405884bbf477096ce515e
Oct 30 07:49:11.683667 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:11.683637    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8514a3-774fb4b48c-b547d"]
Oct 30 07:49:11.750406 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:11.750363    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8514a3-774fb4b48c-b547d" event={"ID":"9d6723d6-a9d5-4a9f-a6d3-edc4dca51a7f","Type":"ContainerStarted","Data":"24abd65a4245b86ec8615fdea218e200ad7c6002578405884bbf477096ce515e"}
Oct 30 07:49:12.759467 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:12.759424    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8514a3-774fb4b48c-b547d" event={"ID":"9d6723d6-a9d5-4a9f-a6d3-edc4dca51a7f","Type":"ContainerStarted","Data":"3d52c6fd801512578394c8f4e4841cfebfdcdbb4d02cc9faf5a25eac75a5f794"}
Oct 30 07:49:12.759930 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:12.759500    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8514a3-774fb4b48c-b547d"
Oct 30 07:49:12.804686 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:12.804632    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8514a3-774fb4b48c-b547d" podStartSLOduration=1.80462011 podStartE2EDuration="1.80462011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:49:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:49:12.802773924 +0000 UTC m=+3111.412180898" watchObservedRunningTime="2026-10-30 07:49:12.80462011 +0000 UTC m=+3111.414027378"
Oct 30 07:49:15.111567 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:15.111524    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-91c8c2/test-model-91c8c2-kserve-78669df767-s5qb6"
Oct 30 07:49:15.124426 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:15.124393    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-91c8c2/test-model-91c8c2-kserve-78669df767-s5qb6"
Oct 30 07:49:15.795039 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:15.795007    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-91c8c2/test-model-91c8c2-kserve-78669df767-s5qb6"
Oct 30 07:49:23.771287 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:23.771254    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8514a3-774fb4b48c-b547d"
Oct 30 07:49:30.783984 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:30.783945    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-8c0e05/test-model-8c0e05-kserve-6797bf4db9-jdlmp"]
Oct 30 07:49:30.790275 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:30.790238    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8c0e05/test-model-8c0e05-kserve-6797bf4db9-jdlmp"
Oct 30 07:49:30.813599 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:30.813534    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-8c0e05/test-model-8c0e05-kserve-6797bf4db9-jdlmp"]
Oct 30 07:49:30.820927 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:30.820900    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-8c0e05\"/\"openshift-service-ca.crt\""
Oct 30 07:49:30.821102 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:30.820902    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-8c0e05\"/\"default-dockercfg-flnfv\""
Oct 30 07:49:30.821203 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:30.820930    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-8c0e05\"/\"test-model-8c0e05-kserve-self-signed-certs\""
Oct 30 07:49:30.821203 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:30.820948    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-8c0e05\"/\"kube-root-ca.crt\""
Oct 30 07:49:30.853241 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:30.853210    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75c9ba4f-dca7-4227-902f-97b2b8a262c6-dshm\") pod \"test-model-8c0e05-kserve-6797bf4db9-jdlmp\" (UID: \"75c9ba4f-dca7-4227-902f-97b2b8a262c6\") " pod="ai-tenant-e2e-inf-b-8c0e05/test-model-8c0e05-kserve-6797bf4db9-jdlmp"
Oct 30 07:49:30.853483 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:30.853460    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75c9ba4f-dca7-4227-902f-97b2b8a262c6-home\") pod \"test-model-8c0e05-kserve-6797bf4db9-jdlmp\" (UID: \"75c9ba4f-dca7-4227-902f-97b2b8a262c6\") " pod="ai-tenant-e2e-inf-b-8c0e05/test-model-8c0e05-kserve-6797bf4db9-jdlmp"
Oct 30 07:49:30.853593 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:30.853578    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75c9ba4f-dca7-4227-902f-97b2b8a262c6-model-cache\") pod \"test-model-8c0e05-kserve-6797bf4db9-jdlmp\" (UID: \"75c9ba4f-dca7-4227-902f-97b2b8a262c6\") " pod="ai-tenant-e2e-inf-b-8c0e05/test-model-8c0e05-kserve-6797bf4db9-jdlmp"
Oct 30 07:49:30.853653 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:30.853640    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75c9ba4f-dca7-4227-902f-97b2b8a262c6-tls-certs\") pod \"test-model-8c0e05-kserve-6797bf4db9-jdlmp\" (UID: \"75c9ba4f-dca7-4227-902f-97b2b8a262c6\") " pod="ai-tenant-e2e-inf-b-8c0e05/test-model-8c0e05-kserve-6797bf4db9-jdlmp"
Oct 30 07:49:30.853696 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:30.853674    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb7hx\" (UniqueName: \"kubernetes.io/projected/75c9ba4f-dca7-4227-902f-97b2b8a262c6-kube-api-access-vb7hx\") pod \"test-model-8c0e05-kserve-6797bf4db9-jdlmp\" (UID: \"75c9ba4f-dca7-4227-902f-97b2b8a262c6\") " pod="ai-tenant-e2e-inf-b-8c0e05/test-model-8c0e05-kserve-6797bf4db9-jdlmp"
Oct 30 07:49:30.853696 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:30.853691    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75c9ba4f-dca7-4227-902f-97b2b8a262c6-tmp-dir\") pod \"test-model-8c0e05-kserve-6797bf4db9-jdlmp\" (UID: \"75c9ba4f-dca7-4227-902f-97b2b8a262c6\") " pod="ai-tenant-e2e-inf-b-8c0e05/test-model-8c0e05-kserve-6797bf4db9-jdlmp"
Oct 30 07:49:30.954915 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:30.954878    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75c9ba4f-dca7-4227-902f-97b2b8a262c6-home\") pod \"test-model-8c0e05-kserve-6797bf4db9-jdlmp\" (UID: \"75c9ba4f-dca7-4227-902f-97b2b8a262c6\") " pod="ai-tenant-e2e-inf-b-8c0e05/test-model-8c0e05-kserve-6797bf4db9-jdlmp"
Oct 30 07:49:30.955122 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:30.954965    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75c9ba4f-dca7-4227-902f-97b2b8a262c6-model-cache\") pod \"test-model-8c0e05-kserve-6797bf4db9-jdlmp\" (UID: \"75c9ba4f-dca7-4227-902f-97b2b8a262c6\") " pod="ai-tenant-e2e-inf-b-8c0e05/test-model-8c0e05-kserve-6797bf4db9-jdlmp"
Oct 30 07:49:30.955122 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:30.955010    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75c9ba4f-dca7-4227-902f-97b2b8a262c6-tls-certs\") pod \"test-model-8c0e05-kserve-6797bf4db9-jdlmp\" (UID: \"75c9ba4f-dca7-4227-902f-97b2b8a262c6\") " pod="ai-tenant-e2e-inf-b-8c0e05/test-model-8c0e05-kserve-6797bf4db9-jdlmp"
Oct 30 07:49:30.955122 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:30.955045    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb7hx\" (UniqueName: \"kubernetes.io/projected/75c9ba4f-dca7-4227-902f-97b2b8a262c6-kube-api-access-vb7hx\") pod \"test-model-8c0e05-kserve-6797bf4db9-jdlmp\" (UID: \"75c9ba4f-dca7-4227-902f-97b2b8a262c6\") " pod="ai-tenant-e2e-inf-b-8c0e05/test-model-8c0e05-kserve-6797bf4db9-jdlmp"
Oct 30 07:49:30.955122 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:30.955068    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75c9ba4f-dca7-4227-902f-97b2b8a262c6-tmp-dir\") pod \"test-model-8c0e05-kserve-6797bf4db9-jdlmp\" (UID: \"75c9ba4f-dca7-4227-902f-97b2b8a262c6\") " pod="ai-tenant-e2e-inf-b-8c0e05/test-model-8c0e05-kserve-6797bf4db9-jdlmp"
Oct 30 07:49:30.955122 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:30.955117    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75c9ba4f-dca7-4227-902f-97b2b8a262c6-dshm\") pod \"test-model-8c0e05-kserve-6797bf4db9-jdlmp\" (UID: \"75c9ba4f-dca7-4227-902f-97b2b8a262c6\") " pod="ai-tenant-e2e-inf-b-8c0e05/test-model-8c0e05-kserve-6797bf4db9-jdlmp"
Oct 30 07:49:30.955428 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:30.955376    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75c9ba4f-dca7-4227-902f-97b2b8a262c6-home\") pod \"test-model-8c0e05-kserve-6797bf4db9-jdlmp\" (UID: \"75c9ba4f-dca7-4227-902f-97b2b8a262c6\") " pod="ai-tenant-e2e-inf-b-8c0e05/test-model-8c0e05-kserve-6797bf4db9-jdlmp"
Oct 30 07:49:30.955492 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:30.955468    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75c9ba4f-dca7-4227-902f-97b2b8a262c6-tmp-dir\") pod \"test-model-8c0e05-kserve-6797bf4db9-jdlmp\" (UID: \"75c9ba4f-dca7-4227-902f-97b2b8a262c6\") " pod="ai-tenant-e2e-inf-b-8c0e05/test-model-8c0e05-kserve-6797bf4db9-jdlmp"
Oct 30 07:49:30.956026 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:30.955996    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75c9ba4f-dca7-4227-902f-97b2b8a262c6-model-cache\") pod \"test-model-8c0e05-kserve-6797bf4db9-jdlmp\" (UID: \"75c9ba4f-dca7-4227-902f-97b2b8a262c6\") " pod="ai-tenant-e2e-inf-b-8c0e05/test-model-8c0e05-kserve-6797bf4db9-jdlmp"
Oct 30 07:49:30.958166 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:30.958139    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75c9ba4f-dca7-4227-902f-97b2b8a262c6-dshm\") pod \"test-model-8c0e05-kserve-6797bf4db9-jdlmp\" (UID: \"75c9ba4f-dca7-4227-902f-97b2b8a262c6\") " pod="ai-tenant-e2e-inf-b-8c0e05/test-model-8c0e05-kserve-6797bf4db9-jdlmp"
Oct 30 07:49:30.958441 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:30.958420    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75c9ba4f-dca7-4227-902f-97b2b8a262c6-tls-certs\") pod \"test-model-8c0e05-kserve-6797bf4db9-jdlmp\" (UID: \"75c9ba4f-dca7-4227-902f-97b2b8a262c6\") " pod="ai-tenant-e2e-inf-b-8c0e05/test-model-8c0e05-kserve-6797bf4db9-jdlmp"
Oct 30 07:49:30.980594 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:30.980559    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb7hx\" (UniqueName: \"kubernetes.io/projected/75c9ba4f-dca7-4227-902f-97b2b8a262c6-kube-api-access-vb7hx\") pod \"test-model-8c0e05-kserve-6797bf4db9-jdlmp\" (UID: \"75c9ba4f-dca7-4227-902f-97b2b8a262c6\") " pod="ai-tenant-e2e-inf-b-8c0e05/test-model-8c0e05-kserve-6797bf4db9-jdlmp"
Oct 30 07:49:31.123150 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:31.121552    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8c0e05/test-model-8c0e05-kserve-6797bf4db9-jdlmp"
Oct 30 07:49:31.448006 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:49:31.447976    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75c9ba4f_dca7_4227_902f_97b2b8a262c6.slice/crio-2158b214fd6c11f766605055b8ae457deec8d3409bfc3bb36be89ac74e1a77e3 WatchSource:0}: Error finding container 2158b214fd6c11f766605055b8ae457deec8d3409bfc3bb36be89ac74e1a77e3: Status 404 returned error can't find the container with id 2158b214fd6c11f766605055b8ae457deec8d3409bfc3bb36be89ac74e1a77e3
Oct 30 07:49:31.449762 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:31.449656    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-8c0e05/test-model-8c0e05-kserve-6797bf4db9-jdlmp"]
Oct 30 07:49:31.860930 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:49:31.860892    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:49:31.930057 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:31.930021    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8c0e05/test-model-8c0e05-kserve-6797bf4db9-jdlmp" event={"ID":"75c9ba4f-dca7-4227-902f-97b2b8a262c6","Type":"ContainerStarted","Data":"ffd9821fcb24f875151738e52a0e8c693be6bc4f78e5a026540b5c45804a8630"}
Oct 30 07:49:31.930223 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:31.930067    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8c0e05/test-model-8c0e05-kserve-6797bf4db9-jdlmp" event={"ID":"75c9ba4f-dca7-4227-902f-97b2b8a262c6","Type":"ContainerStarted","Data":"2158b214fd6c11f766605055b8ae457deec8d3409bfc3bb36be89ac74e1a77e3"}
Oct 30 07:49:31.982270 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:31.982206    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-8c0e05/test-model-8c0e05-kserve-6797bf4db9-jdlmp" podStartSLOduration=1.982187042 podStartE2EDuration="1.982187042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:49:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:49:31.976319132 +0000 UTC m=+3130.585726103" watchObservedRunningTime="2026-10-30 07:49:31.982187042 +0000 UTC m=+3130.591594012"
Oct 30 07:49:32.122978 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:32.122899    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-8c0e05/test-model-8c0e05-kserve-6797bf4db9-jdlmp"
Oct 30 07:49:39.102088 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:39.102046    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:49:41.122762 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:41.122721    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-8c0e05/test-model-8c0e05-kserve-6797bf4db9-jdlmp"
Oct 30 07:49:41.135946 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:41.135920    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-8c0e05/test-model-8c0e05-kserve-6797bf4db9-jdlmp"
Oct 30 07:49:42.026093 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:42.026062    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-8c0e05/test-model-8c0e05-kserve-6797bf4db9-jdlmp"
Oct 30 07:49:54.408833 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:54.406919    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ad434e/test-model-ad434e-kserve-9d799b5f7-zcfvt"]
Oct 30 07:49:58.448739 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:58.448705    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9"]
Oct 30 07:49:58.449230 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:49:58.448950    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9" podUID="7ee43b9a-1501-4a88-842e-564d9f1851f7" containerName="istio-proxy" containerID="cri-o://c61bb1da292be9fd9bb682d9ac0b1d2d3d91f9b401e7af65068ceb460c2ef6ea" gracePeriod=30
Oct 30 07:50:01.831893 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:01.831858    6224 patch_prober.go:29] interesting pod/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 07:50:01.834678 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:01.831913    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9" podUID="7ee43b9a-1501-4a88-842e-564d9f1851f7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 07:50:01.860325 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:50:01.860287    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:50:03.749428 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:03.749399    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9"
Oct 30 07:50:03.803642 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:03.803607    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ee43b9a-1501-4a88-842e-564d9f1851f7-kube-api-access-52w5d\" (UniqueName: \"kubernetes.io/projected/7ee43b9a-1501-4a88-842e-564d9f1851f7-kube-api-access-52w5d\") pod \"7ee43b9a-1501-4a88-842e-564d9f1851f7\" (UID: \"7ee43b9a-1501-4a88-842e-564d9f1851f7\") "
Oct 30 07:50:03.803831 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:03.803746    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ee43b9a-1501-4a88-842e-564d9f1851f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee43b9a-1501-4a88-842e-564d9f1851f7-istio-data\") pod \"7ee43b9a-1501-4a88-842e-564d9f1851f7\" (UID: \"7ee43b9a-1501-4a88-842e-564d9f1851f7\") "
Oct 30 07:50:03.803893 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:03.803835    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ee43b9a-1501-4a88-842e-564d9f1851f7-istio-token\" (UniqueName: \"kubernetes.io/projected/7ee43b9a-1501-4a88-842e-564d9f1851f7-istio-token\") pod \"7ee43b9a-1501-4a88-842e-564d9f1851f7\" (UID: \"7ee43b9a-1501-4a88-842e-564d9f1851f7\") "
Oct 30 07:50:03.803944 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:03.803920    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ee43b9a-1501-4a88-842e-564d9f1851f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee43b9a-1501-4a88-842e-564d9f1851f7-istio-envoy\") pod \"7ee43b9a-1501-4a88-842e-564d9f1851f7\" (UID: \"7ee43b9a-1501-4a88-842e-564d9f1851f7\") "
Oct 30 07:50:03.804031 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:03.804006    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ee43b9a-1501-4a88-842e-564d9f1851f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee43b9a-1501-4a88-842e-564d9f1851f7-workload-certs\") pod \"7ee43b9a-1501-4a88-842e-564d9f1851f7\" (UID: \"7ee43b9a-1501-4a88-842e-564d9f1851f7\") "
Oct 30 07:50:03.804121 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:03.804053    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ee43b9a-1501-4a88-842e-564d9f1851f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee43b9a-1501-4a88-842e-564d9f1851f7-istio-podinfo\") pod \"7ee43b9a-1501-4a88-842e-564d9f1851f7\" (UID: \"7ee43b9a-1501-4a88-842e-564d9f1851f7\") "
Oct 30 07:50:03.804121 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:03.804080    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee43b9a-1501-4a88-842e-564d9f1851f7-istio-data" pod "7ee43b9a-1501-4a88-842e-564d9f1851f7" (UID: "7ee43b9a-1501-4a88-842e-564d9f1851f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:50:03.804243 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:03.804124    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ee43b9a-1501-4a88-842e-564d9f1851f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee43b9a-1501-4a88-842e-564d9f1851f7-workload-socket\") pod \"7ee43b9a-1501-4a88-842e-564d9f1851f7\" (UID: \"7ee43b9a-1501-4a88-842e-564d9f1851f7\") "
Oct 30 07:50:03.804243 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:03.804205    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ee43b9a-1501-4a88-842e-564d9f1851f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee43b9a-1501-4a88-842e-564d9f1851f7-credential-socket\") pod \"7ee43b9a-1501-4a88-842e-564d9f1851f7\" (UID: \"7ee43b9a-1501-4a88-842e-564d9f1851f7\") "
Oct 30 07:50:03.804366 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:03.804245    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ee43b9a-1501-4a88-842e-564d9f1851f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee43b9a-1501-4a88-842e-564d9f1851f7-istiod-ca-cert\") pod \"7ee43b9a-1501-4a88-842e-564d9f1851f7\" (UID: \"7ee43b9a-1501-4a88-842e-564d9f1851f7\") "
Oct 30 07:50:03.804736 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:03.804441    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee43b9a-1501-4a88-842e-564d9f1851f7-workload-socket" pod "7ee43b9a-1501-4a88-842e-564d9f1851f7" (UID: "7ee43b9a-1501-4a88-842e-564d9f1851f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:50:03.804736 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:03.804492    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee43b9a-1501-4a88-842e-564d9f1851f7-workload-certs" pod "7ee43b9a-1501-4a88-842e-564d9f1851f7" (UID: "7ee43b9a-1501-4a88-842e-564d9f1851f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:50:03.804736 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:03.804663    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee43b9a-1501-4a88-842e-564d9f1851f7-credential-socket" pod "7ee43b9a-1501-4a88-842e-564d9f1851f7" (UID: "7ee43b9a-1501-4a88-842e-564d9f1851f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:50:03.804964 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:03.804835    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ee43b9a-1501-4a88-842e-564d9f1851f7-istiod-ca-cert" pod "7ee43b9a-1501-4a88-842e-564d9f1851f7" (UID: "7ee43b9a-1501-4a88-842e-564d9f1851f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:50:03.805048 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:03.805031    6224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee43b9a-1501-4a88-842e-564d9f1851f7-workload-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:50:03.805112 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:03.805053    6224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee43b9a-1501-4a88-842e-564d9f1851f7-workload-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:50:03.805112 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:03.805068    6224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee43b9a-1501-4a88-842e-564d9f1851f7-credential-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:50:03.805112 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:03.805081    6224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee43b9a-1501-4a88-842e-564d9f1851f7-istiod-ca-cert\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:50:03.805112 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:03.805095    6224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee43b9a-1501-4a88-842e-564d9f1851f7-istio-data\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:50:03.806449 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:03.806419    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee43b9a-1501-4a88-842e-564d9f1851f7-istio-envoy" pod "7ee43b9a-1501-4a88-842e-564d9f1851f7" (UID: "7ee43b9a-1501-4a88-842e-564d9f1851f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:50:03.806561 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:03.806457    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ee43b9a-1501-4a88-842e-564d9f1851f7-kube-api-access-52w5d" pod "7ee43b9a-1501-4a88-842e-564d9f1851f7" (UID: "7ee43b9a-1501-4a88-842e-564d9f1851f7"). InnerVolumeSpecName "kube-api-access-52w5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:50:03.806561 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:03.806508    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ee43b9a-1501-4a88-842e-564d9f1851f7-istio-token" pod "7ee43b9a-1501-4a88-842e-564d9f1851f7" (UID: "7ee43b9a-1501-4a88-842e-564d9f1851f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:50:03.806689 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:03.806655    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ee43b9a-1501-4a88-842e-564d9f1851f7-istio-podinfo" pod "7ee43b9a-1501-4a88-842e-564d9f1851f7" (UID: "7ee43b9a-1501-4a88-842e-564d9f1851f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:50:03.906277 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:03.906247    6224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee43b9a-1501-4a88-842e-564d9f1851f7-istio-envoy\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:50:03.906277 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:03.906275    6224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee43b9a-1501-4a88-842e-564d9f1851f7-istio-podinfo\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:50:03.906491 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:03.906285    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52w5d\" (UniqueName: \"kubernetes.io/projected/7ee43b9a-1501-4a88-842e-564d9f1851f7-kube-api-access-52w5d\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:50:03.906491 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:03.906296    6224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ee43b9a-1501-4a88-842e-564d9f1851f7-istio-token\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:50:04.202249 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:04.202209    6224 generic.go:356] "Generic (PLEG): container finished" podID="7ee43b9a-1501-4a88-842e-564d9f1851f7" containerID="c61bb1da292be9fd9bb682d9ac0b1d2d3d91f9b401e7af65068ceb460c2ef6ea" exitCode=0
Oct 30 07:50:04.202433 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:04.202276    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9"
Oct 30 07:50:04.202433 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:04.202293    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9" event={"ID":"7ee43b9a-1501-4a88-842e-564d9f1851f7","Type":"ContainerDied","Data":"c61bb1da292be9fd9bb682d9ac0b1d2d3d91f9b401e7af65068ceb460c2ef6ea"}
Oct 30 07:50:04.202433 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:04.202371    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9" event={"ID":"7ee43b9a-1501-4a88-842e-564d9f1851f7","Type":"ContainerDied","Data":"a4c9d81d2c6a227307ac350089422c58e8fa0ad12527a957b752806ab8b7f600"}
Oct 30 07:50:04.202433 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:04.202394    6224 scope.go:122] "RemoveContainer" containerID="c61bb1da292be9fd9bb682d9ac0b1d2d3d91f9b401e7af65068ceb460c2ef6ea"
Oct 30 07:50:04.215382 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:04.215364    6224 scope.go:122] "RemoveContainer" containerID="c61bb1da292be9fd9bb682d9ac0b1d2d3d91f9b401e7af65068ceb460c2ef6ea"
Oct 30 07:50:04.215643 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:50:04.215624    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c61bb1da292be9fd9bb682d9ac0b1d2d3d91f9b401e7af65068ceb460c2ef6ea\": container with ID starting with c61bb1da292be9fd9bb682d9ac0b1d2d3d91f9b401e7af65068ceb460c2ef6ea not found: ID does not exist" containerID="c61bb1da292be9fd9bb682d9ac0b1d2d3d91f9b401e7af65068ceb460c2ef6ea"
Oct 30 07:50:04.215695 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:04.215654    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c61bb1da292be9fd9bb682d9ac0b1d2d3d91f9b401e7af65068ceb460c2ef6ea"} err="failed to get container status \"c61bb1da292be9fd9bb682d9ac0b1d2d3d91f9b401e7af65068ceb460c2ef6ea\": rpc error: code = NotFound desc = could not find container \"c61bb1da292be9fd9bb682d9ac0b1d2d3d91f9b401e7af65068ceb460c2ef6ea\": container with ID starting with c61bb1da292be9fd9bb682d9ac0b1d2d3d91f9b401e7af65068ceb460c2ef6ea not found: ID does not exist"
Oct 30 07:50:04.262384 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:04.262354    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9"]
Oct 30 07:50:04.276198 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:04.276171    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-9z7w9"]
Oct 30 07:50:05.506659 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:05.506628    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ee43b9a-1501-4a88-842e-564d9f1851f7" path="/var/lib/kubelet/pods/7ee43b9a-1501-4a88-842e-564d9f1851f7/volumes"
Oct 30 07:50:06.235727 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:06.235692    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl"]
Oct 30 07:50:06.235947 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:06.235909    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl" podUID="9cb25921-0688-433d-90ec-e8ea626437b8" containerName="istio-proxy" containerID="cri-o://c7733c3c8cccb7a27ca9753c29eda16382457b61e2d23e953a4026530ee00515" gracePeriod=30
Oct 30 07:50:07.516839 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:07.516790    6224 patch_prober.go:29] interesting pod/e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 07:50:07.517233 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:07.516874    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl" podUID="9cb25921-0688-433d-90ec-e8ea626437b8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 07:50:09.459627 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:09.459589    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-ad434e/test-model-ad434e-kserve-9d799b5f7-zcfvt" podUID="6fd17eaf-b363-4ddb-8dc3-29489cb00a87" containerName="main" containerID="cri-o://733de4fa65bb32571beb7c0a1ce7353a9ced9e13a1177d29b573d615c25589e2" gracePeriod=45
Oct 30 07:50:09.754524 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:09.754496    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ad434e/test-model-ad434e-kserve-9d799b5f7-zcfvt"
Oct 30 07:50:09.892782 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:09.892749    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fd17eaf-b363-4ddb-8dc3-29489cb00a87-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fd17eaf-b363-4ddb-8dc3-29489cb00a87-model-cache\") pod \"6fd17eaf-b363-4ddb-8dc3-29489cb00a87\" (UID: \"6fd17eaf-b363-4ddb-8dc3-29489cb00a87\") "
Oct 30 07:50:09.893064 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:09.892871    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fd17eaf-b363-4ddb-8dc3-29489cb00a87-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fd17eaf-b363-4ddb-8dc3-29489cb00a87-dshm\") pod \"6fd17eaf-b363-4ddb-8dc3-29489cb00a87\" (UID: \"6fd17eaf-b363-4ddb-8dc3-29489cb00a87\") "
Oct 30 07:50:09.893064 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:09.892985    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fd17eaf-b363-4ddb-8dc3-29489cb00a87-home\" (UniqueName: \"kubernetes.io/empty-dir/6fd17eaf-b363-4ddb-8dc3-29489cb00a87-home\") pod \"6fd17eaf-b363-4ddb-8dc3-29489cb00a87\" (UID: \"6fd17eaf-b363-4ddb-8dc3-29489cb00a87\") "
Oct 30 07:50:09.893064 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:09.893001    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fd17eaf-b363-4ddb-8dc3-29489cb00a87-model-cache" pod "6fd17eaf-b363-4ddb-8dc3-29489cb00a87" (UID: "6fd17eaf-b363-4ddb-8dc3-29489cb00a87"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:50:09.893215 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:09.893148    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fd17eaf-b363-4ddb-8dc3-29489cb00a87-tls-certs\" (UniqueName: \"kubernetes.io/secret/6fd17eaf-b363-4ddb-8dc3-29489cb00a87-tls-certs\") pod \"6fd17eaf-b363-4ddb-8dc3-29489cb00a87\" (UID: \"6fd17eaf-b363-4ddb-8dc3-29489cb00a87\") "
Oct 30 07:50:09.893258 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:09.893221    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fd17eaf-b363-4ddb-8dc3-29489cb00a87-home" pod "6fd17eaf-b363-4ddb-8dc3-29489cb00a87" (UID: "6fd17eaf-b363-4ddb-8dc3-29489cb00a87"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:50:09.893258 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:09.893229    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fd17eaf-b363-4ddb-8dc3-29489cb00a87-kube-api-access-48wpx\" (UniqueName: \"kubernetes.io/projected/6fd17eaf-b363-4ddb-8dc3-29489cb00a87-kube-api-access-48wpx\") pod \"6fd17eaf-b363-4ddb-8dc3-29489cb00a87\" (UID: \"6fd17eaf-b363-4ddb-8dc3-29489cb00a87\") "
Oct 30 07:50:09.893345 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:09.893291    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fd17eaf-b363-4ddb-8dc3-29489cb00a87-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fd17eaf-b363-4ddb-8dc3-29489cb00a87-tmp-dir\") pod \"6fd17eaf-b363-4ddb-8dc3-29489cb00a87\" (UID: \"6fd17eaf-b363-4ddb-8dc3-29489cb00a87\") "
Oct 30 07:50:09.893604 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:09.893575    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fd17eaf-b363-4ddb-8dc3-29489cb00a87-tmp-dir" pod "6fd17eaf-b363-4ddb-8dc3-29489cb00a87" (UID: "6fd17eaf-b363-4ddb-8dc3-29489cb00a87"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:50:09.893965 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:09.893946    6224 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fd17eaf-b363-4ddb-8dc3-29489cb00a87-tmp-dir\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:50:09.894060 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:09.893969    6224 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fd17eaf-b363-4ddb-8dc3-29489cb00a87-model-cache\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:50:09.894060 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:09.893982    6224 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fd17eaf-b363-4ddb-8dc3-29489cb00a87-home\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:50:09.895311 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:09.895280    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fd17eaf-b363-4ddb-8dc3-29489cb00a87-dshm" pod "6fd17eaf-b363-4ddb-8dc3-29489cb00a87" (UID: "6fd17eaf-b363-4ddb-8dc3-29489cb00a87"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:50:09.895434 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:09.895357    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fd17eaf-b363-4ddb-8dc3-29489cb00a87-tls-certs" pod "6fd17eaf-b363-4ddb-8dc3-29489cb00a87" (UID: "6fd17eaf-b363-4ddb-8dc3-29489cb00a87"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:50:09.895434 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:09.895371    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fd17eaf-b363-4ddb-8dc3-29489cb00a87-kube-api-access-48wpx" pod "6fd17eaf-b363-4ddb-8dc3-29489cb00a87" (UID: "6fd17eaf-b363-4ddb-8dc3-29489cb00a87"). InnerVolumeSpecName "kube-api-access-48wpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:50:09.995283 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:09.995214    6224 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fd17eaf-b363-4ddb-8dc3-29489cb00a87-tls-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:50:09.995283 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:09.995242    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48wpx\" (UniqueName: \"kubernetes.io/projected/6fd17eaf-b363-4ddb-8dc3-29489cb00a87-kube-api-access-48wpx\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:50:09.995283 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:09.995251    6224 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fd17eaf-b363-4ddb-8dc3-29489cb00a87-dshm\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:50:10.253990 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:10.253907    6224 generic.go:356] "Generic (PLEG): container finished" podID="6fd17eaf-b363-4ddb-8dc3-29489cb00a87" containerID="733de4fa65bb32571beb7c0a1ce7353a9ced9e13a1177d29b573d615c25589e2" exitCode=0
Oct 30 07:50:10.253990 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:10.253960    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ad434e/test-model-ad434e-kserve-9d799b5f7-zcfvt"
Oct 30 07:50:10.254168 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:10.253959    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ad434e/test-model-ad434e-kserve-9d799b5f7-zcfvt" event={"ID":"6fd17eaf-b363-4ddb-8dc3-29489cb00a87","Type":"ContainerDied","Data":"733de4fa65bb32571beb7c0a1ce7353a9ced9e13a1177d29b573d615c25589e2"}
Oct 30 07:50:10.254168 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:10.254077    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ad434e/test-model-ad434e-kserve-9d799b5f7-zcfvt" event={"ID":"6fd17eaf-b363-4ddb-8dc3-29489cb00a87","Type":"ContainerDied","Data":"39f4e929e3720b4cd95c42b86fca7934c75957753c423e3e9bea4b1491dba547"}
Oct 30 07:50:10.254168 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:10.254103    6224 scope.go:122] "RemoveContainer" containerID="733de4fa65bb32571beb7c0a1ce7353a9ced9e13a1177d29b573d615c25589e2"
Oct 30 07:50:10.265703 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:10.265685    6224 scope.go:122] "RemoveContainer" containerID="733de4fa65bb32571beb7c0a1ce7353a9ced9e13a1177d29b573d615c25589e2"
Oct 30 07:50:10.265975 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:50:10.265955    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"733de4fa65bb32571beb7c0a1ce7353a9ced9e13a1177d29b573d615c25589e2\": container with ID starting with 733de4fa65bb32571beb7c0a1ce7353a9ced9e13a1177d29b573d615c25589e2 not found: ID does not exist" containerID="733de4fa65bb32571beb7c0a1ce7353a9ced9e13a1177d29b573d615c25589e2"
Oct 30 07:50:10.266037 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:10.265982    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"733de4fa65bb32571beb7c0a1ce7353a9ced9e13a1177d29b573d615c25589e2"} err="failed to get container status \"733de4fa65bb32571beb7c0a1ce7353a9ced9e13a1177d29b573d615c25589e2\": rpc error: code = NotFound desc = could not find container \"733de4fa65bb32571beb7c0a1ce7353a9ced9e13a1177d29b573d615c25589e2\": container with ID starting with 733de4fa65bb32571beb7c0a1ce7353a9ced9e13a1177d29b573d615c25589e2 not found: ID does not exist"
Oct 30 07:50:10.312766 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:10.312741    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ad434e/test-model-ad434e-kserve-9d799b5f7-zcfvt"]
Oct 30 07:50:10.318321 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:10.318300    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-ad434e/test-model-ad434e-kserve-9d799b5f7-zcfvt"]
Oct 30 07:50:11.508051 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:11.507981    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fd17eaf-b363-4ddb-8dc3-29489cb00a87" path="/var/lib/kubelet/pods/6fd17eaf-b363-4ddb-8dc3-29489cb00a87/volumes"
Oct 30 07:50:11.517037 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:11.517016    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl"
Oct 30 07:50:11.618906 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:11.618872    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cb25921-0688-433d-90ec-e8ea626437b8-kube-api-access-ljzmw\" (UniqueName: \"kubernetes.io/projected/9cb25921-0688-433d-90ec-e8ea626437b8-kube-api-access-ljzmw\") pod \"9cb25921-0688-433d-90ec-e8ea626437b8\" (UID: \"9cb25921-0688-433d-90ec-e8ea626437b8\") "
Oct 30 07:50:11.619080 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:11.619003    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cb25921-0688-433d-90ec-e8ea626437b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cb25921-0688-433d-90ec-e8ea626437b8-istio-envoy\") pod \"9cb25921-0688-433d-90ec-e8ea626437b8\" (UID: \"9cb25921-0688-433d-90ec-e8ea626437b8\") "
Oct 30 07:50:11.619080 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:11.619033    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cb25921-0688-433d-90ec-e8ea626437b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb25921-0688-433d-90ec-e8ea626437b8-workload-socket\") pod \"9cb25921-0688-433d-90ec-e8ea626437b8\" (UID: \"9cb25921-0688-433d-90ec-e8ea626437b8\") "
Oct 30 07:50:11.619192 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:11.619107    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cb25921-0688-433d-90ec-e8ea626437b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cb25921-0688-433d-90ec-e8ea626437b8-workload-certs\") pod \"9cb25921-0688-433d-90ec-e8ea626437b8\" (UID: \"9cb25921-0688-433d-90ec-e8ea626437b8\") "
Oct 30 07:50:11.619192 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:11.619133    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cb25921-0688-433d-90ec-e8ea626437b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cb25921-0688-433d-90ec-e8ea626437b8-istiod-ca-cert\") pod \"9cb25921-0688-433d-90ec-e8ea626437b8\" (UID: \"9cb25921-0688-433d-90ec-e8ea626437b8\") "
Oct 30 07:50:11.619192 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:11.619170    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9cb25921-0688-433d-90ec-e8ea626437b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cb25921-0688-433d-90ec-e8ea626437b8-istio-podinfo\") pod \"9cb25921-0688-433d-90ec-e8ea626437b8\" (UID: \"9cb25921-0688-433d-90ec-e8ea626437b8\") "
Oct 30 07:50:11.619355 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:11.619198    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cb25921-0688-433d-90ec-e8ea626437b8-istio-token\" (UniqueName: \"kubernetes.io/projected/9cb25921-0688-433d-90ec-e8ea626437b8-istio-token\") pod \"9cb25921-0688-433d-90ec-e8ea626437b8\" (UID: \"9cb25921-0688-433d-90ec-e8ea626437b8\") "
Oct 30 07:50:11.619355 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:11.619284    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cb25921-0688-433d-90ec-e8ea626437b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cb25921-0688-433d-90ec-e8ea626437b8-istio-data\") pod \"9cb25921-0688-433d-90ec-e8ea626437b8\" (UID: \"9cb25921-0688-433d-90ec-e8ea626437b8\") "
Oct 30 07:50:11.619355 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:11.619335    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cb25921-0688-433d-90ec-e8ea626437b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb25921-0688-433d-90ec-e8ea626437b8-credential-socket\") pod \"9cb25921-0688-433d-90ec-e8ea626437b8\" (UID: \"9cb25921-0688-433d-90ec-e8ea626437b8\") "
Oct 30 07:50:11.619521 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:11.619412    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cb25921-0688-433d-90ec-e8ea626437b8-workload-socket" pod "9cb25921-0688-433d-90ec-e8ea626437b8" (UID: "9cb25921-0688-433d-90ec-e8ea626437b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:50:11.619578 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:11.619529    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cb25921-0688-433d-90ec-e8ea626437b8-istiod-ca-cert" pod "9cb25921-0688-433d-90ec-e8ea626437b8" (UID: "9cb25921-0688-433d-90ec-e8ea626437b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:50:11.620060 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:11.619627    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cb25921-0688-433d-90ec-e8ea626437b8-credential-socket" pod "9cb25921-0688-433d-90ec-e8ea626437b8" (UID: "9cb25921-0688-433d-90ec-e8ea626437b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:50:11.620060 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:11.619842    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cb25921-0688-433d-90ec-e8ea626437b8-istio-data" pod "9cb25921-0688-433d-90ec-e8ea626437b8" (UID: "9cb25921-0688-433d-90ec-e8ea626437b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:50:11.620060 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:11.619859    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cb25921-0688-433d-90ec-e8ea626437b8-workload-certs" pod "9cb25921-0688-433d-90ec-e8ea626437b8" (UID: "9cb25921-0688-433d-90ec-e8ea626437b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:50:11.620546 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:11.620519    6224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb25921-0688-433d-90ec-e8ea626437b8-workload-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:50:11.620634 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:11.620548    6224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cb25921-0688-433d-90ec-e8ea626437b8-workload-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:50:11.620634 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:11.620562    6224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cb25921-0688-433d-90ec-e8ea626437b8-istiod-ca-cert\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:50:11.620634 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:11.620576    6224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cb25921-0688-433d-90ec-e8ea626437b8-istio-data\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:50:11.620634 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:11.620588    6224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb25921-0688-433d-90ec-e8ea626437b8-credential-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:50:11.621652 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:11.621624    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cb25921-0688-433d-90ec-e8ea626437b8-istio-envoy" pod "9cb25921-0688-433d-90ec-e8ea626437b8" (UID: "9cb25921-0688-433d-90ec-e8ea626437b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:50:11.622562 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:11.622540    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cb25921-0688-433d-90ec-e8ea626437b8-kube-api-access-ljzmw" pod "9cb25921-0688-433d-90ec-e8ea626437b8" (UID: "9cb25921-0688-433d-90ec-e8ea626437b8"). InnerVolumeSpecName "kube-api-access-ljzmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:50:11.622727 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:11.622630    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9cb25921-0688-433d-90ec-e8ea626437b8-istio-podinfo" pod "9cb25921-0688-433d-90ec-e8ea626437b8" (UID: "9cb25921-0688-433d-90ec-e8ea626437b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:50:11.622727 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:11.622700    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cb25921-0688-433d-90ec-e8ea626437b8-istio-token" pod "9cb25921-0688-433d-90ec-e8ea626437b8" (UID: "9cb25921-0688-433d-90ec-e8ea626437b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:50:11.722056 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:11.722023    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljzmw\" (UniqueName: \"kubernetes.io/projected/9cb25921-0688-433d-90ec-e8ea626437b8-kube-api-access-ljzmw\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:50:11.722056 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:11.722053    6224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cb25921-0688-433d-90ec-e8ea626437b8-istio-envoy\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:50:11.722056 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:11.722063    6224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cb25921-0688-433d-90ec-e8ea626437b8-istio-podinfo\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:50:11.722291 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:11.722071    6224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cb25921-0688-433d-90ec-e8ea626437b8-istio-token\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:50:12.245341 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:12.245305    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4ab37e/test-model-4ab37e-kserve-75448f6fc5-ln8dd"]
Oct 30 07:50:12.280343 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:12.280306    6224 generic.go:356] "Generic (PLEG): container finished" podID="9cb25921-0688-433d-90ec-e8ea626437b8" containerID="c7733c3c8cccb7a27ca9753c29eda16382457b61e2d23e953a4026530ee00515" exitCode=0
Oct 30 07:50:12.280535 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:12.280436    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl"
Oct 30 07:50:12.280535 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:12.280453    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl" event={"ID":"9cb25921-0688-433d-90ec-e8ea626437b8","Type":"ContainerDied","Data":"c7733c3c8cccb7a27ca9753c29eda16382457b61e2d23e953a4026530ee00515"}
Oct 30 07:50:12.280535 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:12.280525    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl" event={"ID":"9cb25921-0688-433d-90ec-e8ea626437b8","Type":"ContainerDied","Data":"a40fb3ff0297de113ad336b7313d3303d19e1e0b6cb4dd528ddbab125c79c9e3"}
Oct 30 07:50:12.280693 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:12.280557    6224 scope.go:122] "RemoveContainer" containerID="c7733c3c8cccb7a27ca9753c29eda16382457b61e2d23e953a4026530ee00515"
Oct 30 07:50:12.321255 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:12.321223    6224 scope.go:122] "RemoveContainer" containerID="c7733c3c8cccb7a27ca9753c29eda16382457b61e2d23e953a4026530ee00515"
Oct 30 07:50:12.321754 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:50:12.321722    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7733c3c8cccb7a27ca9753c29eda16382457b61e2d23e953a4026530ee00515\": container with ID starting with c7733c3c8cccb7a27ca9753c29eda16382457b61e2d23e953a4026530ee00515 not found: ID does not exist" containerID="c7733c3c8cccb7a27ca9753c29eda16382457b61e2d23e953a4026530ee00515"
Oct 30 07:50:12.322328 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:12.322302    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7733c3c8cccb7a27ca9753c29eda16382457b61e2d23e953a4026530ee00515"} err="failed to get container status \"c7733c3c8cccb7a27ca9753c29eda16382457b61e2d23e953a4026530ee00515\": rpc error: code = NotFound desc = could not find container \"c7733c3c8cccb7a27ca9753c29eda16382457b61e2d23e953a4026530ee00515\": container with ID starting with c7733c3c8cccb7a27ca9753c29eda16382457b61e2d23e953a4026530ee00515 not found: ID does not exist"
Oct 30 07:50:12.432896 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:12.431857    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl"]
Oct 30 07:50:12.436824 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:12.435911    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-06b26e-openshift-default-76cd74f47-pw7pl"]
Oct 30 07:50:13.506453 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:13.506420    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cb25921-0688-433d-90ec-e8ea626437b8" path="/var/lib/kubelet/pods/9cb25921-0688-433d-90ec-e8ea626437b8/volumes"
Oct 30 07:50:21.241900 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:21.241862    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq"]
Oct 30 07:50:21.242306 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:21.242118    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq" podUID="717ebd45-f957-47c5-a710-d091051fa4eb" containerName="istio-proxy" containerID="cri-o://4ebe55dec33f95d5261d16444187347773d09fbb8c54ededdcf301de2a15312f" gracePeriod=30
Oct 30 07:50:22.406997 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:22.406960    6224 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 07:50:22.407389 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:22.407024    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq" podUID="717ebd45-f957-47c5-a710-d091051fa4eb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 07:50:23.113268 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:23.113234    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ad434e-54db8b9d-v5xv5"]
Oct 30 07:50:26.412574 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:26.412536    6224 generic.go:356] "Generic (PLEG): container finished" podID="717ebd45-f957-47c5-a710-d091051fa4eb" containerID="4ebe55dec33f95d5261d16444187347773d09fbb8c54ededdcf301de2a15312f" exitCode=0
Oct 30 07:50:26.413008 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:26.412611    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq" event={"ID":"717ebd45-f957-47c5-a710-d091051fa4eb","Type":"ContainerDied","Data":"4ebe55dec33f95d5261d16444187347773d09fbb8c54ededdcf301de2a15312f"}
Oct 30 07:50:26.600161 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:26.600116    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq"
Oct 30 07:50:26.707570 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:26.707296    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/717ebd45-f957-47c5-a710-d091051fa4eb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/717ebd45-f957-47c5-a710-d091051fa4eb-istio-envoy\") pod \"717ebd45-f957-47c5-a710-d091051fa4eb\" (UID: \"717ebd45-f957-47c5-a710-d091051fa4eb\") "
Oct 30 07:50:26.707570 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:26.707413    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/717ebd45-f957-47c5-a710-d091051fa4eb-kube-api-access-c8fdc\" (UniqueName: \"kubernetes.io/projected/717ebd45-f957-47c5-a710-d091051fa4eb-kube-api-access-c8fdc\") pod \"717ebd45-f957-47c5-a710-d091051fa4eb\" (UID: \"717ebd45-f957-47c5-a710-d091051fa4eb\") "
Oct 30 07:50:26.707570 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:26.707492    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/717ebd45-f957-47c5-a710-d091051fa4eb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/717ebd45-f957-47c5-a710-d091051fa4eb-istio-data\") pod \"717ebd45-f957-47c5-a710-d091051fa4eb\" (UID: \"717ebd45-f957-47c5-a710-d091051fa4eb\") "
Oct 30 07:50:26.708632 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:26.707615    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/717ebd45-f957-47c5-a710-d091051fa4eb-istio-token\" (UniqueName: \"kubernetes.io/projected/717ebd45-f957-47c5-a710-d091051fa4eb-istio-token\") pod \"717ebd45-f957-47c5-a710-d091051fa4eb\" (UID: \"717ebd45-f957-47c5-a710-d091051fa4eb\") "
Oct 30 07:50:26.708632 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:26.707655    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/717ebd45-f957-47c5-a710-d091051fa4eb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/717ebd45-f957-47c5-a710-d091051fa4eb-istiod-ca-cert\") pod \"717ebd45-f957-47c5-a710-d091051fa4eb\" (UID: \"717ebd45-f957-47c5-a710-d091051fa4eb\") "
Oct 30 07:50:26.708632 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:26.707720    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/717ebd45-f957-47c5-a710-d091051fa4eb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/717ebd45-f957-47c5-a710-d091051fa4eb-credential-socket\") pod \"717ebd45-f957-47c5-a710-d091051fa4eb\" (UID: \"717ebd45-f957-47c5-a710-d091051fa4eb\") "
Oct 30 07:50:26.708632 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:26.707821    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/717ebd45-f957-47c5-a710-d091051fa4eb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/717ebd45-f957-47c5-a710-d091051fa4eb-istio-podinfo\") pod \"717ebd45-f957-47c5-a710-d091051fa4eb\" (UID: \"717ebd45-f957-47c5-a710-d091051fa4eb\") "
Oct 30 07:50:26.708632 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:26.707866    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/717ebd45-f957-47c5-a710-d091051fa4eb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/717ebd45-f957-47c5-a710-d091051fa4eb-workload-certs\") pod \"717ebd45-f957-47c5-a710-d091051fa4eb\" (UID: \"717ebd45-f957-47c5-a710-d091051fa4eb\") "
Oct 30 07:50:26.708632 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:26.707900    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/717ebd45-f957-47c5-a710-d091051fa4eb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/717ebd45-f957-47c5-a710-d091051fa4eb-workload-socket\") pod \"717ebd45-f957-47c5-a710-d091051fa4eb\" (UID: \"717ebd45-f957-47c5-a710-d091051fa4eb\") "
Oct 30 07:50:26.709096 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:26.709068    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/717ebd45-f957-47c5-a710-d091051fa4eb-workload-socket" pod "717ebd45-f957-47c5-a710-d091051fa4eb" (UID: "717ebd45-f957-47c5-a710-d091051fa4eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:50:26.709854 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:26.709798    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/717ebd45-f957-47c5-a710-d091051fa4eb-istiod-ca-cert" pod "717ebd45-f957-47c5-a710-d091051fa4eb" (UID: "717ebd45-f957-47c5-a710-d091051fa4eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:50:26.710580 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:26.710460    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/717ebd45-f957-47c5-a710-d091051fa4eb-istio-data" pod "717ebd45-f957-47c5-a710-d091051fa4eb" (UID: "717ebd45-f957-47c5-a710-d091051fa4eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:50:26.711034 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:26.710731    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/717ebd45-f957-47c5-a710-d091051fa4eb-credential-socket" pod "717ebd45-f957-47c5-a710-d091051fa4eb" (UID: "717ebd45-f957-47c5-a710-d091051fa4eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:50:26.711034 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:26.711002    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/717ebd45-f957-47c5-a710-d091051fa4eb-workload-certs" pod "717ebd45-f957-47c5-a710-d091051fa4eb" (UID: "717ebd45-f957-47c5-a710-d091051fa4eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:50:26.712338 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:26.711787    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/717ebd45-f957-47c5-a710-d091051fa4eb-istio-envoy" pod "717ebd45-f957-47c5-a710-d091051fa4eb" (UID: "717ebd45-f957-47c5-a710-d091051fa4eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:50:26.713859 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:26.713330    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/717ebd45-f957-47c5-a710-d091051fa4eb-kube-api-access-c8fdc" pod "717ebd45-f957-47c5-a710-d091051fa4eb" (UID: "717ebd45-f957-47c5-a710-d091051fa4eb"). InnerVolumeSpecName "kube-api-access-c8fdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:50:26.713859 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:26.713446    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/717ebd45-f957-47c5-a710-d091051fa4eb-istio-token" pod "717ebd45-f957-47c5-a710-d091051fa4eb" (UID: "717ebd45-f957-47c5-a710-d091051fa4eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:50:26.716589 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:26.716561    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/717ebd45-f957-47c5-a710-d091051fa4eb-istio-podinfo" pod "717ebd45-f957-47c5-a710-d091051fa4eb" (UID: "717ebd45-f957-47c5-a710-d091051fa4eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:50:26.810187 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:26.810102    6224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/717ebd45-f957-47c5-a710-d091051fa4eb-credential-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:50:26.810187 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:26.810129    6224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/717ebd45-f957-47c5-a710-d091051fa4eb-istio-podinfo\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:50:26.810187 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:26.810139    6224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/717ebd45-f957-47c5-a710-d091051fa4eb-workload-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:50:26.810187 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:26.810150    6224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/717ebd45-f957-47c5-a710-d091051fa4eb-workload-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:50:26.810187 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:26.810159    6224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/717ebd45-f957-47c5-a710-d091051fa4eb-istio-envoy\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:50:26.810187 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:26.810168    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8fdc\" (UniqueName: \"kubernetes.io/projected/717ebd45-f957-47c5-a710-d091051fa4eb-kube-api-access-c8fdc\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:50:26.810187 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:26.810179    6224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/717ebd45-f957-47c5-a710-d091051fa4eb-istio-data\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:50:26.810187 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:26.810187    6224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/717ebd45-f957-47c5-a710-d091051fa4eb-istio-token\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:50:26.810187 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:26.810194    6224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/717ebd45-f957-47c5-a710-d091051fa4eb-istiod-ca-cert\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:50:27.284828 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:27.281936    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-4ab37e/test-model-4ab37e-kserve-75448f6fc5-ln8dd" podUID="bf2c853f-e673-498f-9135-2e963890f12e" containerName="main" containerID="cri-o://f01d5ab0da57da3f5e565c274402192bb3e3011b203c6537a3e137d14763a726" gracePeriod=45
Oct 30 07:50:27.424252 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:27.424217    6224 generic.go:356] "Generic (PLEG): container finished" podID="bf2c853f-e673-498f-9135-2e963890f12e" containerID="f01d5ab0da57da3f5e565c274402192bb3e3011b203c6537a3e137d14763a726" exitCode=0
Oct 30 07:50:27.424685 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:27.424265    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4ab37e/test-model-4ab37e-kserve-75448f6fc5-ln8dd" event={"ID":"bf2c853f-e673-498f-9135-2e963890f12e","Type":"ContainerDied","Data":"f01d5ab0da57da3f5e565c274402192bb3e3011b203c6537a3e137d14763a726"}
Oct 30 07:50:27.426325 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:27.426297    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq" event={"ID":"717ebd45-f957-47c5-a710-d091051fa4eb","Type":"ContainerDied","Data":"10afb0de805bd5a30b412de7ea796c14823111412f9a959b1bc0e650cf2d680f"}
Oct 30 07:50:27.426479 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:27.426322    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq"
Oct 30 07:50:27.426479 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:27.426371    6224 scope.go:122] "RemoveContainer" containerID="4ebe55dec33f95d5261d16444187347773d09fbb8c54ededdcf301de2a15312f"
Oct 30 07:50:27.524316 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:27.522368    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq"]
Oct 30 07:50:27.524316 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:27.524276    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe2ed-openshift-default-d7c86499-t6dwq"]
Oct 30 07:50:27.647152 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:27.647119    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4ab37e/test-model-4ab37e-kserve-75448f6fc5-ln8dd"
Oct 30 07:50:27.828017 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:27.827987    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf2c853f-e673-498f-9135-2e963890f12e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf2c853f-e673-498f-9135-2e963890f12e-model-cache\") pod \"bf2c853f-e673-498f-9135-2e963890f12e\" (UID: \"bf2c853f-e673-498f-9135-2e963890f12e\") "
Oct 30 07:50:27.828180 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:27.828140    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf2c853f-e673-498f-9135-2e963890f12e-kube-api-access-hcv5x\" (UniqueName: \"kubernetes.io/projected/bf2c853f-e673-498f-9135-2e963890f12e-kube-api-access-hcv5x\") pod \"bf2c853f-e673-498f-9135-2e963890f12e\" (UID: \"bf2c853f-e673-498f-9135-2e963890f12e\") "
Oct 30 07:50:27.828226 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:27.828194    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf2c853f-e673-498f-9135-2e963890f12e-home\" (UniqueName: \"kubernetes.io/empty-dir/bf2c853f-e673-498f-9135-2e963890f12e-home\") pod \"bf2c853f-e673-498f-9135-2e963890f12e\" (UID: \"bf2c853f-e673-498f-9135-2e963890f12e\") "
Oct 30 07:50:27.828280 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:27.828262    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf2c853f-e673-498f-9135-2e963890f12e-model-cache" pod "bf2c853f-e673-498f-9135-2e963890f12e" (UID: "bf2c853f-e673-498f-9135-2e963890f12e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:50:27.828326 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:27.828310    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf2c853f-e673-498f-9135-2e963890f12e-tls-certs\" (UniqueName: \"kubernetes.io/secret/bf2c853f-e673-498f-9135-2e963890f12e-tls-certs\") pod \"bf2c853f-e673-498f-9135-2e963890f12e\" (UID: \"bf2c853f-e673-498f-9135-2e963890f12e\") "
Oct 30 07:50:27.828407 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:27.828395    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf2c853f-e673-498f-9135-2e963890f12e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf2c853f-e673-498f-9135-2e963890f12e-tmp-dir\") pod \"bf2c853f-e673-498f-9135-2e963890f12e\" (UID: \"bf2c853f-e673-498f-9135-2e963890f12e\") "
Oct 30 07:50:27.828503 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:27.828485    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf2c853f-e673-498f-9135-2e963890f12e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf2c853f-e673-498f-9135-2e963890f12e-dshm\") pod \"bf2c853f-e673-498f-9135-2e963890f12e\" (UID: \"bf2c853f-e673-498f-9135-2e963890f12e\") "
Oct 30 07:50:27.828657 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:27.828500    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf2c853f-e673-498f-9135-2e963890f12e-home" pod "bf2c853f-e673-498f-9135-2e963890f12e" (UID: "bf2c853f-e673-498f-9135-2e963890f12e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:50:27.828726 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:27.828688    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf2c853f-e673-498f-9135-2e963890f12e-tmp-dir" pod "bf2c853f-e673-498f-9135-2e963890f12e" (UID: "bf2c853f-e673-498f-9135-2e963890f12e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:50:27.829104 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:27.829081    6224 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf2c853f-e673-498f-9135-2e963890f12e-model-cache\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:50:27.829205 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:27.829108    6224 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf2c853f-e673-498f-9135-2e963890f12e-home\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:50:27.829205 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:27.829130    6224 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf2c853f-e673-498f-9135-2e963890f12e-tmp-dir\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:50:27.830555 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:27.830527    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf2c853f-e673-498f-9135-2e963890f12e-kube-api-access-hcv5x" pod "bf2c853f-e673-498f-9135-2e963890f12e" (UID: "bf2c853f-e673-498f-9135-2e963890f12e"). InnerVolumeSpecName "kube-api-access-hcv5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:50:27.831077 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:27.831049    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf2c853f-e673-498f-9135-2e963890f12e-tls-certs" pod "bf2c853f-e673-498f-9135-2e963890f12e" (UID: "bf2c853f-e673-498f-9135-2e963890f12e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:50:27.831395 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:27.831375    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf2c853f-e673-498f-9135-2e963890f12e-dshm" pod "bf2c853f-e673-498f-9135-2e963890f12e" (UID: "bf2c853f-e673-498f-9135-2e963890f12e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:50:27.930568 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:27.930495    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcv5x\" (UniqueName: \"kubernetes.io/projected/bf2c853f-e673-498f-9135-2e963890f12e-kube-api-access-hcv5x\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:50:27.930568 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:27.930523    6224 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf2c853f-e673-498f-9135-2e963890f12e-tls-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:50:27.930568 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:27.930532    6224 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf2c853f-e673-498f-9135-2e963890f12e-dshm\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:50:28.114060 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:28.114013    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-ad434e-54db8b9d-v5xv5" podUID="ccc61d72-ac4d-4784-9ad1-cea24f30182b" containerName="payload-processing" containerID="cri-o://a3cd90918074d4fc48f0f7a1edd596f36e73433982f2ae7d1e2156ec3fe38976" gracePeriod=25
Oct 30 07:50:28.436342 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:28.436310    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4ab37e/test-model-4ab37e-kserve-75448f6fc5-ln8dd" event={"ID":"bf2c853f-e673-498f-9135-2e963890f12e","Type":"ContainerDied","Data":"e60f3f0b22485a4777a8d4461b7113c0d8a2455d72711dae7aec68fec812ca0d"}
Oct 30 07:50:28.436342 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:28.436326    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4ab37e/test-model-4ab37e-kserve-75448f6fc5-ln8dd"
Oct 30 07:50:28.436880 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:28.436356    6224 scope.go:122] "RemoveContainer" containerID="f01d5ab0da57da3f5e565c274402192bb3e3011b203c6537a3e137d14763a726"
Oct 30 07:50:28.499852 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:28.499825    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4ab37e/test-model-4ab37e-kserve-75448f6fc5-ln8dd"]
Oct 30 07:50:28.513237 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:28.513212    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-4ab37e/test-model-4ab37e-kserve-75448f6fc5-ln8dd"]
Oct 30 07:50:28.605334 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:28.605282    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-4ab37e/test-model-4ab37e-kserve-75448f6fc5-ln8dd" podUID="bf2c853f-e673-498f-9135-2e963890f12e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.191:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:50:29.507591 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:29.507551    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="717ebd45-f957-47c5-a710-d091051fa4eb" path="/var/lib/kubelet/pods/717ebd45-f957-47c5-a710-d091051fa4eb/volumes"
Oct 30 07:50:29.508293 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:29.508269    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf2c853f-e673-498f-9135-2e963890f12e" path="/var/lib/kubelet/pods/bf2c853f-e673-498f-9135-2e963890f12e/volumes"
Oct 30 07:50:31.862162 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:50:31.862095    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:50:32.026236 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:32.026189    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-ad434e-54db8b9d-v5xv5" podUID="ccc61d72-ac4d-4784-9ad1-cea24f30182b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 07:50:32.298008 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:32.297971    6224 patch_prober.go:29] interesting pod/e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 07:50:32.298155 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:32.298027    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688" podUID="ecd9f1bd-d4f6-468f-9994-203a3620a1a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:50:33.480342 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:33.480261    6224 generic.go:356] "Generic (PLEG): container finished" podID="ecd9f1bd-d4f6-468f-9994-203a3620a1a3" containerID="55da16416b464b23e612a907bafbbed04a2c7eb62d44560a1fc60d393233730e" exitCode=0
Oct 30 07:50:33.480856 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:33.480332    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688" event={"ID":"ecd9f1bd-d4f6-468f-9994-203a3620a1a3","Type":"ContainerDied","Data":"55da16416b464b23e612a907bafbbed04a2c7eb62d44560a1fc60d393233730e"}
Oct 30 07:50:33.480856 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:33.480764    6224 scope.go:122] "RemoveContainer" containerID="55da16416b464b23e612a907bafbbed04a2c7eb62d44560a1fc60d393233730e"
Oct 30 07:50:33.482861 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:33.482829    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:50:33.482956 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:33.482902    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:50:33.482956 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:33.482935    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:50:34.498293 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:34.498264    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688" event={"ID":"ecd9f1bd-d4f6-468f-9994-203a3620a1a3","Type":"ContainerStarted","Data":"361ac3190018129223fe4d8c7a12945a7d8ca60cbacfea0cf70ee00679ac1482"}
Oct 30 07:50:34.947782 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:34.947746    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688"
Oct 30 07:50:34.948052 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:34.948031    6224 patch_prober.go:29] interesting pod/e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body=
Oct 30 07:50:34.948103 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:34.948084    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688" podUID="ecd9f1bd-d4f6-468f-9994-203a3620a1a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused"
Oct 30 07:50:35.948177 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:35.948141    6224 patch_prober.go:29] interesting pod/e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body=
Oct 30 07:50:35.948569 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:35.948196    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688" podUID="ecd9f1bd-d4f6-468f-9994-203a3620a1a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused"
Oct 30 07:50:36.948565 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:36.948524    6224 patch_prober.go:29] interesting pod/e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body=
Oct 30 07:50:36.949032 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:36.948575    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688" podUID="ecd9f1bd-d4f6-468f-9994-203a3620a1a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused"
Oct 30 07:50:37.947789 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:37.947753    6224 patch_prober.go:29] interesting pod/e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body=
Oct 30 07:50:37.948006 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:37.947835    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688" podUID="ecd9f1bd-d4f6-468f-9994-203a3620a1a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused"
Oct 30 07:50:38.948403 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:38.948363    6224 patch_prober.go:29] interesting pod/e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body=
Oct 30 07:50:38.948919 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:38.948438    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688" podUID="ecd9f1bd-d4f6-468f-9994-203a3620a1a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused"
Oct 30 07:50:39.953087 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:39.953057    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688"
Oct 30 07:50:39.953582 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:39.953531    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688"
Oct 30 07:50:39.954182 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:39.954165    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688"
Oct 30 07:50:40.108672 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:40.108639    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-ccac6f/test-model-ccac6f-kserve-7c9865d8cf-9sw9m"]
Oct 30 07:50:40.109535 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:40.109510    6224 cpu_manager.go:414] "containerMap: removing container" podUID="6fd17eaf-b363-4ddb-8dc3-29489cb00a87" containerName="main"
Oct 30 07:50:40.109535 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:40.109534    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fd17eaf-b363-4ddb-8dc3-29489cb00a87" containerName="main"
Oct 30 07:50:40.109722 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:40.109591    6224 cpu_manager.go:414] "containerMap: removing container" podUID="9cb25921-0688-433d-90ec-e8ea626437b8" containerName="istio-proxy"
Oct 30 07:50:40.109722 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:40.109602    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cb25921-0688-433d-90ec-e8ea626437b8" containerName="istio-proxy"
Oct 30 07:50:40.109722 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:40.109619    6224 cpu_manager.go:414] "containerMap: removing container" podUID="717ebd45-f957-47c5-a710-d091051fa4eb" containerName="istio-proxy"
Oct 30 07:50:40.109722 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:40.109628    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="717ebd45-f957-47c5-a710-d091051fa4eb" containerName="istio-proxy"
Oct 30 07:50:40.109722 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:40.109661    6224 cpu_manager.go:414] "containerMap: removing container" podUID="7ee43b9a-1501-4a88-842e-564d9f1851f7" containerName="istio-proxy"
Oct 30 07:50:40.109722 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:40.109668    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ee43b9a-1501-4a88-842e-564d9f1851f7" containerName="istio-proxy"
Oct 30 07:50:40.109722 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:40.109689    6224 cpu_manager.go:414] "containerMap: removing container" podUID="bf2c853f-e673-498f-9135-2e963890f12e" containerName="main"
Oct 30 07:50:40.109722 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:40.109697    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf2c853f-e673-498f-9135-2e963890f12e" containerName="main"
Oct 30 07:50:40.110174 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:40.109894    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cb25921-0688-433d-90ec-e8ea626437b8" containerName="istio-proxy"
Oct 30 07:50:40.110174 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:40.109916    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="717ebd45-f957-47c5-a710-d091051fa4eb" containerName="istio-proxy"
Oct 30 07:50:40.110174 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:40.109930    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ee43b9a-1501-4a88-842e-564d9f1851f7" containerName="istio-proxy"
Oct 30 07:50:40.110174 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:40.109941    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf2c853f-e673-498f-9135-2e963890f12e" containerName="main"
Oct 30 07:50:40.110174 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:40.109953    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fd17eaf-b363-4ddb-8dc3-29489cb00a87" containerName="main"
Oct 30 07:50:40.114503 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:40.114479    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ccac6f/test-model-ccac6f-kserve-7c9865d8cf-9sw9m"
Oct 30 07:50:40.133321 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:40.133296    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ccac6f\"/\"default-dockercfg-xqzgx\""
Oct 30 07:50:40.133943 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:40.133917    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ccac6f\"/\"test-model-ccac6f-kserve-self-signed-certs\""
Oct 30 07:50:40.134221 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:40.134200    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ccac6f\"/\"kube-root-ca.crt\""
Oct 30 07:50:40.134363 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:40.134277    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ccac6f\"/\"openshift-service-ca.crt\""
Oct 30 07:50:40.140034 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:40.140010    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ccac6f/test-model-ccac6f-kserve-7c9865d8cf-9sw9m"]
Oct 30 07:50:40.235437 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:40.235340    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c021874-306b-45db-b32a-e3953625ecc8-tls-certs\") pod \"test-model-ccac6f-kserve-7c9865d8cf-9sw9m\" (UID: \"5c021874-306b-45db-b32a-e3953625ecc8\") " pod="ai-tenant-e2e-inf-a-ccac6f/test-model-ccac6f-kserve-7c9865d8cf-9sw9m"
Oct 30 07:50:40.235437 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:40.235411    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c021874-306b-45db-b32a-e3953625ecc8-home\") pod \"test-model-ccac6f-kserve-7c9865d8cf-9sw9m\" (UID: \"5c021874-306b-45db-b32a-e3953625ecc8\") " pod="ai-tenant-e2e-inf-a-ccac6f/test-model-ccac6f-kserve-7c9865d8cf-9sw9m"
Oct 30 07:50:40.235670 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:40.235521    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c021874-306b-45db-b32a-e3953625ecc8-tmp-dir\") pod \"test-model-ccac6f-kserve-7c9865d8cf-9sw9m\" (UID: \"5c021874-306b-45db-b32a-e3953625ecc8\") " pod="ai-tenant-e2e-inf-a-ccac6f/test-model-ccac6f-kserve-7c9865d8cf-9sw9m"
Oct 30 07:50:40.235670 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:40.235584    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c021874-306b-45db-b32a-e3953625ecc8-model-cache\") pod \"test-model-ccac6f-kserve-7c9865d8cf-9sw9m\" (UID: \"5c021874-306b-45db-b32a-e3953625ecc8\") " pod="ai-tenant-e2e-inf-a-ccac6f/test-model-ccac6f-kserve-7c9865d8cf-9sw9m"
Oct 30 07:50:40.235747 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:40.235695    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c021874-306b-45db-b32a-e3953625ecc8-dshm\") pod \"test-model-ccac6f-kserve-7c9865d8cf-9sw9m\" (UID: \"5c021874-306b-45db-b32a-e3953625ecc8\") " pod="ai-tenant-e2e-inf-a-ccac6f/test-model-ccac6f-kserve-7c9865d8cf-9sw9m"
Oct 30 07:50:40.235885 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:40.235866    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld8cj\" (UniqueName: \"kubernetes.io/projected/5c021874-306b-45db-b32a-e3953625ecc8-kube-api-access-ld8cj\") pod \"test-model-ccac6f-kserve-7c9865d8cf-9sw9m\" (UID: \"5c021874-306b-45db-b32a-e3953625ecc8\") " pod="ai-tenant-e2e-inf-a-ccac6f/test-model-ccac6f-kserve-7c9865d8cf-9sw9m"
Oct 30 07:50:40.336993 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:40.336957    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c021874-306b-45db-b32a-e3953625ecc8-tls-certs\") pod \"test-model-ccac6f-kserve-7c9865d8cf-9sw9m\" (UID: \"5c021874-306b-45db-b32a-e3953625ecc8\") " pod="ai-tenant-e2e-inf-a-ccac6f/test-model-ccac6f-kserve-7c9865d8cf-9sw9m"
Oct 30 07:50:40.337211 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:40.337013    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c021874-306b-45db-b32a-e3953625ecc8-home\") pod \"test-model-ccac6f-kserve-7c9865d8cf-9sw9m\" (UID: \"5c021874-306b-45db-b32a-e3953625ecc8\") " pod="ai-tenant-e2e-inf-a-ccac6f/test-model-ccac6f-kserve-7c9865d8cf-9sw9m"
Oct 30 07:50:40.337211 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:40.337047    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c021874-306b-45db-b32a-e3953625ecc8-tmp-dir\") pod \"test-model-ccac6f-kserve-7c9865d8cf-9sw9m\" (UID: \"5c021874-306b-45db-b32a-e3953625ecc8\") " pod="ai-tenant-e2e-inf-a-ccac6f/test-model-ccac6f-kserve-7c9865d8cf-9sw9m"
Oct 30 07:50:40.337211 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:40.337087    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c021874-306b-45db-b32a-e3953625ecc8-model-cache\") pod \"test-model-ccac6f-kserve-7c9865d8cf-9sw9m\" (UID: \"5c021874-306b-45db-b32a-e3953625ecc8\") " pod="ai-tenant-e2e-inf-a-ccac6f/test-model-ccac6f-kserve-7c9865d8cf-9sw9m"
Oct 30 07:50:40.337211 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:40.337164    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c021874-306b-45db-b32a-e3953625ecc8-dshm\") pod \"test-model-ccac6f-kserve-7c9865d8cf-9sw9m\" (UID: \"5c021874-306b-45db-b32a-e3953625ecc8\") " pod="ai-tenant-e2e-inf-a-ccac6f/test-model-ccac6f-kserve-7c9865d8cf-9sw9m"
Oct 30 07:50:40.337453 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:40.337294    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld8cj\" (UniqueName: \"kubernetes.io/projected/5c021874-306b-45db-b32a-e3953625ecc8-kube-api-access-ld8cj\") pod \"test-model-ccac6f-kserve-7c9865d8cf-9sw9m\" (UID: \"5c021874-306b-45db-b32a-e3953625ecc8\") " pod="ai-tenant-e2e-inf-a-ccac6f/test-model-ccac6f-kserve-7c9865d8cf-9sw9m"
Oct 30 07:50:40.337516 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:40.337464    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c021874-306b-45db-b32a-e3953625ecc8-home\") pod \"test-model-ccac6f-kserve-7c9865d8cf-9sw9m\" (UID: \"5c021874-306b-45db-b32a-e3953625ecc8\") " pod="ai-tenant-e2e-inf-a-ccac6f/test-model-ccac6f-kserve-7c9865d8cf-9sw9m"
Oct 30 07:50:40.337576 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:40.337509    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c021874-306b-45db-b32a-e3953625ecc8-tmp-dir\") pod \"test-model-ccac6f-kserve-7c9865d8cf-9sw9m\" (UID: \"5c021874-306b-45db-b32a-e3953625ecc8\") " pod="ai-tenant-e2e-inf-a-ccac6f/test-model-ccac6f-kserve-7c9865d8cf-9sw9m"
Oct 30 07:50:40.337786 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:40.337764    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c021874-306b-45db-b32a-e3953625ecc8-model-cache\") pod \"test-model-ccac6f-kserve-7c9865d8cf-9sw9m\" (UID: \"5c021874-306b-45db-b32a-e3953625ecc8\") " pod="ai-tenant-e2e-inf-a-ccac6f/test-model-ccac6f-kserve-7c9865d8cf-9sw9m"
Oct 30 07:50:40.339954 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:40.339929    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c021874-306b-45db-b32a-e3953625ecc8-dshm\") pod \"test-model-ccac6f-kserve-7c9865d8cf-9sw9m\" (UID: \"5c021874-306b-45db-b32a-e3953625ecc8\") " pod="ai-tenant-e2e-inf-a-ccac6f/test-model-ccac6f-kserve-7c9865d8cf-9sw9m"
Oct 30 07:50:40.340353 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:40.340329    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c021874-306b-45db-b32a-e3953625ecc8-tls-certs\") pod \"test-model-ccac6f-kserve-7c9865d8cf-9sw9m\" (UID: \"5c021874-306b-45db-b32a-e3953625ecc8\") " pod="ai-tenant-e2e-inf-a-ccac6f/test-model-ccac6f-kserve-7c9865d8cf-9sw9m"
Oct 30 07:50:40.376094 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:40.376063    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld8cj\" (UniqueName: \"kubernetes.io/projected/5c021874-306b-45db-b32a-e3953625ecc8-kube-api-access-ld8cj\") pod \"test-model-ccac6f-kserve-7c9865d8cf-9sw9m\" (UID: \"5c021874-306b-45db-b32a-e3953625ecc8\") " pod="ai-tenant-e2e-inf-a-ccac6f/test-model-ccac6f-kserve-7c9865d8cf-9sw9m"
Oct 30 07:50:40.441577 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:40.441146    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ccac6f/test-model-ccac6f-kserve-7c9865d8cf-9sw9m"
Oct 30 07:50:40.828133 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:50:40.828102    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5c021874_306b_45db_b32a_e3953625ecc8.slice/crio-a532ca0c8893af1dba5cff0b3070cfc78f0501318a64655c7b87b629fb64a0fb WatchSource:0}: Error finding container a532ca0c8893af1dba5cff0b3070cfc78f0501318a64655c7b87b629fb64a0fb: Status 404 returned error can't find the container with id a532ca0c8893af1dba5cff0b3070cfc78f0501318a64655c7b87b629fb64a0fb
Oct 30 07:50:40.829054 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:40.828981    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ccac6f/test-model-ccac6f-kserve-7c9865d8cf-9sw9m"]
Oct 30 07:50:41.563956 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:41.563918    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ccac6f/test-model-ccac6f-kserve-7c9865d8cf-9sw9m" event={"ID":"5c021874-306b-45db-b32a-e3953625ecc8","Type":"ContainerStarted","Data":"fcfb792cb95ac074f5681d3b5b240b3afa30f3e06a1242647492f4fd1ac9885e"}
Oct 30 07:50:41.564473 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:41.563965    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ccac6f/test-model-ccac6f-kserve-7c9865d8cf-9sw9m" event={"ID":"5c021874-306b-45db-b32a-e3953625ecc8","Type":"ContainerStarted","Data":"a532ca0c8893af1dba5cff0b3070cfc78f0501318a64655c7b87b629fb64a0fb"}
Oct 30 07:50:41.633595 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:41.633539    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-ccac6f/test-model-ccac6f-kserve-7c9865d8cf-9sw9m" podStartSLOduration=1.6335246410000002 podStartE2EDuration="1.633524641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:50:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:50:41.63066743 +0000 UTC m=+3200.240074401" watchObservedRunningTime="2026-10-30 07:50:41.633524641 +0000 UTC m=+3200.242931610"
Oct 30 07:50:42.026796 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:42.026709    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-ad434e-54db8b9d-v5xv5" podUID="ccc61d72-ac4d-4784-9ad1-cea24f30182b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 07:50:42.442607 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:42.442571    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-ccac6f/test-model-ccac6f-kserve-7c9865d8cf-9sw9m"
Oct 30 07:50:48.403146 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:48.403118    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ad434e-54db8b9d-v5xv5"
Oct 30 07:50:48.558272 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:48.558240    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccc61d72-ac4d-4784-9ad1-cea24f30182b-kube-api-access-l46qp\" (UniqueName: \"kubernetes.io/projected/ccc61d72-ac4d-4784-9ad1-cea24f30182b-kube-api-access-l46qp\") pod \"ccc61d72-ac4d-4784-9ad1-cea24f30182b\" (UID: \"ccc61d72-ac4d-4784-9ad1-cea24f30182b\") "
Oct 30 07:50:48.558496 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:48.558473    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccc61d72-ac4d-4784-9ad1-cea24f30182b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccc61d72-ac4d-4784-9ad1-cea24f30182b-plugins-config-volume\") pod \"ccc61d72-ac4d-4784-9ad1-cea24f30182b\" (UID: \"ccc61d72-ac4d-4784-9ad1-cea24f30182b\") "
Oct 30 07:50:48.558998 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:48.558971    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccc61d72-ac4d-4784-9ad1-cea24f30182b-plugins-config-volume" pod "ccc61d72-ac4d-4784-9ad1-cea24f30182b" (UID: "ccc61d72-ac4d-4784-9ad1-cea24f30182b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:50:48.559263 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:48.559239    6224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccc61d72-ac4d-4784-9ad1-cea24f30182b-plugins-config-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:50:48.560863 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:48.560839    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccc61d72-ac4d-4784-9ad1-cea24f30182b-kube-api-access-l46qp" pod "ccc61d72-ac4d-4784-9ad1-cea24f30182b" (UID: "ccc61d72-ac4d-4784-9ad1-cea24f30182b"). InnerVolumeSpecName "kube-api-access-l46qp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:50:48.617260 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:48.617226    6224 generic.go:356] "Generic (PLEG): container finished" podID="ccc61d72-ac4d-4784-9ad1-cea24f30182b" containerID="a3cd90918074d4fc48f0f7a1edd596f36e73433982f2ae7d1e2156ec3fe38976" exitCode=0
Oct 30 07:50:48.617413 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:48.617281    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ad434e-54db8b9d-v5xv5"
Oct 30 07:50:48.617413 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:48.617332    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ad434e-54db8b9d-v5xv5" event={"ID":"ccc61d72-ac4d-4784-9ad1-cea24f30182b","Type":"ContainerDied","Data":"a3cd90918074d4fc48f0f7a1edd596f36e73433982f2ae7d1e2156ec3fe38976"}
Oct 30 07:50:48.617413 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:48.617392    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ad434e-54db8b9d-v5xv5" event={"ID":"ccc61d72-ac4d-4784-9ad1-cea24f30182b","Type":"ContainerDied","Data":"9fda6ab0ad9c25936c437fec8c9659f5153f2b7354bbf35da1e5d6743abaa664"}
Oct 30 07:50:48.617545 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:48.617421    6224 scope.go:122] "RemoveContainer" containerID="a3cd90918074d4fc48f0f7a1edd596f36e73433982f2ae7d1e2156ec3fe38976"
Oct 30 07:50:48.629704 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:48.629685    6224 scope.go:122] "RemoveContainer" containerID="a3cd90918074d4fc48f0f7a1edd596f36e73433982f2ae7d1e2156ec3fe38976"
Oct 30 07:50:48.629980 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:50:48.629958    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3cd90918074d4fc48f0f7a1edd596f36e73433982f2ae7d1e2156ec3fe38976\": container with ID starting with a3cd90918074d4fc48f0f7a1edd596f36e73433982f2ae7d1e2156ec3fe38976 not found: ID does not exist" containerID="a3cd90918074d4fc48f0f7a1edd596f36e73433982f2ae7d1e2156ec3fe38976"
Oct 30 07:50:48.630057 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:48.629992    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3cd90918074d4fc48f0f7a1edd596f36e73433982f2ae7d1e2156ec3fe38976"} err="failed to get container status \"a3cd90918074d4fc48f0f7a1edd596f36e73433982f2ae7d1e2156ec3fe38976\": rpc error: code = NotFound desc = could not find container \"a3cd90918074d4fc48f0f7a1edd596f36e73433982f2ae7d1e2156ec3fe38976\": container with ID starting with a3cd90918074d4fc48f0f7a1edd596f36e73433982f2ae7d1e2156ec3fe38976 not found: ID does not exist"
Oct 30 07:50:48.660813 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:48.660773    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l46qp\" (UniqueName: \"kubernetes.io/projected/ccc61d72-ac4d-4784-9ad1-cea24f30182b-kube-api-access-l46qp\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:50:48.677692 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:48.677665    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ad434e-54db8b9d-v5xv5"]
Oct 30 07:50:48.682339 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:48.682316    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ad434e-54db8b9d-v5xv5"]
Oct 30 07:50:49.505424 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:49.505391    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccc61d72-ac4d-4784-9ad1-cea24f30182b" path="/var/lib/kubelet/pods/ccc61d72-ac4d-4784-9ad1-cea24f30182b/volumes"
Oct 30 07:50:50.442317 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:50.442281    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-ccac6f/test-model-ccac6f-kserve-7c9865d8cf-9sw9m"
Oct 30 07:50:50.454992 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:50.454961    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-ccac6f/test-model-ccac6f-kserve-7c9865d8cf-9sw9m"
Oct 30 07:50:50.647377 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:50.647345    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-ccac6f/test-model-ccac6f-kserve-7c9865d8cf-9sw9m"
Oct 30 07:50:53.216035 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:53.215997    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:50:57.296372 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:57.296340    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ad434e-6b5b5484cf-g5m97"]
Oct 30 07:50:57.296916 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:57.296582    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad434e-6b5b5484cf-g5m97" podUID="b2db85c9-1904-4eb2-928d-3b8588b8082c" containerName="maas-api" containerID="cri-o://50413d74795df7feab1b2ff2f20e5af60687b5b313819425339a803caa47db32" gracePeriod=30
Oct 30 07:50:57.703692 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:57.703652    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad434e-6b5b5484cf-g5m97"
Oct 30 07:50:57.704042 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:57.704000    6224 generic.go:356] "Generic (PLEG): container finished" podID="b2db85c9-1904-4eb2-928d-3b8588b8082c" containerID="50413d74795df7feab1b2ff2f20e5af60687b5b313819425339a803caa47db32" exitCode=0
Oct 30 07:50:57.704182 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:57.704162    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad434e-6b5b5484cf-g5m97" event={"ID":"b2db85c9-1904-4eb2-928d-3b8588b8082c","Type":"ContainerDied","Data":"50413d74795df7feab1b2ff2f20e5af60687b5b313819425339a803caa47db32"}
Oct 30 07:50:57.704254 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:57.704239    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad434e-6b5b5484cf-g5m97" event={"ID":"b2db85c9-1904-4eb2-928d-3b8588b8082c","Type":"ContainerDied","Data":"851d7b055bf2743c3ae359bf7fca785b04de7d3298fffcb086b4ace38574abe7"}
Oct 30 07:50:57.704320 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:57.704288    6224 scope.go:122] "RemoveContainer" containerID="50413d74795df7feab1b2ff2f20e5af60687b5b313819425339a803caa47db32"
Oct 30 07:50:57.758738 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:57.758709    6224 scope.go:122] "RemoveContainer" containerID="50413d74795df7feab1b2ff2f20e5af60687b5b313819425339a803caa47db32"
Oct 30 07:50:57.759338 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:50:57.759314    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50413d74795df7feab1b2ff2f20e5af60687b5b313819425339a803caa47db32\": container with ID starting with 50413d74795df7feab1b2ff2f20e5af60687b5b313819425339a803caa47db32 not found: ID does not exist" containerID="50413d74795df7feab1b2ff2f20e5af60687b5b313819425339a803caa47db32"
Oct 30 07:50:57.759504 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:57.759475    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50413d74795df7feab1b2ff2f20e5af60687b5b313819425339a803caa47db32"} err="failed to get container status \"50413d74795df7feab1b2ff2f20e5af60687b5b313819425339a803caa47db32\": rpc error: code = NotFound desc = could not find container \"50413d74795df7feab1b2ff2f20e5af60687b5b313819425339a803caa47db32\": container with ID starting with 50413d74795df7feab1b2ff2f20e5af60687b5b313819425339a803caa47db32 not found: ID does not exist"
Oct 30 07:50:57.844497 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:57.844449    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2db85c9-1904-4eb2-928d-3b8588b8082c-kube-api-access-bvv5j\" (UniqueName: \"kubernetes.io/projected/b2db85c9-1904-4eb2-928d-3b8588b8082c-kube-api-access-bvv5j\") pod \"b2db85c9-1904-4eb2-928d-3b8588b8082c\" (UID: \"b2db85c9-1904-4eb2-928d-3b8588b8082c\") "
Oct 30 07:50:57.844767 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:57.844645    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2db85c9-1904-4eb2-928d-3b8588b8082c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2db85c9-1904-4eb2-928d-3b8588b8082c-openshift-service-ca\") pod \"b2db85c9-1904-4eb2-928d-3b8588b8082c\" (UID: \"b2db85c9-1904-4eb2-928d-3b8588b8082c\") "
Oct 30 07:50:57.844767 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:57.844729    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2db85c9-1904-4eb2-928d-3b8588b8082c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2db85c9-1904-4eb2-928d-3b8588b8082c-metrics-certs\") pod \"b2db85c9-1904-4eb2-928d-3b8588b8082c\" (UID: \"b2db85c9-1904-4eb2-928d-3b8588b8082c\") "
Oct 30 07:50:57.844968 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:57.844937    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2db85c9-1904-4eb2-928d-3b8588b8082c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2db85c9-1904-4eb2-928d-3b8588b8082c-maas-api-tls\") pod \"b2db85c9-1904-4eb2-928d-3b8588b8082c\" (UID: \"b2db85c9-1904-4eb2-928d-3b8588b8082c\") "
Oct 30 07:50:57.845054 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:57.845027    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2db85c9-1904-4eb2-928d-3b8588b8082c-openshift-service-ca" pod "b2db85c9-1904-4eb2-928d-3b8588b8082c" (UID: "b2db85c9-1904-4eb2-928d-3b8588b8082c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:50:57.845639 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:57.845611    6224 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2db85c9-1904-4eb2-928d-3b8588b8082c-openshift-service-ca\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:50:57.847329 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:57.847301    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2db85c9-1904-4eb2-928d-3b8588b8082c-metrics-certs" pod "b2db85c9-1904-4eb2-928d-3b8588b8082c" (UID: "b2db85c9-1904-4eb2-928d-3b8588b8082c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:50:57.847510 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:57.847483    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2db85c9-1904-4eb2-928d-3b8588b8082c-kube-api-access-bvv5j" pod "b2db85c9-1904-4eb2-928d-3b8588b8082c" (UID: "b2db85c9-1904-4eb2-928d-3b8588b8082c"). InnerVolumeSpecName "kube-api-access-bvv5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:50:57.847633 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:57.847547    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2db85c9-1904-4eb2-928d-3b8588b8082c-maas-api-tls" pod "b2db85c9-1904-4eb2-928d-3b8588b8082c" (UID: "b2db85c9-1904-4eb2-928d-3b8588b8082c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:50:57.947203 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:57.947109    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvv5j\" (UniqueName: \"kubernetes.io/projected/b2db85c9-1904-4eb2-928d-3b8588b8082c-kube-api-access-bvv5j\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:50:57.947203 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:57.947144    6224 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2db85c9-1904-4eb2-928d-3b8588b8082c-metrics-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:50:57.947203 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:57.947153    6224 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2db85c9-1904-4eb2-928d-3b8588b8082c-maas-api-tls\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:50:58.713387 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:58.713350    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad434e-6b5b5484cf-g5m97"
Oct 30 07:50:58.771773 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:58.771738    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ad434e-6b5b5484cf-g5m97"]
Oct 30 07:50:58.785901 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:58.785868    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ad434e-6b5b5484cf-g5m97"]
Oct 30 07:50:59.505490 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:50:59.505454    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2db85c9-1904-4eb2-928d-3b8588b8082c" path="/var/lib/kubelet/pods/b2db85c9-1904-4eb2-928d-3b8588b8082c/volumes"
Oct 30 07:51:01.859874 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:51:01.859833    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:51:08.059297 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:08.059260    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-896ca4/test-model-896ca4-kserve-654ff764b9-lgqsk"]
Oct 30 07:51:08.060237 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:08.060212    6224 cpu_manager.go:414] "containerMap: removing container" podUID="b2db85c9-1904-4eb2-928d-3b8588b8082c" containerName="maas-api"
Oct 30 07:51:08.060360 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:08.060237    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2db85c9-1904-4eb2-928d-3b8588b8082c" containerName="maas-api"
Oct 30 07:51:08.060360 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:08.060288    6224 cpu_manager.go:414] "containerMap: removing container" podUID="ccc61d72-ac4d-4784-9ad1-cea24f30182b" containerName="payload-processing"
Oct 30 07:51:08.060360 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:08.060296    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccc61d72-ac4d-4784-9ad1-cea24f30182b" containerName="payload-processing"
Oct 30 07:51:08.060518 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:08.060499    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccc61d72-ac4d-4784-9ad1-cea24f30182b" containerName="payload-processing"
Oct 30 07:51:08.060577 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:08.060525    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2db85c9-1904-4eb2-928d-3b8588b8082c" containerName="maas-api"
Oct 30 07:51:08.065755 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:08.065730    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-896ca4/test-model-896ca4-kserve-654ff764b9-lgqsk"
Oct 30 07:51:08.083441 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:08.083411    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-896ca4/test-model-896ca4-kserve-654ff764b9-lgqsk"]
Oct 30 07:51:08.091089 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:08.091067    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-896ca4\"/\"openshift-service-ca.crt\""
Oct 30 07:51:08.091271 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:08.091249    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-896ca4\"/\"kube-root-ca.crt\""
Oct 30 07:51:08.096368 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:08.096335    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-896ca4\"/\"default-dockercfg-98pqb\""
Oct 30 07:51:08.096491 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:08.096423    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-896ca4\"/\"test-model-896ca4-kserve-self-signed-certs\""
Oct 30 07:51:08.200482 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:08.200442    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7b8da2a-15d6-4beb-ab11-ad80a7f162e8-dshm\") pod \"test-model-896ca4-kserve-654ff764b9-lgqsk\" (UID: \"c7b8da2a-15d6-4beb-ab11-ad80a7f162e8\") " pod="ai-tenant-e2e-inf-b-896ca4/test-model-896ca4-kserve-654ff764b9-lgqsk"
Oct 30 07:51:08.200656 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:08.200505    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7b8da2a-15d6-4beb-ab11-ad80a7f162e8-tmp-dir\") pod \"test-model-896ca4-kserve-654ff764b9-lgqsk\" (UID: \"c7b8da2a-15d6-4beb-ab11-ad80a7f162e8\") " pod="ai-tenant-e2e-inf-b-896ca4/test-model-896ca4-kserve-654ff764b9-lgqsk"
Oct 30 07:51:08.200757 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:08.200730    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7b8da2a-15d6-4beb-ab11-ad80a7f162e8-tls-certs\") pod \"test-model-896ca4-kserve-654ff764b9-lgqsk\" (UID: \"c7b8da2a-15d6-4beb-ab11-ad80a7f162e8\") " pod="ai-tenant-e2e-inf-b-896ca4/test-model-896ca4-kserve-654ff764b9-lgqsk"
Oct 30 07:51:08.200888 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:08.200865    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7b8da2a-15d6-4beb-ab11-ad80a7f162e8-home\") pod \"test-model-896ca4-kserve-654ff764b9-lgqsk\" (UID: \"c7b8da2a-15d6-4beb-ab11-ad80a7f162e8\") " pod="ai-tenant-e2e-inf-b-896ca4/test-model-896ca4-kserve-654ff764b9-lgqsk"
Oct 30 07:51:08.200940 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:08.200904    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvgqp\" (UniqueName: \"kubernetes.io/projected/c7b8da2a-15d6-4beb-ab11-ad80a7f162e8-kube-api-access-pvgqp\") pod \"test-model-896ca4-kserve-654ff764b9-lgqsk\" (UID: \"c7b8da2a-15d6-4beb-ab11-ad80a7f162e8\") " pod="ai-tenant-e2e-inf-b-896ca4/test-model-896ca4-kserve-654ff764b9-lgqsk"
Oct 30 07:51:08.201034 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:08.201014    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7b8da2a-15d6-4beb-ab11-ad80a7f162e8-model-cache\") pod \"test-model-896ca4-kserve-654ff764b9-lgqsk\" (UID: \"c7b8da2a-15d6-4beb-ab11-ad80a7f162e8\") " pod="ai-tenant-e2e-inf-b-896ca4/test-model-896ca4-kserve-654ff764b9-lgqsk"
Oct 30 07:51:08.302976 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:08.302932    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7b8da2a-15d6-4beb-ab11-ad80a7f162e8-model-cache\") pod \"test-model-896ca4-kserve-654ff764b9-lgqsk\" (UID: \"c7b8da2a-15d6-4beb-ab11-ad80a7f162e8\") " pod="ai-tenant-e2e-inf-b-896ca4/test-model-896ca4-kserve-654ff764b9-lgqsk"
Oct 30 07:51:08.303177 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:08.303051    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7b8da2a-15d6-4beb-ab11-ad80a7f162e8-dshm\") pod \"test-model-896ca4-kserve-654ff764b9-lgqsk\" (UID: \"c7b8da2a-15d6-4beb-ab11-ad80a7f162e8\") " pod="ai-tenant-e2e-inf-b-896ca4/test-model-896ca4-kserve-654ff764b9-lgqsk"
Oct 30 07:51:08.303177 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:08.303117    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7b8da2a-15d6-4beb-ab11-ad80a7f162e8-tmp-dir\") pod \"test-model-896ca4-kserve-654ff764b9-lgqsk\" (UID: \"c7b8da2a-15d6-4beb-ab11-ad80a7f162e8\") " pod="ai-tenant-e2e-inf-b-896ca4/test-model-896ca4-kserve-654ff764b9-lgqsk"
Oct 30 07:51:08.303303 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:08.303270    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7b8da2a-15d6-4beb-ab11-ad80a7f162e8-tls-certs\") pod \"test-model-896ca4-kserve-654ff764b9-lgqsk\" (UID: \"c7b8da2a-15d6-4beb-ab11-ad80a7f162e8\") " pod="ai-tenant-e2e-inf-b-896ca4/test-model-896ca4-kserve-654ff764b9-lgqsk"
Oct 30 07:51:08.303359 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:08.303338    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7b8da2a-15d6-4beb-ab11-ad80a7f162e8-home\") pod \"test-model-896ca4-kserve-654ff764b9-lgqsk\" (UID: \"c7b8da2a-15d6-4beb-ab11-ad80a7f162e8\") " pod="ai-tenant-e2e-inf-b-896ca4/test-model-896ca4-kserve-654ff764b9-lgqsk"
Oct 30 07:51:08.303421 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:08.303364    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvgqp\" (UniqueName: \"kubernetes.io/projected/c7b8da2a-15d6-4beb-ab11-ad80a7f162e8-kube-api-access-pvgqp\") pod \"test-model-896ca4-kserve-654ff764b9-lgqsk\" (UID: \"c7b8da2a-15d6-4beb-ab11-ad80a7f162e8\") " pod="ai-tenant-e2e-inf-b-896ca4/test-model-896ca4-kserve-654ff764b9-lgqsk"
Oct 30 07:51:08.303481 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:08.303426    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7b8da2a-15d6-4beb-ab11-ad80a7f162e8-model-cache\") pod \"test-model-896ca4-kserve-654ff764b9-lgqsk\" (UID: \"c7b8da2a-15d6-4beb-ab11-ad80a7f162e8\") " pod="ai-tenant-e2e-inf-b-896ca4/test-model-896ca4-kserve-654ff764b9-lgqsk"
Oct 30 07:51:08.303765 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:08.303730    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7b8da2a-15d6-4beb-ab11-ad80a7f162e8-tmp-dir\") pod \"test-model-896ca4-kserve-654ff764b9-lgqsk\" (UID: \"c7b8da2a-15d6-4beb-ab11-ad80a7f162e8\") " pod="ai-tenant-e2e-inf-b-896ca4/test-model-896ca4-kserve-654ff764b9-lgqsk"
Oct 30 07:51:08.304017 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:08.303961    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7b8da2a-15d6-4beb-ab11-ad80a7f162e8-home\") pod \"test-model-896ca4-kserve-654ff764b9-lgqsk\" (UID: \"c7b8da2a-15d6-4beb-ab11-ad80a7f162e8\") " pod="ai-tenant-e2e-inf-b-896ca4/test-model-896ca4-kserve-654ff764b9-lgqsk"
Oct 30 07:51:08.306324 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:08.306296    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7b8da2a-15d6-4beb-ab11-ad80a7f162e8-dshm\") pod \"test-model-896ca4-kserve-654ff764b9-lgqsk\" (UID: \"c7b8da2a-15d6-4beb-ab11-ad80a7f162e8\") " pod="ai-tenant-e2e-inf-b-896ca4/test-model-896ca4-kserve-654ff764b9-lgqsk"
Oct 30 07:51:08.307162 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:08.307132    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7b8da2a-15d6-4beb-ab11-ad80a7f162e8-tls-certs\") pod \"test-model-896ca4-kserve-654ff764b9-lgqsk\" (UID: \"c7b8da2a-15d6-4beb-ab11-ad80a7f162e8\") " pod="ai-tenant-e2e-inf-b-896ca4/test-model-896ca4-kserve-654ff764b9-lgqsk"
Oct 30 07:51:08.332293 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:08.332208    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvgqp\" (UniqueName: \"kubernetes.io/projected/c7b8da2a-15d6-4beb-ab11-ad80a7f162e8-kube-api-access-pvgqp\") pod \"test-model-896ca4-kserve-654ff764b9-lgqsk\" (UID: \"c7b8da2a-15d6-4beb-ab11-ad80a7f162e8\") " pod="ai-tenant-e2e-inf-b-896ca4/test-model-896ca4-kserve-654ff764b9-lgqsk"
Oct 30 07:51:08.385129 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:08.385080    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-896ca4/test-model-896ca4-kserve-654ff764b9-lgqsk"
Oct 30 07:51:08.726951 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:51:08.726921    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc7b8da2a_15d6_4beb_ab11_ad80a7f162e8.slice/crio-ab34ac8b8e3f6e597aa538f6732eff6f430e75eb1f799358cb32d31df8d31fb7 WatchSource:0}: Error finding container ab34ac8b8e3f6e597aa538f6732eff6f430e75eb1f799358cb32d31df8d31fb7: Status 404 returned error can't find the container with id ab34ac8b8e3f6e597aa538f6732eff6f430e75eb1f799358cb32d31df8d31fb7
Oct 30 07:51:08.728024 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:08.727999    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-896ca4/test-model-896ca4-kserve-654ff764b9-lgqsk"]
Oct 30 07:51:08.839756 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:08.839711    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-896ca4/test-model-896ca4-kserve-654ff764b9-lgqsk" event={"ID":"c7b8da2a-15d6-4beb-ab11-ad80a7f162e8","Type":"ContainerStarted","Data":"0f6aebc79ccc2eb2a60246c665823b4b209d4c6a56e9fe07c1af4dabfc6f1b77"}
Oct 30 07:51:08.839962 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:08.839766    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-896ca4/test-model-896ca4-kserve-654ff764b9-lgqsk" event={"ID":"c7b8da2a-15d6-4beb-ab11-ad80a7f162e8","Type":"ContainerStarted","Data":"ab34ac8b8e3f6e597aa538f6732eff6f430e75eb1f799358cb32d31df8d31fb7"}
Oct 30 07:51:08.886286 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:08.886239    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-896ca4/test-model-896ca4-kserve-654ff764b9-lgqsk" podStartSLOduration=0.886225053 podStartE2EDuration="886.225053ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:51:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:51:08.884608983 +0000 UTC m=+3227.494015955" watchObservedRunningTime="2026-10-30 07:51:08.886225053 +0000 UTC m=+3227.495632023"
Oct 30 07:51:09.386306 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:09.386268    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-896ca4/test-model-896ca4-kserve-654ff764b9-lgqsk"
Oct 30 07:51:16.829207 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:16.829176    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4ab37e-7cdb99fdd6-mlnbs"]
Oct 30 07:51:18.386560 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:18.386526    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-896ca4/test-model-896ca4-kserve-654ff764b9-lgqsk"
Oct 30 07:51:18.399672 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:18.399648    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-896ca4/test-model-896ca4-kserve-654ff764b9-lgqsk"
Oct 30 07:51:18.942033 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:18.942004    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-896ca4/test-model-896ca4-kserve-654ff764b9-lgqsk"
Oct 30 07:51:21.829524 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:21.829458    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-4ab37e-7cdb99fdd6-mlnbs" podUID="8a2822e1-a884-41cb-a5c2-34972b1d4de4" containerName="payload-processing" containerID="cri-o://07185a558bf3277002cf38b81c4f9d24dd252c37c718a9d1120049d9a505e56c" gracePeriod=25
Oct 30 07:51:24.030035 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:24.030000    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-4ab37e-7cdb99fdd6-mlnbs" podUID="8a2822e1-a884-41cb-a5c2-34972b1d4de4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 07:51:31.866461 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:51:31.866393    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:51:33.276506 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:33.276420    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-0e9723/test-model-0e9723-kserve-67974fd467-jpgfq"]
Oct 30 07:51:33.282744 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:33.282717    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0e9723/test-model-0e9723-kserve-67974fd467-jpgfq"
Oct 30 07:51:33.299675 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:33.299648    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-0e9723/test-model-0e9723-kserve-67974fd467-jpgfq"]
Oct 30 07:51:33.309000 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:33.308975    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-0e9723\"/\"kube-root-ca.crt\""
Oct 30 07:51:33.309153 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:33.308975    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-0e9723\"/\"openshift-service-ca.crt\""
Oct 30 07:51:33.309153 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:33.309060    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-0e9723\"/\"default-dockercfg-p9mw4\""
Oct 30 07:51:33.313927 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:33.313904    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-0e9723\"/\"test-model-0e9723-kserve-self-signed-certs\""
Oct 30 07:51:33.463212 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:33.463172    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd55h\" (UniqueName: \"kubernetes.io/projected/57d482ac-c4e1-4a09-8d52-ed12f04c0b3b-kube-api-access-dd55h\") pod \"test-model-0e9723-kserve-67974fd467-jpgfq\" (UID: \"57d482ac-c4e1-4a09-8d52-ed12f04c0b3b\") " pod="ai-tenant-e2e-inf-a-0e9723/test-model-0e9723-kserve-67974fd467-jpgfq"
Oct 30 07:51:33.463428 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:33.463240    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57d482ac-c4e1-4a09-8d52-ed12f04c0b3b-tls-certs\") pod \"test-model-0e9723-kserve-67974fd467-jpgfq\" (UID: \"57d482ac-c4e1-4a09-8d52-ed12f04c0b3b\") " pod="ai-tenant-e2e-inf-a-0e9723/test-model-0e9723-kserve-67974fd467-jpgfq"
Oct 30 07:51:33.463428 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:33.463337    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57d482ac-c4e1-4a09-8d52-ed12f04c0b3b-home\") pod \"test-model-0e9723-kserve-67974fd467-jpgfq\" (UID: \"57d482ac-c4e1-4a09-8d52-ed12f04c0b3b\") " pod="ai-tenant-e2e-inf-a-0e9723/test-model-0e9723-kserve-67974fd467-jpgfq"
Oct 30 07:51:33.463593 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:33.463561    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57d482ac-c4e1-4a09-8d52-ed12f04c0b3b-tmp-dir\") pod \"test-model-0e9723-kserve-67974fd467-jpgfq\" (UID: \"57d482ac-c4e1-4a09-8d52-ed12f04c0b3b\") " pod="ai-tenant-e2e-inf-a-0e9723/test-model-0e9723-kserve-67974fd467-jpgfq"
Oct 30 07:51:33.463657 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:33.463624    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57d482ac-c4e1-4a09-8d52-ed12f04c0b3b-dshm\") pod \"test-model-0e9723-kserve-67974fd467-jpgfq\" (UID: \"57d482ac-c4e1-4a09-8d52-ed12f04c0b3b\") " pod="ai-tenant-e2e-inf-a-0e9723/test-model-0e9723-kserve-67974fd467-jpgfq"
Oct 30 07:51:33.463716 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:33.463663    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57d482ac-c4e1-4a09-8d52-ed12f04c0b3b-model-cache\") pod \"test-model-0e9723-kserve-67974fd467-jpgfq\" (UID: \"57d482ac-c4e1-4a09-8d52-ed12f04c0b3b\") " pod="ai-tenant-e2e-inf-a-0e9723/test-model-0e9723-kserve-67974fd467-jpgfq"
Oct 30 07:51:33.569951 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:33.569894    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57d482ac-c4e1-4a09-8d52-ed12f04c0b3b-model-cache\") pod \"test-model-0e9723-kserve-67974fd467-jpgfq\" (UID: \"57d482ac-c4e1-4a09-8d52-ed12f04c0b3b\") " pod="ai-tenant-e2e-inf-a-0e9723/test-model-0e9723-kserve-67974fd467-jpgfq"
Oct 30 07:51:33.570902 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:33.570874    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd55h\" (UniqueName: \"kubernetes.io/projected/57d482ac-c4e1-4a09-8d52-ed12f04c0b3b-kube-api-access-dd55h\") pod \"test-model-0e9723-kserve-67974fd467-jpgfq\" (UID: \"57d482ac-c4e1-4a09-8d52-ed12f04c0b3b\") " pod="ai-tenant-e2e-inf-a-0e9723/test-model-0e9723-kserve-67974fd467-jpgfq"
Oct 30 07:51:33.577061 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:33.577025    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57d482ac-c4e1-4a09-8d52-ed12f04c0b3b-tls-certs\") pod \"test-model-0e9723-kserve-67974fd467-jpgfq\" (UID: \"57d482ac-c4e1-4a09-8d52-ed12f04c0b3b\") " pod="ai-tenant-e2e-inf-a-0e9723/test-model-0e9723-kserve-67974fd467-jpgfq"
Oct 30 07:51:33.577761 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:33.570650    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57d482ac-c4e1-4a09-8d52-ed12f04c0b3b-model-cache\") pod \"test-model-0e9723-kserve-67974fd467-jpgfq\" (UID: \"57d482ac-c4e1-4a09-8d52-ed12f04c0b3b\") " pod="ai-tenant-e2e-inf-a-0e9723/test-model-0e9723-kserve-67974fd467-jpgfq"
Oct 30 07:51:33.578070 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:33.578049    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57d482ac-c4e1-4a09-8d52-ed12f04c0b3b-home\") pod \"test-model-0e9723-kserve-67974fd467-jpgfq\" (UID: \"57d482ac-c4e1-4a09-8d52-ed12f04c0b3b\") " pod="ai-tenant-e2e-inf-a-0e9723/test-model-0e9723-kserve-67974fd467-jpgfq"
Oct 30 07:51:33.578690 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:33.578671    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57d482ac-c4e1-4a09-8d52-ed12f04c0b3b-tmp-dir\") pod \"test-model-0e9723-kserve-67974fd467-jpgfq\" (UID: \"57d482ac-c4e1-4a09-8d52-ed12f04c0b3b\") " pod="ai-tenant-e2e-inf-a-0e9723/test-model-0e9723-kserve-67974fd467-jpgfq"
Oct 30 07:51:33.579318 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:33.579297    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57d482ac-c4e1-4a09-8d52-ed12f04c0b3b-dshm\") pod \"test-model-0e9723-kserve-67974fd467-jpgfq\" (UID: \"57d482ac-c4e1-4a09-8d52-ed12f04c0b3b\") " pod="ai-tenant-e2e-inf-a-0e9723/test-model-0e9723-kserve-67974fd467-jpgfq"
Oct 30 07:51:33.579977 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:33.579191    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57d482ac-c4e1-4a09-8d52-ed12f04c0b3b-tmp-dir\") pod \"test-model-0e9723-kserve-67974fd467-jpgfq\" (UID: \"57d482ac-c4e1-4a09-8d52-ed12f04c0b3b\") " pod="ai-tenant-e2e-inf-a-0e9723/test-model-0e9723-kserve-67974fd467-jpgfq"
Oct 30 07:51:33.580120 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:33.578473    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57d482ac-c4e1-4a09-8d52-ed12f04c0b3b-home\") pod \"test-model-0e9723-kserve-67974fd467-jpgfq\" (UID: \"57d482ac-c4e1-4a09-8d52-ed12f04c0b3b\") " pod="ai-tenant-e2e-inf-a-0e9723/test-model-0e9723-kserve-67974fd467-jpgfq"
Oct 30 07:51:33.584411 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:33.584375    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57d482ac-c4e1-4a09-8d52-ed12f04c0b3b-tls-certs\") pod \"test-model-0e9723-kserve-67974fd467-jpgfq\" (UID: \"57d482ac-c4e1-4a09-8d52-ed12f04c0b3b\") " pod="ai-tenant-e2e-inf-a-0e9723/test-model-0e9723-kserve-67974fd467-jpgfq"
Oct 30 07:51:33.587391 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:33.587343    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57d482ac-c4e1-4a09-8d52-ed12f04c0b3b-dshm\") pod \"test-model-0e9723-kserve-67974fd467-jpgfq\" (UID: \"57d482ac-c4e1-4a09-8d52-ed12f04c0b3b\") " pod="ai-tenant-e2e-inf-a-0e9723/test-model-0e9723-kserve-67974fd467-jpgfq"
Oct 30 07:51:33.597345 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:33.597316    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd55h\" (UniqueName: \"kubernetes.io/projected/57d482ac-c4e1-4a09-8d52-ed12f04c0b3b-kube-api-access-dd55h\") pod \"test-model-0e9723-kserve-67974fd467-jpgfq\" (UID: \"57d482ac-c4e1-4a09-8d52-ed12f04c0b3b\") " pod="ai-tenant-e2e-inf-a-0e9723/test-model-0e9723-kserve-67974fd467-jpgfq"
Oct 30 07:51:33.897540 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:33.897454    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0e9723/test-model-0e9723-kserve-67974fd467-jpgfq"
Oct 30 07:51:34.030687 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:34.030643    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-4ab37e-7cdb99fdd6-mlnbs" podUID="8a2822e1-a884-41cb-a5c2-34972b1d4de4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 07:51:34.202388 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:51:34.202359    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod57d482ac_c4e1_4a09_8d52_ed12f04c0b3b.slice/crio-021420b63bf78ae4726a2ad5b6f2908b7f3245c468fece9682301862daa77b36 WatchSource:0}: Error finding container 021420b63bf78ae4726a2ad5b6f2908b7f3245c468fece9682301862daa77b36: Status 404 returned error can't find the container with id 021420b63bf78ae4726a2ad5b6f2908b7f3245c468fece9682301862daa77b36
Oct 30 07:51:34.205158 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:34.205128    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-0e9723/test-model-0e9723-kserve-67974fd467-jpgfq"]
Oct 30 07:51:35.064121 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:35.064079    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0e9723/test-model-0e9723-kserve-67974fd467-jpgfq" event={"ID":"57d482ac-c4e1-4a09-8d52-ed12f04c0b3b","Type":"ContainerStarted","Data":"4f17fdd7e7eca1bc8d2f40cfc205be4b80d8e2376442cc3be7ae2225ab8e4b3f"}
Oct 30 07:51:35.064626 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:35.064135    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0e9723/test-model-0e9723-kserve-67974fd467-jpgfq" event={"ID":"57d482ac-c4e1-4a09-8d52-ed12f04c0b3b","Type":"ContainerStarted","Data":"021420b63bf78ae4726a2ad5b6f2908b7f3245c468fece9682301862daa77b36"}
Oct 30 07:51:35.112533 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:35.112481    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-0e9723/test-model-0e9723-kserve-67974fd467-jpgfq" podStartSLOduration=2.112467298 podStartE2EDuration="2.112467298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:51:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:51:35.108489775 +0000 UTC m=+3253.717896767" watchObservedRunningTime="2026-10-30 07:51:35.112467298 +0000 UTC m=+3253.721874268"
Oct 30 07:51:35.898235 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:35.898193    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-0e9723/test-model-0e9723-kserve-67974fd467-jpgfq"
Oct 30 07:51:42.125265 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:42.125236    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4ab37e-7cdb99fdd6-mlnbs"
Oct 30 07:51:42.150043 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:42.150016    6224 generic.go:356] "Generic (PLEG): container finished" podID="8a2822e1-a884-41cb-a5c2-34972b1d4de4" containerID="07185a558bf3277002cf38b81c4f9d24dd252c37c718a9d1120049d9a505e56c" exitCode=0
Oct 30 07:51:42.150183 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:42.150054    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4ab37e-7cdb99fdd6-mlnbs"
Oct 30 07:51:42.150183 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:42.150104    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4ab37e-7cdb99fdd6-mlnbs" event={"ID":"8a2822e1-a884-41cb-a5c2-34972b1d4de4","Type":"ContainerDied","Data":"07185a558bf3277002cf38b81c4f9d24dd252c37c718a9d1120049d9a505e56c"}
Oct 30 07:51:42.150183 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:42.150165    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4ab37e-7cdb99fdd6-mlnbs" event={"ID":"8a2822e1-a884-41cb-a5c2-34972b1d4de4","Type":"ContainerDied","Data":"530ff575fbdb1524a8a335a85fffccb3e9843741e40a634c9aefbc2c6afc393b"}
Oct 30 07:51:42.150183 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:42.150176    6224 scope.go:122] "RemoveContainer" containerID="07185a558bf3277002cf38b81c4f9d24dd252c37c718a9d1120049d9a505e56c"
Oct 30 07:51:42.160662 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:42.160646    6224 scope.go:122] "RemoveContainer" containerID="07185a558bf3277002cf38b81c4f9d24dd252c37c718a9d1120049d9a505e56c"
Oct 30 07:51:42.160964 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:51:42.160945    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07185a558bf3277002cf38b81c4f9d24dd252c37c718a9d1120049d9a505e56c\": container with ID starting with 07185a558bf3277002cf38b81c4f9d24dd252c37c718a9d1120049d9a505e56c not found: ID does not exist" containerID="07185a558bf3277002cf38b81c4f9d24dd252c37c718a9d1120049d9a505e56c"
Oct 30 07:51:42.161041 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:42.160976    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07185a558bf3277002cf38b81c4f9d24dd252c37c718a9d1120049d9a505e56c"} err="failed to get container status \"07185a558bf3277002cf38b81c4f9d24dd252c37c718a9d1120049d9a505e56c\": rpc error: code = NotFound desc = could not find container \"07185a558bf3277002cf38b81c4f9d24dd252c37c718a9d1120049d9a505e56c\": container with ID starting with 07185a558bf3277002cf38b81c4f9d24dd252c37c718a9d1120049d9a505e56c not found: ID does not exist"
Oct 30 07:51:42.199996 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:42.199954    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a2822e1-a884-41cb-a5c2-34972b1d4de4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a2822e1-a884-41cb-a5c2-34972b1d4de4-plugins-config-volume\") pod \"8a2822e1-a884-41cb-a5c2-34972b1d4de4\" (UID: \"8a2822e1-a884-41cb-a5c2-34972b1d4de4\") "
Oct 30 07:51:42.200691 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:42.200594    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a2822e1-a884-41cb-a5c2-34972b1d4de4-plugins-config-volume" pod "8a2822e1-a884-41cb-a5c2-34972b1d4de4" (UID: "8a2822e1-a884-41cb-a5c2-34972b1d4de4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:51:42.202256 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:42.202233    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a2822e1-a884-41cb-a5c2-34972b1d4de4-kube-api-access-pndnp\" (UniqueName: \"kubernetes.io/projected/8a2822e1-a884-41cb-a5c2-34972b1d4de4-kube-api-access-pndnp\") pod \"8a2822e1-a884-41cb-a5c2-34972b1d4de4\" (UID: \"8a2822e1-a884-41cb-a5c2-34972b1d4de4\") "
Oct 30 07:51:42.203171 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:42.203137    6224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a2822e1-a884-41cb-a5c2-34972b1d4de4-plugins-config-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:51:42.204633 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:42.204605    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a2822e1-a884-41cb-a5c2-34972b1d4de4-kube-api-access-pndnp" pod "8a2822e1-a884-41cb-a5c2-34972b1d4de4" (UID: "8a2822e1-a884-41cb-a5c2-34972b1d4de4"). InnerVolumeSpecName "kube-api-access-pndnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:51:42.304958 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:42.304929    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pndnp\" (UniqueName: \"kubernetes.io/projected/8a2822e1-a884-41cb-a5c2-34972b1d4de4-kube-api-access-pndnp\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:51:42.502085 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:42.502051    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4ab37e-7cdb99fdd6-mlnbs"]
Oct 30 07:51:42.508272 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:42.508240    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4ab37e-7cdb99fdd6-mlnbs"]
Oct 30 07:51:43.338038 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:43.338004    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-8fef73/test-model-8fef73-kserve-54ff956b49-429dm"]
Oct 30 07:51:43.537156 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:43.537121    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a2822e1-a884-41cb-a5c2-34972b1d4de4" path="/var/lib/kubelet/pods/8a2822e1-a884-41cb-a5c2-34972b1d4de4/volumes"
Oct 30 07:51:43.898070 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:43.897985    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-0e9723/test-model-0e9723-kserve-67974fd467-jpgfq"
Oct 30 07:51:43.912551 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:43.912523    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-0e9723/test-model-0e9723-kserve-67974fd467-jpgfq"
Oct 30 07:51:44.182855 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:44.182754    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-0e9723/test-model-0e9723-kserve-67974fd467-jpgfq"
Oct 30 07:51:58.366107 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:58.366068    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-8fef73/test-model-8fef73-kserve-54ff956b49-429dm" podUID="27700938-db52-4225-90cd-244060765bbc" containerName="main" containerID="cri-o://437cec0b042f64cfaf3cca470402569407e84494291aa5cfc4913d42f1a1cdc2" gracePeriod=45
Oct 30 07:51:58.680793 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:58.680759    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8fef73/test-model-8fef73-kserve-54ff956b49-429dm"
Oct 30 07:51:58.727665 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:58.727623    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27700938-db52-4225-90cd-244060765bbc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/27700938-db52-4225-90cd-244060765bbc-dshm\") pod \"27700938-db52-4225-90cd-244060765bbc\" (UID: \"27700938-db52-4225-90cd-244060765bbc\") "
Oct 30 07:51:58.727867 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:58.727695    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27700938-db52-4225-90cd-244060765bbc-kube-api-access-wwk7p\" (UniqueName: \"kubernetes.io/projected/27700938-db52-4225-90cd-244060765bbc-kube-api-access-wwk7p\") pod \"27700938-db52-4225-90cd-244060765bbc\" (UID: \"27700938-db52-4225-90cd-244060765bbc\") "
Oct 30 07:51:58.727867 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:58.727764    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27700938-db52-4225-90cd-244060765bbc-tls-certs\" (UniqueName: \"kubernetes.io/secret/27700938-db52-4225-90cd-244060765bbc-tls-certs\") pod \"27700938-db52-4225-90cd-244060765bbc\" (UID: \"27700938-db52-4225-90cd-244060765bbc\") "
Oct 30 07:51:58.728016 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:58.727941    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27700938-db52-4225-90cd-244060765bbc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27700938-db52-4225-90cd-244060765bbc-model-cache\") pod \"27700938-db52-4225-90cd-244060765bbc\" (UID: \"27700938-db52-4225-90cd-244060765bbc\") "
Oct 30 07:51:58.728110 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:58.728087    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27700938-db52-4225-90cd-244060765bbc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27700938-db52-4225-90cd-244060765bbc-tmp-dir\") pod \"27700938-db52-4225-90cd-244060765bbc\" (UID: \"27700938-db52-4225-90cd-244060765bbc\") "
Oct 30 07:51:58.728272 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:58.728181    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27700938-db52-4225-90cd-244060765bbc-home\" (UniqueName: \"kubernetes.io/empty-dir/27700938-db52-4225-90cd-244060765bbc-home\") pod \"27700938-db52-4225-90cd-244060765bbc\" (UID: \"27700938-db52-4225-90cd-244060765bbc\") "
Oct 30 07:51:58.728272 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:58.728210    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27700938-db52-4225-90cd-244060765bbc-model-cache" pod "27700938-db52-4225-90cd-244060765bbc" (UID: "27700938-db52-4225-90cd-244060765bbc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:51:58.728453 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:58.728428    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27700938-db52-4225-90cd-244060765bbc-tmp-dir" pod "27700938-db52-4225-90cd-244060765bbc" (UID: "27700938-db52-4225-90cd-244060765bbc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:51:58.728583 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:58.728564    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27700938-db52-4225-90cd-244060765bbc-home" pod "27700938-db52-4225-90cd-244060765bbc" (UID: "27700938-db52-4225-90cd-244060765bbc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:51:58.728842 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:58.728801    6224 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27700938-db52-4225-90cd-244060765bbc-model-cache\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:51:58.728842 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:58.728834    6224 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27700938-db52-4225-90cd-244060765bbc-tmp-dir\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:51:58.728842 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:58.728842    6224 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27700938-db52-4225-90cd-244060765bbc-home\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:51:58.730582 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:58.730552    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27700938-db52-4225-90cd-244060765bbc-tls-certs" pod "27700938-db52-4225-90cd-244060765bbc" (UID: "27700938-db52-4225-90cd-244060765bbc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:51:58.730666 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:58.730582    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27700938-db52-4225-90cd-244060765bbc-dshm" pod "27700938-db52-4225-90cd-244060765bbc" (UID: "27700938-db52-4225-90cd-244060765bbc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:51:58.730666 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:58.730601    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27700938-db52-4225-90cd-244060765bbc-kube-api-access-wwk7p" pod "27700938-db52-4225-90cd-244060765bbc" (UID: "27700938-db52-4225-90cd-244060765bbc"). InnerVolumeSpecName "kube-api-access-wwk7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:51:58.830698 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:58.830669    6224 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27700938-db52-4225-90cd-244060765bbc-dshm\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:51:58.830698 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:58.830697    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwk7p\" (UniqueName: \"kubernetes.io/projected/27700938-db52-4225-90cd-244060765bbc-kube-api-access-wwk7p\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:51:58.830900 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:58.830707    6224 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27700938-db52-4225-90cd-244060765bbc-tls-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:51:59.295465 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:59.295425    6224 generic.go:356] "Generic (PLEG): container finished" podID="27700938-db52-4225-90cd-244060765bbc" containerID="437cec0b042f64cfaf3cca470402569407e84494291aa5cfc4913d42f1a1cdc2" exitCode=0
Oct 30 07:51:59.295616 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:59.295468    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8fef73/test-model-8fef73-kserve-54ff956b49-429dm"
Oct 30 07:51:59.295616 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:59.295514    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8fef73/test-model-8fef73-kserve-54ff956b49-429dm" event={"ID":"27700938-db52-4225-90cd-244060765bbc","Type":"ContainerDied","Data":"437cec0b042f64cfaf3cca470402569407e84494291aa5cfc4913d42f1a1cdc2"}
Oct 30 07:51:59.295616 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:59.295557    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8fef73/test-model-8fef73-kserve-54ff956b49-429dm" event={"ID":"27700938-db52-4225-90cd-244060765bbc","Type":"ContainerDied","Data":"465613d3ba8239d5cf53cbd416d94cac8de8f86289dd8ae636fb87bb82228caf"}
Oct 30 07:51:59.295616 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:59.295577    6224 scope.go:122] "RemoveContainer" containerID="437cec0b042f64cfaf3cca470402569407e84494291aa5cfc4913d42f1a1cdc2"
Oct 30 07:51:59.307267 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:59.307244    6224 scope.go:122] "RemoveContainer" containerID="437cec0b042f64cfaf3cca470402569407e84494291aa5cfc4913d42f1a1cdc2"
Oct 30 07:51:59.307527 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:51:59.307507    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"437cec0b042f64cfaf3cca470402569407e84494291aa5cfc4913d42f1a1cdc2\": container with ID starting with 437cec0b042f64cfaf3cca470402569407e84494291aa5cfc4913d42f1a1cdc2 not found: ID does not exist" containerID="437cec0b042f64cfaf3cca470402569407e84494291aa5cfc4913d42f1a1cdc2"
Oct 30 07:51:59.307604 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:59.307540    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"437cec0b042f64cfaf3cca470402569407e84494291aa5cfc4913d42f1a1cdc2"} err="failed to get container status \"437cec0b042f64cfaf3cca470402569407e84494291aa5cfc4913d42f1a1cdc2\": rpc error: code = NotFound desc = could not find container \"437cec0b042f64cfaf3cca470402569407e84494291aa5cfc4913d42f1a1cdc2\": container with ID starting with 437cec0b042f64cfaf3cca470402569407e84494291aa5cfc4913d42f1a1cdc2 not found: ID does not exist"
Oct 30 07:51:59.358001 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:59.357971    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-8fef73/test-model-8fef73-kserve-54ff956b49-429dm"]
Oct 30 07:51:59.362343 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:59.362322    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-8fef73/test-model-8fef73-kserve-54ff956b49-429dm"]
Oct 30 07:51:59.506733 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:59.506699    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27700938-db52-4225-90cd-244060765bbc" path="/var/lib/kubelet/pods/27700938-db52-4225-90cd-244060765bbc/volumes"
Oct 30 07:51:59.542797 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:51:59.542759    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-8fef73/test-model-8fef73-kserve-54ff956b49-429dm" podUID="27700938-db52-4225-90cd-244060765bbc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.194:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:52:01.864891 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:52:01.860785    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:52:02.223190 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:02.223106    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-8514a3/test-model-8514a3-kserve-564c455ff5-sxd7p"]
Oct 30 07:52:02.224070 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:02.224045    6224 cpu_manager.go:414] "containerMap: removing container" podUID="27700938-db52-4225-90cd-244060765bbc" containerName="main"
Oct 30 07:52:02.224070 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:02.224069    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27700938-db52-4225-90cd-244060765bbc" containerName="main"
Oct 30 07:52:02.224292 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:02.224112    6224 cpu_manager.go:414] "containerMap: removing container" podUID="8a2822e1-a884-41cb-a5c2-34972b1d4de4" containerName="payload-processing"
Oct 30 07:52:02.224292 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:02.224121    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a2822e1-a884-41cb-a5c2-34972b1d4de4" containerName="payload-processing"
Oct 30 07:52:02.224406 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:02.224327    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="27700938-db52-4225-90cd-244060765bbc" containerName="main"
Oct 30 07:52:02.224406 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:02.224347    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a2822e1-a884-41cb-a5c2-34972b1d4de4" containerName="payload-processing"
Oct 30 07:52:02.229244 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:02.229220    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8514a3/test-model-8514a3-kserve-564c455ff5-sxd7p"
Oct 30 07:52:02.254435 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:02.254401    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-8514a3/test-model-8514a3-kserve-564c455ff5-sxd7p"]
Oct 30 07:52:02.258395 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:02.258369    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-8514a3\"/\"default-dockercfg-5jcc8\""
Oct 30 07:52:02.258395 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:02.258372    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-8514a3\"/\"test-model-8514a3-kserve-self-signed-certs\""
Oct 30 07:52:02.260471 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:02.260448    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-8514a3\"/\"openshift-service-ca.crt\""
Oct 30 07:52:02.260601 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:02.260491    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-8514a3\"/\"kube-root-ca.crt\""
Oct 30 07:52:02.277817 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:02.277778    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z85s\" (UniqueName: \"kubernetes.io/projected/12cd7103-6458-49fd-b23b-160a32f26b87-kube-api-access-2z85s\") pod \"test-model-8514a3-kserve-564c455ff5-sxd7p\" (UID: \"12cd7103-6458-49fd-b23b-160a32f26b87\") " pod="ai-tenant-e2e-inf-b-8514a3/test-model-8514a3-kserve-564c455ff5-sxd7p"
Oct 30 07:52:02.277963 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:02.277927    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12cd7103-6458-49fd-b23b-160a32f26b87-model-cache\") pod \"test-model-8514a3-kserve-564c455ff5-sxd7p\" (UID: \"12cd7103-6458-49fd-b23b-160a32f26b87\") " pod="ai-tenant-e2e-inf-b-8514a3/test-model-8514a3-kserve-564c455ff5-sxd7p"
Oct 30 07:52:02.278045 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:02.277996    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12cd7103-6458-49fd-b23b-160a32f26b87-dshm\") pod \"test-model-8514a3-kserve-564c455ff5-sxd7p\" (UID: \"12cd7103-6458-49fd-b23b-160a32f26b87\") " pod="ai-tenant-e2e-inf-b-8514a3/test-model-8514a3-kserve-564c455ff5-sxd7p"
Oct 30 07:52:02.278045 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:02.278030    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12cd7103-6458-49fd-b23b-160a32f26b87-tls-certs\") pod \"test-model-8514a3-kserve-564c455ff5-sxd7p\" (UID: \"12cd7103-6458-49fd-b23b-160a32f26b87\") " pod="ai-tenant-e2e-inf-b-8514a3/test-model-8514a3-kserve-564c455ff5-sxd7p"
Oct 30 07:52:02.278167 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:02.278134    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12cd7103-6458-49fd-b23b-160a32f26b87-tmp-dir\") pod \"test-model-8514a3-kserve-564c455ff5-sxd7p\" (UID: \"12cd7103-6458-49fd-b23b-160a32f26b87\") " pod="ai-tenant-e2e-inf-b-8514a3/test-model-8514a3-kserve-564c455ff5-sxd7p"
Oct 30 07:52:02.278303 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:02.278286    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12cd7103-6458-49fd-b23b-160a32f26b87-home\") pod \"test-model-8514a3-kserve-564c455ff5-sxd7p\" (UID: \"12cd7103-6458-49fd-b23b-160a32f26b87\") " pod="ai-tenant-e2e-inf-b-8514a3/test-model-8514a3-kserve-564c455ff5-sxd7p"
Oct 30 07:52:02.380153 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:02.380113    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12cd7103-6458-49fd-b23b-160a32f26b87-home\") pod \"test-model-8514a3-kserve-564c455ff5-sxd7p\" (UID: \"12cd7103-6458-49fd-b23b-160a32f26b87\") " pod="ai-tenant-e2e-inf-b-8514a3/test-model-8514a3-kserve-564c455ff5-sxd7p"
Oct 30 07:52:02.380364 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:02.380255    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z85s\" (UniqueName: \"kubernetes.io/projected/12cd7103-6458-49fd-b23b-160a32f26b87-kube-api-access-2z85s\") pod \"test-model-8514a3-kserve-564c455ff5-sxd7p\" (UID: \"12cd7103-6458-49fd-b23b-160a32f26b87\") " pod="ai-tenant-e2e-inf-b-8514a3/test-model-8514a3-kserve-564c455ff5-sxd7p"
Oct 30 07:52:02.380364 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:02.380291    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12cd7103-6458-49fd-b23b-160a32f26b87-model-cache\") pod \"test-model-8514a3-kserve-564c455ff5-sxd7p\" (UID: \"12cd7103-6458-49fd-b23b-160a32f26b87\") " pod="ai-tenant-e2e-inf-b-8514a3/test-model-8514a3-kserve-564c455ff5-sxd7p"
Oct 30 07:52:02.380364 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:02.380306    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12cd7103-6458-49fd-b23b-160a32f26b87-dshm\") pod \"test-model-8514a3-kserve-564c455ff5-sxd7p\" (UID: \"12cd7103-6458-49fd-b23b-160a32f26b87\") " pod="ai-tenant-e2e-inf-b-8514a3/test-model-8514a3-kserve-564c455ff5-sxd7p"
Oct 30 07:52:02.380364 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:02.380325    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12cd7103-6458-49fd-b23b-160a32f26b87-tls-certs\") pod \"test-model-8514a3-kserve-564c455ff5-sxd7p\" (UID: \"12cd7103-6458-49fd-b23b-160a32f26b87\") " pod="ai-tenant-e2e-inf-b-8514a3/test-model-8514a3-kserve-564c455ff5-sxd7p"
Oct 30 07:52:02.380614 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:02.380374    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12cd7103-6458-49fd-b23b-160a32f26b87-tmp-dir\") pod \"test-model-8514a3-kserve-564c455ff5-sxd7p\" (UID: \"12cd7103-6458-49fd-b23b-160a32f26b87\") " pod="ai-tenant-e2e-inf-b-8514a3/test-model-8514a3-kserve-564c455ff5-sxd7p"
Oct 30 07:52:02.380614 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:02.380588    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12cd7103-6458-49fd-b23b-160a32f26b87-home\") pod \"test-model-8514a3-kserve-564c455ff5-sxd7p\" (UID: \"12cd7103-6458-49fd-b23b-160a32f26b87\") " pod="ai-tenant-e2e-inf-b-8514a3/test-model-8514a3-kserve-564c455ff5-sxd7p"
Oct 30 07:52:02.380703 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:02.380672    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12cd7103-6458-49fd-b23b-160a32f26b87-model-cache\") pod \"test-model-8514a3-kserve-564c455ff5-sxd7p\" (UID: \"12cd7103-6458-49fd-b23b-160a32f26b87\") " pod="ai-tenant-e2e-inf-b-8514a3/test-model-8514a3-kserve-564c455ff5-sxd7p"
Oct 30 07:52:02.380789 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:02.380762    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12cd7103-6458-49fd-b23b-160a32f26b87-tmp-dir\") pod \"test-model-8514a3-kserve-564c455ff5-sxd7p\" (UID: \"12cd7103-6458-49fd-b23b-160a32f26b87\") " pod="ai-tenant-e2e-inf-b-8514a3/test-model-8514a3-kserve-564c455ff5-sxd7p"
Oct 30 07:52:02.382940 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:02.382918    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12cd7103-6458-49fd-b23b-160a32f26b87-dshm\") pod \"test-model-8514a3-kserve-564c455ff5-sxd7p\" (UID: \"12cd7103-6458-49fd-b23b-160a32f26b87\") " pod="ai-tenant-e2e-inf-b-8514a3/test-model-8514a3-kserve-564c455ff5-sxd7p"
Oct 30 07:52:02.383125 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:02.383099    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12cd7103-6458-49fd-b23b-160a32f26b87-tls-certs\") pod \"test-model-8514a3-kserve-564c455ff5-sxd7p\" (UID: \"12cd7103-6458-49fd-b23b-160a32f26b87\") " pod="ai-tenant-e2e-inf-b-8514a3/test-model-8514a3-kserve-564c455ff5-sxd7p"
Oct 30 07:52:02.406387 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:02.406353    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z85s\" (UniqueName: \"kubernetes.io/projected/12cd7103-6458-49fd-b23b-160a32f26b87-kube-api-access-2z85s\") pod \"test-model-8514a3-kserve-564c455ff5-sxd7p\" (UID: \"12cd7103-6458-49fd-b23b-160a32f26b87\") " pod="ai-tenant-e2e-inf-b-8514a3/test-model-8514a3-kserve-564c455ff5-sxd7p"
Oct 30 07:52:02.548211 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:02.548172    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8514a3/test-model-8514a3-kserve-564c455ff5-sxd7p"
Oct 30 07:52:02.879692 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:02.879665    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-8514a3/test-model-8514a3-kserve-564c455ff5-sxd7p"]
Oct 30 07:52:02.881639 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:52:02.881615    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod12cd7103_6458_49fd_b23b_160a32f26b87.slice/crio-7f6cc5fd63f87926d9909832ffb8d74c901c73f82ce774d36a0573905ce5048f WatchSource:0}: Error finding container 7f6cc5fd63f87926d9909832ffb8d74c901c73f82ce774d36a0573905ce5048f: Status 404 returned error can't find the container with id 7f6cc5fd63f87926d9909832ffb8d74c901c73f82ce774d36a0573905ce5048f
Oct 30 07:52:03.331429 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:03.331393    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8514a3/test-model-8514a3-kserve-564c455ff5-sxd7p" event={"ID":"12cd7103-6458-49fd-b23b-160a32f26b87","Type":"ContainerStarted","Data":"791be768c7c4c64ad632a45cd2492d3a76f790814fd2b1a7315c7653fd4e1b8b"}
Oct 30 07:52:03.331649 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:03.331444    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8514a3/test-model-8514a3-kserve-564c455ff5-sxd7p" event={"ID":"12cd7103-6458-49fd-b23b-160a32f26b87","Type":"ContainerStarted","Data":"7f6cc5fd63f87926d9909832ffb8d74c901c73f82ce774d36a0573905ce5048f"}
Oct 30 07:52:03.376914 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:03.376865    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-8514a3/test-model-8514a3-kserve-564c455ff5-sxd7p" podStartSLOduration=1.376850152 podStartE2EDuration="1.376850152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:52:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:52:03.376266836 +0000 UTC m=+3281.985673811" watchObservedRunningTime="2026-10-30 07:52:03.376850152 +0000 UTC m=+3281.986257123"
Oct 30 07:52:03.548982 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:03.548942    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-8514a3/test-model-8514a3-kserve-564c455ff5-sxd7p"
Oct 30 07:52:06.270796 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:06.270760    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:52:10.005738 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:10.005706    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4ab37e-65c46fb659-dbr69"]
Oct 30 07:52:10.006205 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:10.005908    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4ab37e-65c46fb659-dbr69" podUID="efa670c6-14aa-4c5e-be1c-9976b974bdff" containerName="maas-api" containerID="cri-o://fcf444f4238faafad50883aba8ab24c820ed10f9af9648152b12394a2c7d4b08" gracePeriod=30
Oct 30 07:52:10.392778 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:10.392743    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4ab37e-65c46fb659-dbr69"
Oct 30 07:52:10.398313 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:10.398284    6224 generic.go:356] "Generic (PLEG): container finished" podID="efa670c6-14aa-4c5e-be1c-9976b974bdff" containerID="fcf444f4238faafad50883aba8ab24c820ed10f9af9648152b12394a2c7d4b08" exitCode=0
Oct 30 07:52:10.398484 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:10.398322    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4ab37e-65c46fb659-dbr69"
Oct 30 07:52:10.398484 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:10.398336    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4ab37e-65c46fb659-dbr69" event={"ID":"efa670c6-14aa-4c5e-be1c-9976b974bdff","Type":"ContainerDied","Data":"fcf444f4238faafad50883aba8ab24c820ed10f9af9648152b12394a2c7d4b08"}
Oct 30 07:52:10.398484 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:10.398405    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4ab37e-65c46fb659-dbr69" event={"ID":"efa670c6-14aa-4c5e-be1c-9976b974bdff","Type":"ContainerDied","Data":"2626bab648bc167dc1f3861bc8df0989d6c15f4ba7a7c8810d58b2a6d84fe696"}
Oct 30 07:52:10.398484 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:10.398427    6224 scope.go:122] "RemoveContainer" containerID="fcf444f4238faafad50883aba8ab24c820ed10f9af9648152b12394a2c7d4b08"
Oct 30 07:52:10.410654 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:10.410630    6224 scope.go:122] "RemoveContainer" containerID="fcf444f4238faafad50883aba8ab24c820ed10f9af9648152b12394a2c7d4b08"
Oct 30 07:52:10.410982 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:52:10.410959    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcf444f4238faafad50883aba8ab24c820ed10f9af9648152b12394a2c7d4b08\": container with ID starting with fcf444f4238faafad50883aba8ab24c820ed10f9af9648152b12394a2c7d4b08 not found: ID does not exist" containerID="fcf444f4238faafad50883aba8ab24c820ed10f9af9648152b12394a2c7d4b08"
Oct 30 07:52:10.411070 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:10.410994    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcf444f4238faafad50883aba8ab24c820ed10f9af9648152b12394a2c7d4b08"} err="failed to get container status \"fcf444f4238faafad50883aba8ab24c820ed10f9af9648152b12394a2c7d4b08\": rpc error: code = NotFound desc = could not find container \"fcf444f4238faafad50883aba8ab24c820ed10f9af9648152b12394a2c7d4b08\": container with ID starting with fcf444f4238faafad50883aba8ab24c820ed10f9af9648152b12394a2c7d4b08 not found: ID does not exist"
Oct 30 07:52:10.536990 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:10.536954    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efa670c6-14aa-4c5e-be1c-9976b974bdff-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efa670c6-14aa-4c5e-be1c-9976b974bdff-maas-api-tls\") pod \"efa670c6-14aa-4c5e-be1c-9976b974bdff\" (UID: \"efa670c6-14aa-4c5e-be1c-9976b974bdff\") "
Oct 30 07:52:10.537177 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:10.537033    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efa670c6-14aa-4c5e-be1c-9976b974bdff-metrics-certs\" (UniqueName: \"kubernetes.io/secret/efa670c6-14aa-4c5e-be1c-9976b974bdff-metrics-certs\") pod \"efa670c6-14aa-4c5e-be1c-9976b974bdff\" (UID: \"efa670c6-14aa-4c5e-be1c-9976b974bdff\") "
Oct 30 07:52:10.537228 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:10.537183    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efa670c6-14aa-4c5e-be1c-9976b974bdff-kube-api-access-kgqh2\" (UniqueName: \"kubernetes.io/projected/efa670c6-14aa-4c5e-be1c-9976b974bdff-kube-api-access-kgqh2\") pod \"efa670c6-14aa-4c5e-be1c-9976b974bdff\" (UID: \"efa670c6-14aa-4c5e-be1c-9976b974bdff\") "
Oct 30 07:52:10.537437 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:10.537416    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efa670c6-14aa-4c5e-be1c-9976b974bdff-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efa670c6-14aa-4c5e-be1c-9976b974bdff-openshift-service-ca\") pod \"efa670c6-14aa-4c5e-be1c-9976b974bdff\" (UID: \"efa670c6-14aa-4c5e-be1c-9976b974bdff\") "
Oct 30 07:52:10.537885 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:10.537855    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efa670c6-14aa-4c5e-be1c-9976b974bdff-openshift-service-ca" pod "efa670c6-14aa-4c5e-be1c-9976b974bdff" (UID: "efa670c6-14aa-4c5e-be1c-9976b974bdff"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:52:10.538087 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:10.538065    6224 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efa670c6-14aa-4c5e-be1c-9976b974bdff-openshift-service-ca\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:52:10.539844 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:10.539775    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efa670c6-14aa-4c5e-be1c-9976b974bdff-maas-api-tls" pod "efa670c6-14aa-4c5e-be1c-9976b974bdff" (UID: "efa670c6-14aa-4c5e-be1c-9976b974bdff"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:52:10.539844 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:10.539788    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efa670c6-14aa-4c5e-be1c-9976b974bdff-kube-api-access-kgqh2" pod "efa670c6-14aa-4c5e-be1c-9976b974bdff" (UID: "efa670c6-14aa-4c5e-be1c-9976b974bdff"). InnerVolumeSpecName "kube-api-access-kgqh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:52:10.539986 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:10.539865    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efa670c6-14aa-4c5e-be1c-9976b974bdff-metrics-certs" pod "efa670c6-14aa-4c5e-be1c-9976b974bdff" (UID: "efa670c6-14aa-4c5e-be1c-9976b974bdff"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:52:10.641880 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:10.639736    6224 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efa670c6-14aa-4c5e-be1c-9976b974bdff-maas-api-tls\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:52:10.641880 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:10.639791    6224 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efa670c6-14aa-4c5e-be1c-9976b974bdff-metrics-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:52:10.641880 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:10.639857    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgqh2\" (UniqueName: \"kubernetes.io/projected/efa670c6-14aa-4c5e-be1c-9976b974bdff-kube-api-access-kgqh2\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:52:10.751481 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:10.751441    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4ab37e-65c46fb659-dbr69"]
Oct 30 07:52:10.764954 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:10.764923    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4ab37e-65c46fb659-dbr69"]
Oct 30 07:52:11.505734 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:11.505698    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efa670c6-14aa-4c5e-be1c-9976b974bdff" path="/var/lib/kubelet/pods/efa670c6-14aa-4c5e-be1c-9976b974bdff/volumes"
Oct 30 07:52:12.259401 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:12.259367    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f480c6/test-model-f480c6-kserve-7bbd97ff6f-75xld"]
Oct 30 07:52:12.549823 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:12.548898    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-8514a3/test-model-8514a3-kserve-564c455ff5-sxd7p"
Oct 30 07:52:12.562365 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:12.562339    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-8514a3/test-model-8514a3-kserve-564c455ff5-sxd7p"
Oct 30 07:52:13.443212 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:13.443185    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-8514a3/test-model-8514a3-kserve-564c455ff5-sxd7p"
Oct 30 07:52:15.598617 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:15.598579    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dtptn"]
Oct 30 07:52:15.599109 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:15.598761    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dtptn" podUID="0a0e028c-e9eb-4260-bfa3-37104c26af0e" containerName="istio-proxy" containerID="cri-o://38419b1742aa3fdf56b3aca550444cdbdcfa1a9ebf7069bf5ffecde45b004f61" gracePeriod=30
Oct 30 07:52:15.613207 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:15.613173    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gshg6"]
Oct 30 07:52:15.613446 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:15.613400    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gshg6" podUID="02fa3182-1c4a-418e-882b-08db6f71a0a7" containerName="istio-proxy" containerID="cri-o://64497b29ea7254f96b9c5b866e4ba23ff66249e289eb73b8e56c235f49b4b153" gracePeriod=30
Oct 30 07:52:18.154241 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:18.154205    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4"]
Oct 30 07:52:18.155084 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:18.155040    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4" podUID="0f2070a8-8f2b-4673-a431-b3640f77d332" containerName="istio-proxy" containerID="cri-o://652a781cfc9622e2da68ceb37967d8d7cf520aa7d538faf1c26dc2ac46291958" gracePeriod=30
Oct 30 07:52:20.943674 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:20.943638    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dtptn"
Oct 30 07:52:20.947997 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:20.947974    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gshg6"
Oct 30 07:52:20.981628 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:20.981600    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02fa3182-1c4a-418e-882b-08db6f71a0a7-istio-token\" (UniqueName: \"kubernetes.io/projected/02fa3182-1c4a-418e-882b-08db6f71a0a7-istio-token\") pod \"02fa3182-1c4a-418e-882b-08db6f71a0a7\" (UID: \"02fa3182-1c4a-418e-882b-08db6f71a0a7\") "
Oct 30 07:52:20.981766 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:20.981660    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02fa3182-1c4a-418e-882b-08db6f71a0a7-kube-api-access-zqcxx\" (UniqueName: \"kubernetes.io/projected/02fa3182-1c4a-418e-882b-08db6f71a0a7-kube-api-access-zqcxx\") pod \"02fa3182-1c4a-418e-882b-08db6f71a0a7\" (UID: \"02fa3182-1c4a-418e-882b-08db6f71a0a7\") "
Oct 30 07:52:20.981766 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:20.981694    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/02fa3182-1c4a-418e-882b-08db6f71a0a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02fa3182-1c4a-418e-882b-08db6f71a0a7-istio-podinfo\") pod \"02fa3182-1c4a-418e-882b-08db6f71a0a7\" (UID: \"02fa3182-1c4a-418e-882b-08db6f71a0a7\") "
Oct 30 07:52:20.981766 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:20.981726    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a0e028c-e9eb-4260-bfa3-37104c26af0e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a0e028c-e9eb-4260-bfa3-37104c26af0e-istio-data\") pod \"0a0e028c-e9eb-4260-bfa3-37104c26af0e\" (UID: \"0a0e028c-e9eb-4260-bfa3-37104c26af0e\") "
Oct 30 07:52:20.982089 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:20.982044    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02fa3182-1c4a-418e-882b-08db6f71a0a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02fa3182-1c4a-418e-882b-08db6f71a0a7-credential-socket\") pod \"02fa3182-1c4a-418e-882b-08db6f71a0a7\" (UID: \"02fa3182-1c4a-418e-882b-08db6f71a0a7\") "
Oct 30 07:52:20.983552 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:20.982417    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a0e028c-e9eb-4260-bfa3-37104c26af0e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a0e028c-e9eb-4260-bfa3-37104c26af0e-workload-certs\") pod \"0a0e028c-e9eb-4260-bfa3-37104c26af0e\" (UID: \"0a0e028c-e9eb-4260-bfa3-37104c26af0e\") "
Oct 30 07:52:20.983552 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:20.982471    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02fa3182-1c4a-418e-882b-08db6f71a0a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02fa3182-1c4a-418e-882b-08db6f71a0a7-istiod-ca-cert\") pod \"02fa3182-1c4a-418e-882b-08db6f71a0a7\" (UID: \"02fa3182-1c4a-418e-882b-08db6f71a0a7\") "
Oct 30 07:52:20.983552 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:20.982514    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a0e028c-e9eb-4260-bfa3-37104c26af0e-kube-api-access-2bhmd\" (UniqueName: \"kubernetes.io/projected/0a0e028c-e9eb-4260-bfa3-37104c26af0e-kube-api-access-2bhmd\") pod \"0a0e028c-e9eb-4260-bfa3-37104c26af0e\" (UID: \"0a0e028c-e9eb-4260-bfa3-37104c26af0e\") "
Oct 30 07:52:20.983552 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:20.982583    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02fa3182-1c4a-418e-882b-08db6f71a0a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02fa3182-1c4a-418e-882b-08db6f71a0a7-istio-envoy\") pod \"02fa3182-1c4a-418e-882b-08db6f71a0a7\" (UID: \"02fa3182-1c4a-418e-882b-08db6f71a0a7\") "
Oct 30 07:52:20.983552 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:20.982610    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a0e028c-e9eb-4260-bfa3-37104c26af0e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a0e028c-e9eb-4260-bfa3-37104c26af0e-istio-envoy\") pod \"0a0e028c-e9eb-4260-bfa3-37104c26af0e\" (UID: \"0a0e028c-e9eb-4260-bfa3-37104c26af0e\") "
Oct 30 07:52:20.983552 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:20.982677    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02fa3182-1c4a-418e-882b-08db6f71a0a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02fa3182-1c4a-418e-882b-08db6f71a0a7-workload-certs\") pod \"02fa3182-1c4a-418e-882b-08db6f71a0a7\" (UID: \"02fa3182-1c4a-418e-882b-08db6f71a0a7\") "
Oct 30 07:52:20.983552 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:20.982720    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02fa3182-1c4a-418e-882b-08db6f71a0a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02fa3182-1c4a-418e-882b-08db6f71a0a7-istio-data\") pod \"02fa3182-1c4a-418e-882b-08db6f71a0a7\" (UID: \"02fa3182-1c4a-418e-882b-08db6f71a0a7\") "
Oct 30 07:52:20.983552 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:20.982748    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a0e028c-e9eb-4260-bfa3-37104c26af0e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a0e028c-e9eb-4260-bfa3-37104c26af0e-istiod-ca-cert\") pod \"0a0e028c-e9eb-4260-bfa3-37104c26af0e\" (UID: \"0a0e028c-e9eb-4260-bfa3-37104c26af0e\") "
Oct 30 07:52:20.983552 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:20.982985    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a0e028c-e9eb-4260-bfa3-37104c26af0e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a0e028c-e9eb-4260-bfa3-37104c26af0e-istio-podinfo\") pod \"0a0e028c-e9eb-4260-bfa3-37104c26af0e\" (UID: \"0a0e028c-e9eb-4260-bfa3-37104c26af0e\") "
Oct 30 07:52:20.983552 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:20.983039    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a0e028c-e9eb-4260-bfa3-37104c26af0e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a0e028c-e9eb-4260-bfa3-37104c26af0e-credential-socket\") pod \"0a0e028c-e9eb-4260-bfa3-37104c26af0e\" (UID: \"0a0e028c-e9eb-4260-bfa3-37104c26af0e\") "
Oct 30 07:52:20.984378 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:20.982356    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02fa3182-1c4a-418e-882b-08db6f71a0a7-credential-socket" pod "02fa3182-1c4a-418e-882b-08db6f71a0a7" (UID: "02fa3182-1c4a-418e-882b-08db6f71a0a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:52:20.984378 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:20.983243    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a0e028c-e9eb-4260-bfa3-37104c26af0e-istio-data" pod "0a0e028c-e9eb-4260-bfa3-37104c26af0e" (UID: "0a0e028c-e9eb-4260-bfa3-37104c26af0e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:52:20.984378 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:20.983571    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02fa3182-1c4a-418e-882b-08db6f71a0a7-istiod-ca-cert" pod "02fa3182-1c4a-418e-882b-08db6f71a0a7" (UID: "02fa3182-1c4a-418e-882b-08db6f71a0a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:52:20.984378 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:20.983764    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a0e028c-e9eb-4260-bfa3-37104c26af0e-workload-certs" pod "0a0e028c-e9eb-4260-bfa3-37104c26af0e" (UID: "0a0e028c-e9eb-4260-bfa3-37104c26af0e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:52:20.984378 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:20.983866    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02fa3182-1c4a-418e-882b-08db6f71a0a7-istio-data" pod "02fa3182-1c4a-418e-882b-08db6f71a0a7" (UID: "02fa3182-1c4a-418e-882b-08db6f71a0a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:52:20.984378 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:20.984169    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a0e028c-e9eb-4260-bfa3-37104c26af0e-istiod-ca-cert" pod "0a0e028c-e9eb-4260-bfa3-37104c26af0e" (UID: "0a0e028c-e9eb-4260-bfa3-37104c26af0e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:52:20.984832 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:20.984793    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a0e028c-e9eb-4260-bfa3-37104c26af0e-credential-socket" pod "0a0e028c-e9eb-4260-bfa3-37104c26af0e" (UID: "0a0e028c-e9eb-4260-bfa3-37104c26af0e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:52:20.984931 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:20.984914    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02fa3182-1c4a-418e-882b-08db6f71a0a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02fa3182-1c4a-418e-882b-08db6f71a0a7-workload-socket\") pod \"02fa3182-1c4a-418e-882b-08db6f71a0a7\" (UID: \"02fa3182-1c4a-418e-882b-08db6f71a0a7\") "
Oct 30 07:52:20.985020 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:20.985000    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a0e028c-e9eb-4260-bfa3-37104c26af0e-istio-token\" (UniqueName: \"kubernetes.io/projected/0a0e028c-e9eb-4260-bfa3-37104c26af0e-istio-token\") pod \"0a0e028c-e9eb-4260-bfa3-37104c26af0e\" (UID: \"0a0e028c-e9eb-4260-bfa3-37104c26af0e\") "
Oct 30 07:52:20.985120 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:20.985075    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a0e028c-e9eb-4260-bfa3-37104c26af0e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a0e028c-e9eb-4260-bfa3-37104c26af0e-workload-socket\") pod \"0a0e028c-e9eb-4260-bfa3-37104c26af0e\" (UID: \"0a0e028c-e9eb-4260-bfa3-37104c26af0e\") "
Oct 30 07:52:20.985764 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:20.985733    6224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02fa3182-1c4a-418e-882b-08db6f71a0a7-istio-data\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:52:20.985764 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:20.985756    6224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a0e028c-e9eb-4260-bfa3-37104c26af0e-istiod-ca-cert\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:52:20.985764 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:20.985768    6224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a0e028c-e9eb-4260-bfa3-37104c26af0e-credential-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:52:20.986102 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:20.985777    6224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a0e028c-e9eb-4260-bfa3-37104c26af0e-istio-data\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:52:20.986102 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:20.985785    6224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02fa3182-1c4a-418e-882b-08db6f71a0a7-credential-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:52:20.986102 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:20.985793    6224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a0e028c-e9eb-4260-bfa3-37104c26af0e-workload-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:52:20.986102 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:20.985817    6224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02fa3182-1c4a-418e-882b-08db6f71a0a7-istiod-ca-cert\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:52:20.986102 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:20.986047    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a0e028c-e9eb-4260-bfa3-37104c26af0e-workload-socket" pod "0a0e028c-e9eb-4260-bfa3-37104c26af0e" (UID: "0a0e028c-e9eb-4260-bfa3-37104c26af0e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:52:20.986531 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:20.986231    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02fa3182-1c4a-418e-882b-08db6f71a0a7-workload-socket" pod "02fa3182-1c4a-418e-882b-08db6f71a0a7" (UID: "02fa3182-1c4a-418e-882b-08db6f71a0a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:52:20.987049 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:20.986909    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a0e028c-e9eb-4260-bfa3-37104c26af0e-istio-envoy" pod "0a0e028c-e9eb-4260-bfa3-37104c26af0e" (UID: "0a0e028c-e9eb-4260-bfa3-37104c26af0e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:52:20.988200 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:20.987103    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02fa3182-1c4a-418e-882b-08db6f71a0a7-workload-certs" pod "02fa3182-1c4a-418e-882b-08db6f71a0a7" (UID: "02fa3182-1c4a-418e-882b-08db6f71a0a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:52:20.990646 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:20.990618    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02fa3182-1c4a-418e-882b-08db6f71a0a7-istio-token" pod "02fa3182-1c4a-418e-882b-08db6f71a0a7" (UID: "02fa3182-1c4a-418e-882b-08db6f71a0a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:52:20.991294 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:20.991254    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a0e028c-e9eb-4260-bfa3-37104c26af0e-kube-api-access-2bhmd" pod "0a0e028c-e9eb-4260-bfa3-37104c26af0e" (UID: "0a0e028c-e9eb-4260-bfa3-37104c26af0e"). InnerVolumeSpecName "kube-api-access-2bhmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:52:20.991415 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:20.991385    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02fa3182-1c4a-418e-882b-08db6f71a0a7-istio-envoy" pod "02fa3182-1c4a-418e-882b-08db6f71a0a7" (UID: "02fa3182-1c4a-418e-882b-08db6f71a0a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:52:20.991415 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:20.991407    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/02fa3182-1c4a-418e-882b-08db6f71a0a7-istio-podinfo" pod "02fa3182-1c4a-418e-882b-08db6f71a0a7" (UID: "02fa3182-1c4a-418e-882b-08db6f71a0a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:52:20.991598 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:20.991576    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02fa3182-1c4a-418e-882b-08db6f71a0a7-kube-api-access-zqcxx" pod "02fa3182-1c4a-418e-882b-08db6f71a0a7" (UID: "02fa3182-1c4a-418e-882b-08db6f71a0a7"). InnerVolumeSpecName "kube-api-access-zqcxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:52:20.999266 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:20.999234    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a0e028c-e9eb-4260-bfa3-37104c26af0e-istio-podinfo" pod "0a0e028c-e9eb-4260-bfa3-37104c26af0e" (UID: "0a0e028c-e9eb-4260-bfa3-37104c26af0e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:52:20.999396 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:20.999284    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a0e028c-e9eb-4260-bfa3-37104c26af0e-istio-token" pod "0a0e028c-e9eb-4260-bfa3-37104c26af0e" (UID: "0a0e028c-e9eb-4260-bfa3-37104c26af0e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:52:21.027537 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:21.027505    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm"]
Oct 30 07:52:21.027943 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:21.027720    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm" podUID="68dd39f6-f527-431d-94d4-52a6101d08de" containerName="istio-proxy" containerID="cri-o://9bf9195a84314feb2f56ae23b330b25f169c170076cfa5a010775827cedde575" gracePeriod=30
Oct 30 07:52:21.087154 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:21.087123    6224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a0e028c-e9eb-4260-bfa3-37104c26af0e-istio-token\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:52:21.087154 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:21.087151    6224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a0e028c-e9eb-4260-bfa3-37104c26af0e-workload-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:52:21.087154 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:21.087161    6224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02fa3182-1c4a-418e-882b-08db6f71a0a7-istio-token\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:52:21.087503 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:21.087170    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqcxx\" (UniqueName: \"kubernetes.io/projected/02fa3182-1c4a-418e-882b-08db6f71a0a7-kube-api-access-zqcxx\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:52:21.087503 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:21.087179    6224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02fa3182-1c4a-418e-882b-08db6f71a0a7-istio-podinfo\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:52:21.087503 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:21.087193    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bhmd\" (UniqueName: \"kubernetes.io/projected/0a0e028c-e9eb-4260-bfa3-37104c26af0e-kube-api-access-2bhmd\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:52:21.087503 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:21.087206    6224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02fa3182-1c4a-418e-882b-08db6f71a0a7-istio-envoy\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:52:21.087503 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:21.087214    6224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a0e028c-e9eb-4260-bfa3-37104c26af0e-istio-envoy\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:52:21.087503 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:21.087222    6224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02fa3182-1c4a-418e-882b-08db6f71a0a7-workload-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:52:21.087503 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:21.087231    6224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a0e028c-e9eb-4260-bfa3-37104c26af0e-istio-podinfo\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:52:21.087503 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:21.087242    6224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02fa3182-1c4a-418e-882b-08db6f71a0a7-workload-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:52:21.497382 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:21.497345    6224 generic.go:356] "Generic (PLEG): container finished" podID="02fa3182-1c4a-418e-882b-08db6f71a0a7" containerID="64497b29ea7254f96b9c5b866e4ba23ff66249e289eb73b8e56c235f49b4b153" exitCode=0
Oct 30 07:52:21.499524 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:21.499498    6224 generic.go:356] "Generic (PLEG): container finished" podID="0a0e028c-e9eb-4260-bfa3-37104c26af0e" containerID="38419b1742aa3fdf56b3aca550444cdbdcfa1a9ebf7069bf5ffecde45b004f61" exitCode=0
Oct 30 07:52:21.501834 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:21.501779    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gshg6"
Oct 30 07:52:21.501949 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:21.501828    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dtptn"
Oct 30 07:52:21.506534 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:21.506502    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gshg6" event={"ID":"02fa3182-1c4a-418e-882b-08db6f71a0a7","Type":"ContainerDied","Data":"64497b29ea7254f96b9c5b866e4ba23ff66249e289eb73b8e56c235f49b4b153"}
Oct 30 07:52:21.506662 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:21.506551    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gshg6" event={"ID":"02fa3182-1c4a-418e-882b-08db6f71a0a7","Type":"ContainerDied","Data":"909d86539fe4f91aeb59ca7b0e0029075eea245cb1851db390cbd83e825c2705"}
Oct 30 07:52:21.506662 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:21.506568    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dtptn" event={"ID":"0a0e028c-e9eb-4260-bfa3-37104c26af0e","Type":"ContainerDied","Data":"38419b1742aa3fdf56b3aca550444cdbdcfa1a9ebf7069bf5ffecde45b004f61"}
Oct 30 07:52:21.506662 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:21.506583    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dtptn" event={"ID":"0a0e028c-e9eb-4260-bfa3-37104c26af0e","Type":"ContainerDied","Data":"59a6482eb088c36fc7a30dddad74708643ca1448ea2bfcd4a4a5f314edb81d6a"}
Oct 30 07:52:21.506662 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:21.506608    6224 scope.go:122] "RemoveContainer" containerID="64497b29ea7254f96b9c5b866e4ba23ff66249e289eb73b8e56c235f49b4b153"
Oct 30 07:52:21.519743 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:21.519701    6224 scope.go:122] "RemoveContainer" containerID="64497b29ea7254f96b9c5b866e4ba23ff66249e289eb73b8e56c235f49b4b153"
Oct 30 07:52:21.520040 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:52:21.520018    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64497b29ea7254f96b9c5b866e4ba23ff66249e289eb73b8e56c235f49b4b153\": container with ID starting with 64497b29ea7254f96b9c5b866e4ba23ff66249e289eb73b8e56c235f49b4b153 not found: ID does not exist" containerID="64497b29ea7254f96b9c5b866e4ba23ff66249e289eb73b8e56c235f49b4b153"
Oct 30 07:52:21.520131 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:21.520049    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64497b29ea7254f96b9c5b866e4ba23ff66249e289eb73b8e56c235f49b4b153"} err="failed to get container status \"64497b29ea7254f96b9c5b866e4ba23ff66249e289eb73b8e56c235f49b4b153\": rpc error: code = NotFound desc = could not find container \"64497b29ea7254f96b9c5b866e4ba23ff66249e289eb73b8e56c235f49b4b153\": container with ID starting with 64497b29ea7254f96b9c5b866e4ba23ff66249e289eb73b8e56c235f49b4b153 not found: ID does not exist"
Oct 30 07:52:21.520131 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:21.520071    6224 scope.go:122] "RemoveContainer" containerID="38419b1742aa3fdf56b3aca550444cdbdcfa1a9ebf7069bf5ffecde45b004f61"
Oct 30 07:52:21.531370 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:21.531346    6224 scope.go:122] "RemoveContainer" containerID="38419b1742aa3fdf56b3aca550444cdbdcfa1a9ebf7069bf5ffecde45b004f61"
Oct 30 07:52:21.531636 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:52:21.531615    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38419b1742aa3fdf56b3aca550444cdbdcfa1a9ebf7069bf5ffecde45b004f61\": container with ID starting with 38419b1742aa3fdf56b3aca550444cdbdcfa1a9ebf7069bf5ffecde45b004f61 not found: ID does not exist" containerID="38419b1742aa3fdf56b3aca550444cdbdcfa1a9ebf7069bf5ffecde45b004f61"
Oct 30 07:52:21.531692 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:21.531644    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38419b1742aa3fdf56b3aca550444cdbdcfa1a9ebf7069bf5ffecde45b004f61"} err="failed to get container status \"38419b1742aa3fdf56b3aca550444cdbdcfa1a9ebf7069bf5ffecde45b004f61\": rpc error: code = NotFound desc = could not find container \"38419b1742aa3fdf56b3aca550444cdbdcfa1a9ebf7069bf5ffecde45b004f61\": container with ID starting with 38419b1742aa3fdf56b3aca550444cdbdcfa1a9ebf7069bf5ffecde45b004f61 not found: ID does not exist"
Oct 30 07:52:21.577394 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:21.577297    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dtptn"]
Oct 30 07:52:21.591855 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:21.591824    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dtptn"]
Oct 30 07:52:21.622542 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:21.622508    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gshg6"]
Oct 30 07:52:21.627516 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:21.627489    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gshg6"]
Oct 30 07:52:22.897227 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:22.897193    6224 patch_prober.go:29] interesting pod/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 07:52:22.897670 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:22.897247    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm" podUID="68dd39f6-f527-431d-94d4-52a6101d08de" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 07:52:22.973259 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:22.973027    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8fef73-656cd4fdb-st5tl"]
Oct 30 07:52:23.007228 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:23.007196    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8fef73-99b947487-lf4k2"]
Oct 30 07:52:23.453684 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:23.453659    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4"
Oct 30 07:52:23.506929 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:23.506894    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02fa3182-1c4a-418e-882b-08db6f71a0a7" path="/var/lib/kubelet/pods/02fa3182-1c4a-418e-882b-08db6f71a0a7/volumes"
Oct 30 07:52:23.507692 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:23.507660    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a0e028c-e9eb-4260-bfa3-37104c26af0e" path="/var/lib/kubelet/pods/0a0e028c-e9eb-4260-bfa3-37104c26af0e/volumes"
Oct 30 07:52:23.518225 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:23.518195    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f2070a8-8f2b-4673-a431-b3640f77d332-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f2070a8-8f2b-4673-a431-b3640f77d332-istiod-ca-cert\") pod \"0f2070a8-8f2b-4673-a431-b3640f77d332\" (UID: \"0f2070a8-8f2b-4673-a431-b3640f77d332\") "
Oct 30 07:52:23.518376 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:23.518239    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f2070a8-8f2b-4673-a431-b3640f77d332-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f2070a8-8f2b-4673-a431-b3640f77d332-istio-data\") pod \"0f2070a8-8f2b-4673-a431-b3640f77d332\" (UID: \"0f2070a8-8f2b-4673-a431-b3640f77d332\") "
Oct 30 07:52:23.518376 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:23.518258    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f2070a8-8f2b-4673-a431-b3640f77d332-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f2070a8-8f2b-4673-a431-b3640f77d332-workload-socket\") pod \"0f2070a8-8f2b-4673-a431-b3640f77d332\" (UID: \"0f2070a8-8f2b-4673-a431-b3640f77d332\") "
Oct 30 07:52:23.518507 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:23.518385    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f2070a8-8f2b-4673-a431-b3640f77d332-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f2070a8-8f2b-4673-a431-b3640f77d332-istio-envoy\") pod \"0f2070a8-8f2b-4673-a431-b3640f77d332\" (UID: \"0f2070a8-8f2b-4673-a431-b3640f77d332\") "
Oct 30 07:52:23.518507 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:23.518465    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0f2070a8-8f2b-4673-a431-b3640f77d332-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f2070a8-8f2b-4673-a431-b3640f77d332-istio-podinfo\") pod \"0f2070a8-8f2b-4673-a431-b3640f77d332\" (UID: \"0f2070a8-8f2b-4673-a431-b3640f77d332\") "
Oct 30 07:52:23.518617 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:23.518504    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f2070a8-8f2b-4673-a431-b3640f77d332-istio-data" pod "0f2070a8-8f2b-4673-a431-b3640f77d332" (UID: "0f2070a8-8f2b-4673-a431-b3640f77d332"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:52:23.518617 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:23.518537    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f2070a8-8f2b-4673-a431-b3640f77d332-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f2070a8-8f2b-4673-a431-b3640f77d332-credential-socket\") pod \"0f2070a8-8f2b-4673-a431-b3640f77d332\" (UID: \"0f2070a8-8f2b-4673-a431-b3640f77d332\") "
Oct 30 07:52:23.518617 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:23.518558    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f2070a8-8f2b-4673-a431-b3640f77d332-kube-api-access-68qk5\" (UniqueName: \"kubernetes.io/projected/0f2070a8-8f2b-4673-a431-b3640f77d332-kube-api-access-68qk5\") pod \"0f2070a8-8f2b-4673-a431-b3640f77d332\" (UID: \"0f2070a8-8f2b-4673-a431-b3640f77d332\") "
Oct 30 07:52:23.518772 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:23.518684    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f2070a8-8f2b-4673-a431-b3640f77d332-istio-token\" (UniqueName: \"kubernetes.io/projected/0f2070a8-8f2b-4673-a431-b3640f77d332-istio-token\") pod \"0f2070a8-8f2b-4673-a431-b3640f77d332\" (UID: \"0f2070a8-8f2b-4673-a431-b3640f77d332\") "
Oct 30 07:52:23.518772 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:23.518718    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f2070a8-8f2b-4673-a431-b3640f77d332-workload-socket" pod "0f2070a8-8f2b-4673-a431-b3640f77d332" (UID: "0f2070a8-8f2b-4673-a431-b3640f77d332"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:52:23.518772 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:23.518636    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f2070a8-8f2b-4673-a431-b3640f77d332-istiod-ca-cert" pod "0f2070a8-8f2b-4673-a431-b3640f77d332" (UID: "0f2070a8-8f2b-4673-a431-b3640f77d332"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:52:23.518772 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:23.518743    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f2070a8-8f2b-4673-a431-b3640f77d332-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f2070a8-8f2b-4673-a431-b3640f77d332-workload-certs\") pod \"0f2070a8-8f2b-4673-a431-b3640f77d332\" (UID: \"0f2070a8-8f2b-4673-a431-b3640f77d332\") "
Oct 30 07:52:23.519127 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:23.519029    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f2070a8-8f2b-4673-a431-b3640f77d332-workload-certs" pod "0f2070a8-8f2b-4673-a431-b3640f77d332" (UID: "0f2070a8-8f2b-4673-a431-b3640f77d332"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:52:23.519127 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:23.519095    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f2070a8-8f2b-4673-a431-b3640f77d332-credential-socket" pod "0f2070a8-8f2b-4673-a431-b3640f77d332" (UID: "0f2070a8-8f2b-4673-a431-b3640f77d332"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:52:23.519462 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:23.519435    6224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f2070a8-8f2b-4673-a431-b3640f77d332-istiod-ca-cert\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:52:23.519563 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:23.519464    6224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f2070a8-8f2b-4673-a431-b3640f77d332-istio-data\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:52:23.519563 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:23.519478    6224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f2070a8-8f2b-4673-a431-b3640f77d332-workload-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:52:23.519563 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:23.519491    6224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f2070a8-8f2b-4673-a431-b3640f77d332-credential-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:52:23.519563 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:23.519506    6224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f2070a8-8f2b-4673-a431-b3640f77d332-workload-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:52:23.521126 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:23.521088    6224 generic.go:356] "Generic (PLEG): container finished" podID="0f2070a8-8f2b-4673-a431-b3640f77d332" containerID="652a781cfc9622e2da68ceb37967d8d7cf520aa7d538faf1c26dc2ac46291958" exitCode=0
Oct 30 07:52:23.521234 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:23.521130    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4" event={"ID":"0f2070a8-8f2b-4673-a431-b3640f77d332","Type":"ContainerDied","Data":"652a781cfc9622e2da68ceb37967d8d7cf520aa7d538faf1c26dc2ac46291958"}
Oct 30 07:52:23.521234 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:23.521188    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4" event={"ID":"0f2070a8-8f2b-4673-a431-b3640f77d332","Type":"ContainerDied","Data":"bca8de4366804f503894537b316f7d36882854664eb7d39b33c93d692c5e1647"}
Oct 30 07:52:23.521234 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:23.521194    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4"
Oct 30 07:52:23.521234 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:23.521217    6224 scope.go:122] "RemoveContainer" containerID="652a781cfc9622e2da68ceb37967d8d7cf520aa7d538faf1c26dc2ac46291958"
Oct 30 07:52:23.521435 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:23.521255    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f2070a8-8f2b-4673-a431-b3640f77d332-istio-envoy" pod "0f2070a8-8f2b-4673-a431-b3640f77d332" (UID: "0f2070a8-8f2b-4673-a431-b3640f77d332"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:52:23.521600 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:23.521577    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f2070a8-8f2b-4673-a431-b3640f77d332-istio-token" pod "0f2070a8-8f2b-4673-a431-b3640f77d332" (UID: "0f2070a8-8f2b-4673-a431-b3640f77d332"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:52:23.521738 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:23.521717    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f2070a8-8f2b-4673-a431-b3640f77d332-istio-podinfo" pod "0f2070a8-8f2b-4673-a431-b3640f77d332" (UID: "0f2070a8-8f2b-4673-a431-b3640f77d332"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:52:23.522291 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:23.522264    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f2070a8-8f2b-4673-a431-b3640f77d332-kube-api-access-68qk5" pod "0f2070a8-8f2b-4673-a431-b3640f77d332" (UID: "0f2070a8-8f2b-4673-a431-b3640f77d332"). InnerVolumeSpecName "kube-api-access-68qk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:52:23.564992 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:23.564966    6224 scope.go:122] "RemoveContainer" containerID="652a781cfc9622e2da68ceb37967d8d7cf520aa7d538faf1c26dc2ac46291958"
Oct 30 07:52:23.565361 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:52:23.565336    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"652a781cfc9622e2da68ceb37967d8d7cf520aa7d538faf1c26dc2ac46291958\": container with ID starting with 652a781cfc9622e2da68ceb37967d8d7cf520aa7d538faf1c26dc2ac46291958 not found: ID does not exist" containerID="652a781cfc9622e2da68ceb37967d8d7cf520aa7d538faf1c26dc2ac46291958"
Oct 30 07:52:23.565466 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:23.565368    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"652a781cfc9622e2da68ceb37967d8d7cf520aa7d538faf1c26dc2ac46291958"} err="failed to get container status \"652a781cfc9622e2da68ceb37967d8d7cf520aa7d538faf1c26dc2ac46291958\": rpc error: code = NotFound desc = could not find container \"652a781cfc9622e2da68ceb37967d8d7cf520aa7d538faf1c26dc2ac46291958\": container with ID starting with 652a781cfc9622e2da68ceb37967d8d7cf520aa7d538faf1c26dc2ac46291958 not found: ID does not exist"
Oct 30 07:52:23.620457 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:23.620423    6224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f2070a8-8f2b-4673-a431-b3640f77d332-istio-envoy\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:52:23.620457 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:23.620457    6224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f2070a8-8f2b-4673-a431-b3640f77d332-istio-podinfo\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:52:23.620669 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:23.620468    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68qk5\" (UniqueName: \"kubernetes.io/projected/0f2070a8-8f2b-4673-a431-b3640f77d332-kube-api-access-68qk5\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:52:23.620669 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:23.620479    6224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f2070a8-8f2b-4673-a431-b3640f77d332-istio-token\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:52:23.881247 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:23.881215    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4"]
Oct 30 07:52:23.895073 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:23.895046    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-4ab37e-openshift-default-7999c86545-sgct4"]
Oct 30 07:52:25.507769 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:25.507725    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f2070a8-8f2b-4673-a431-b3640f77d332" path="/var/lib/kubelet/pods/0f2070a8-8f2b-4673-a431-b3640f77d332/volumes"
Oct 30 07:52:26.309368 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:26.309306    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm"
Oct 30 07:52:26.356855 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:26.356826    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68dd39f6-f527-431d-94d4-52a6101d08de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68dd39f6-f527-431d-94d4-52a6101d08de-istiod-ca-cert\") pod \"68dd39f6-f527-431d-94d4-52a6101d08de\" (UID: \"68dd39f6-f527-431d-94d4-52a6101d08de\") "
Oct 30 07:52:26.357008 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:26.356921    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/68dd39f6-f527-431d-94d4-52a6101d08de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68dd39f6-f527-431d-94d4-52a6101d08de-istio-podinfo\") pod \"68dd39f6-f527-431d-94d4-52a6101d08de\" (UID: \"68dd39f6-f527-431d-94d4-52a6101d08de\") "
Oct 30 07:52:26.357008 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:26.356963    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68dd39f6-f527-431d-94d4-52a6101d08de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68dd39f6-f527-431d-94d4-52a6101d08de-istio-data\") pod \"68dd39f6-f527-431d-94d4-52a6101d08de\" (UID: \"68dd39f6-f527-431d-94d4-52a6101d08de\") "
Oct 30 07:52:26.357154 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:26.357005    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68dd39f6-f527-431d-94d4-52a6101d08de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68dd39f6-f527-431d-94d4-52a6101d08de-credential-socket\") pod \"68dd39f6-f527-431d-94d4-52a6101d08de\" (UID: \"68dd39f6-f527-431d-94d4-52a6101d08de\") "
Oct 30 07:52:26.357154 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:26.357050    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68dd39f6-f527-431d-94d4-52a6101d08de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68dd39f6-f527-431d-94d4-52a6101d08de-istio-envoy\") pod \"68dd39f6-f527-431d-94d4-52a6101d08de\" (UID: \"68dd39f6-f527-431d-94d4-52a6101d08de\") "
Oct 30 07:52:26.357154 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:26.357149    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68dd39f6-f527-431d-94d4-52a6101d08de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68dd39f6-f527-431d-94d4-52a6101d08de-workload-socket\") pod \"68dd39f6-f527-431d-94d4-52a6101d08de\" (UID: \"68dd39f6-f527-431d-94d4-52a6101d08de\") "
Oct 30 07:52:26.357320 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:26.357205    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68dd39f6-f527-431d-94d4-52a6101d08de-istiod-ca-cert" pod "68dd39f6-f527-431d-94d4-52a6101d08de" (UID: "68dd39f6-f527-431d-94d4-52a6101d08de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:52:26.357320 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:26.357289    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68dd39f6-f527-431d-94d4-52a6101d08de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68dd39f6-f527-431d-94d4-52a6101d08de-workload-certs\") pod \"68dd39f6-f527-431d-94d4-52a6101d08de\" (UID: \"68dd39f6-f527-431d-94d4-52a6101d08de\") "
Oct 30 07:52:26.357439 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:26.357348    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68dd39f6-f527-431d-94d4-52a6101d08de-istio-token\" (UniqueName: \"kubernetes.io/projected/68dd39f6-f527-431d-94d4-52a6101d08de-istio-token\") pod \"68dd39f6-f527-431d-94d4-52a6101d08de\" (UID: \"68dd39f6-f527-431d-94d4-52a6101d08de\") "
Oct 30 07:52:26.357439 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:26.357380    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68dd39f6-f527-431d-94d4-52a6101d08de-kube-api-access-fdt5z\" (UniqueName: \"kubernetes.io/projected/68dd39f6-f527-431d-94d4-52a6101d08de-kube-api-access-fdt5z\") pod \"68dd39f6-f527-431d-94d4-52a6101d08de\" (UID: \"68dd39f6-f527-431d-94d4-52a6101d08de\") "
Oct 30 07:52:26.357439 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:26.357387    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68dd39f6-f527-431d-94d4-52a6101d08de-credential-socket" pod "68dd39f6-f527-431d-94d4-52a6101d08de" (UID: "68dd39f6-f527-431d-94d4-52a6101d08de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:52:26.357767 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:26.357684    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68dd39f6-f527-431d-94d4-52a6101d08de-workload-certs" pod "68dd39f6-f527-431d-94d4-52a6101d08de" (UID: "68dd39f6-f527-431d-94d4-52a6101d08de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:52:26.357767 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:26.357704    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68dd39f6-f527-431d-94d4-52a6101d08de-workload-socket" pod "68dd39f6-f527-431d-94d4-52a6101d08de" (UID: "68dd39f6-f527-431d-94d4-52a6101d08de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:52:26.357938 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:26.357862    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68dd39f6-f527-431d-94d4-52a6101d08de-istio-data" pod "68dd39f6-f527-431d-94d4-52a6101d08de" (UID: "68dd39f6-f527-431d-94d4-52a6101d08de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:52:26.358102 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:26.358062    6224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68dd39f6-f527-431d-94d4-52a6101d08de-istiod-ca-cert\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:52:26.358269 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:26.358090    6224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68dd39f6-f527-431d-94d4-52a6101d08de-istio-data\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:52:26.358269 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:26.358158    6224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68dd39f6-f527-431d-94d4-52a6101d08de-credential-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:52:26.358269 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:26.358174    6224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68dd39f6-f527-431d-94d4-52a6101d08de-workload-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:52:26.358269 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:26.358212    6224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68dd39f6-f527-431d-94d4-52a6101d08de-workload-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:52:26.359900 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:26.359874    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/68dd39f6-f527-431d-94d4-52a6101d08de-istio-podinfo" pod "68dd39f6-f527-431d-94d4-52a6101d08de" (UID: "68dd39f6-f527-431d-94d4-52a6101d08de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:52:26.359900 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:26.359885    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68dd39f6-f527-431d-94d4-52a6101d08de-istio-envoy" pod "68dd39f6-f527-431d-94d4-52a6101d08de" (UID: "68dd39f6-f527-431d-94d4-52a6101d08de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:52:26.360197 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:26.360161    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68dd39f6-f527-431d-94d4-52a6101d08de-kube-api-access-fdt5z" pod "68dd39f6-f527-431d-94d4-52a6101d08de" (UID: "68dd39f6-f527-431d-94d4-52a6101d08de"). InnerVolumeSpecName "kube-api-access-fdt5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:52:26.360463 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:26.360443    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68dd39f6-f527-431d-94d4-52a6101d08de-istio-token" pod "68dd39f6-f527-431d-94d4-52a6101d08de" (UID: "68dd39f6-f527-431d-94d4-52a6101d08de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:52:26.459293 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:26.459257    6224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68dd39f6-f527-431d-94d4-52a6101d08de-istio-token\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:52:26.459293 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:26.459287    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdt5z\" (UniqueName: \"kubernetes.io/projected/68dd39f6-f527-431d-94d4-52a6101d08de-kube-api-access-fdt5z\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:52:26.459293 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:26.459298    6224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68dd39f6-f527-431d-94d4-52a6101d08de-istio-podinfo\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:52:26.459537 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:26.459306    6224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68dd39f6-f527-431d-94d4-52a6101d08de-istio-envoy\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:52:26.548607 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:26.548570    6224 generic.go:356] "Generic (PLEG): container finished" podID="68dd39f6-f527-431d-94d4-52a6101d08de" containerID="9bf9195a84314feb2f56ae23b330b25f169c170076cfa5a010775827cedde575" exitCode=0
Oct 30 07:52:26.549112 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:26.548710    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm"
Oct 30 07:52:26.549112 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:26.548725    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm" event={"ID":"68dd39f6-f527-431d-94d4-52a6101d08de","Type":"ContainerDied","Data":"9bf9195a84314feb2f56ae23b330b25f169c170076cfa5a010775827cedde575"}
Oct 30 07:52:26.549112 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:26.548877    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm" event={"ID":"68dd39f6-f527-431d-94d4-52a6101d08de","Type":"ContainerDied","Data":"070b42e2a384314fd3b3c3127e9c7fb4e381d69db541cfaadc159187ecc49303"}
Oct 30 07:52:26.549112 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:26.548912    6224 scope.go:122] "RemoveContainer" containerID="9bf9195a84314feb2f56ae23b330b25f169c170076cfa5a010775827cedde575"
Oct 30 07:52:26.564254 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:26.564230    6224 scope.go:122] "RemoveContainer" containerID="9bf9195a84314feb2f56ae23b330b25f169c170076cfa5a010775827cedde575"
Oct 30 07:52:26.564640 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:52:26.564605    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bf9195a84314feb2f56ae23b330b25f169c170076cfa5a010775827cedde575\": container with ID starting with 9bf9195a84314feb2f56ae23b330b25f169c170076cfa5a010775827cedde575 not found: ID does not exist" containerID="9bf9195a84314feb2f56ae23b330b25f169c170076cfa5a010775827cedde575"
Oct 30 07:52:26.564742 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:26.564643    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bf9195a84314feb2f56ae23b330b25f169c170076cfa5a010775827cedde575"} err="failed to get container status \"9bf9195a84314feb2f56ae23b330b25f169c170076cfa5a010775827cedde575\": rpc error: code = NotFound desc = could not find container \"9bf9195a84314feb2f56ae23b330b25f169c170076cfa5a010775827cedde575\": container with ID starting with 9bf9195a84314feb2f56ae23b330b25f169c170076cfa5a010775827cedde575 not found: ID does not exist"
Oct 30 07:52:26.608140 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:26.608118    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm"]
Oct 30 07:52:26.615204 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:26.615172    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2eda2c-openshift-default-56cb7f654f-kqkfm"]
Oct 30 07:52:27.289859 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:27.289780    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-f480c6/test-model-f480c6-kserve-7bbd97ff6f-75xld" podUID="2ffa4b03-7acf-4a5c-9045-59f171bb9bfa" containerName="main" containerID="cri-o://3449da77d4786fc38c0f9ae2bd5ffe75c5c1a5133510244a7989b4c984ca8974" gracePeriod=45
Oct 30 07:52:27.511255 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:27.511218    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68dd39f6-f527-431d-94d4-52a6101d08de" path="/var/lib/kubelet/pods/68dd39f6-f527-431d-94d4-52a6101d08de/volumes"
Oct 30 07:52:27.559317 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:27.559288    6224 generic.go:356] "Generic (PLEG): container finished" podID="2ffa4b03-7acf-4a5c-9045-59f171bb9bfa" containerID="3449da77d4786fc38c0f9ae2bd5ffe75c5c1a5133510244a7989b4c984ca8974" exitCode=0
Oct 30 07:52:27.559643 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:27.559352    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f480c6/test-model-f480c6-kserve-7bbd97ff6f-75xld" event={"ID":"2ffa4b03-7acf-4a5c-9045-59f171bb9bfa","Type":"ContainerDied","Data":"3449da77d4786fc38c0f9ae2bd5ffe75c5c1a5133510244a7989b4c984ca8974"}
Oct 30 07:52:27.559643 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:27.559390    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f480c6/test-model-f480c6-kserve-7bbd97ff6f-75xld" event={"ID":"2ffa4b03-7acf-4a5c-9045-59f171bb9bfa","Type":"ContainerDied","Data":"3c25a12514fa57986a6be01f53ce41305c9e761e57b77707e30a32dae69e7143"}
Oct 30 07:52:27.559643 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:27.559403    6224 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c25a12514fa57986a6be01f53ce41305c9e761e57b77707e30a32dae69e7143"
Oct 30 07:52:27.569730 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:27.569703    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f480c6/test-model-f480c6-kserve-7bbd97ff6f-75xld"
Oct 30 07:52:27.674976 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:27.674946    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ffa4b03-7acf-4a5c-9045-59f171bb9bfa-home\" (UniqueName: \"kubernetes.io/empty-dir/2ffa4b03-7acf-4a5c-9045-59f171bb9bfa-home\") pod \"2ffa4b03-7acf-4a5c-9045-59f171bb9bfa\" (UID: \"2ffa4b03-7acf-4a5c-9045-59f171bb9bfa\") "
Oct 30 07:52:27.675166 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:27.674982    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ffa4b03-7acf-4a5c-9045-59f171bb9bfa-tls-certs\" (UniqueName: \"kubernetes.io/secret/2ffa4b03-7acf-4a5c-9045-59f171bb9bfa-tls-certs\") pod \"2ffa4b03-7acf-4a5c-9045-59f171bb9bfa\" (UID: \"2ffa4b03-7acf-4a5c-9045-59f171bb9bfa\") "
Oct 30 07:52:27.675166 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:27.675031    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ffa4b03-7acf-4a5c-9045-59f171bb9bfa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ffa4b03-7acf-4a5c-9045-59f171bb9bfa-model-cache\") pod \"2ffa4b03-7acf-4a5c-9045-59f171bb9bfa\" (UID: \"2ffa4b03-7acf-4a5c-9045-59f171bb9bfa\") "
Oct 30 07:52:27.675166 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:27.675080    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ffa4b03-7acf-4a5c-9045-59f171bb9bfa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ffa4b03-7acf-4a5c-9045-59f171bb9bfa-dshm\") pod \"2ffa4b03-7acf-4a5c-9045-59f171bb9bfa\" (UID: \"2ffa4b03-7acf-4a5c-9045-59f171bb9bfa\") "
Oct 30 07:52:27.675166 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:27.675147    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ffa4b03-7acf-4a5c-9045-59f171bb9bfa-kube-api-access-7gl9p\" (UniqueName: \"kubernetes.io/projected/2ffa4b03-7acf-4a5c-9045-59f171bb9bfa-kube-api-access-7gl9p\") pod \"2ffa4b03-7acf-4a5c-9045-59f171bb9bfa\" (UID: \"2ffa4b03-7acf-4a5c-9045-59f171bb9bfa\") "
Oct 30 07:52:27.675400 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:27.675187    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ffa4b03-7acf-4a5c-9045-59f171bb9bfa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ffa4b03-7acf-4a5c-9045-59f171bb9bfa-tmp-dir\") pod \"2ffa4b03-7acf-4a5c-9045-59f171bb9bfa\" (UID: \"2ffa4b03-7acf-4a5c-9045-59f171bb9bfa\") "
Oct 30 07:52:27.675400 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:27.675220    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ffa4b03-7acf-4a5c-9045-59f171bb9bfa-home" pod "2ffa4b03-7acf-4a5c-9045-59f171bb9bfa" (UID: "2ffa4b03-7acf-4a5c-9045-59f171bb9bfa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:52:27.675400 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:27.675335    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ffa4b03-7acf-4a5c-9045-59f171bb9bfa-model-cache" pod "2ffa4b03-7acf-4a5c-9045-59f171bb9bfa" (UID: "2ffa4b03-7acf-4a5c-9045-59f171bb9bfa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:52:27.675543 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:27.675494    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ffa4b03-7acf-4a5c-9045-59f171bb9bfa-tmp-dir" pod "2ffa4b03-7acf-4a5c-9045-59f171bb9bfa" (UID: "2ffa4b03-7acf-4a5c-9045-59f171bb9bfa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:52:27.675770 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:27.675749    6224 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ffa4b03-7acf-4a5c-9045-59f171bb9bfa-home\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:52:27.675770 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:27.675771    6224 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ffa4b03-7acf-4a5c-9045-59f171bb9bfa-model-cache\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:52:27.675982 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:27.675784    6224 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ffa4b03-7acf-4a5c-9045-59f171bb9bfa-tmp-dir\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:52:27.677401 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:27.677374    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ffa4b03-7acf-4a5c-9045-59f171bb9bfa-dshm" pod "2ffa4b03-7acf-4a5c-9045-59f171bb9bfa" (UID: "2ffa4b03-7acf-4a5c-9045-59f171bb9bfa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:52:27.677487 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:27.677462    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ffa4b03-7acf-4a5c-9045-59f171bb9bfa-tls-certs" pod "2ffa4b03-7acf-4a5c-9045-59f171bb9bfa" (UID: "2ffa4b03-7acf-4a5c-9045-59f171bb9bfa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:52:27.677487 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:27.677461    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ffa4b03-7acf-4a5c-9045-59f171bb9bfa-kube-api-access-7gl9p" pod "2ffa4b03-7acf-4a5c-9045-59f171bb9bfa" (UID: "2ffa4b03-7acf-4a5c-9045-59f171bb9bfa"). InnerVolumeSpecName "kube-api-access-7gl9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:52:27.777163 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:27.777127    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gl9p\" (UniqueName: \"kubernetes.io/projected/2ffa4b03-7acf-4a5c-9045-59f171bb9bfa-kube-api-access-7gl9p\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:52:27.777163 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:27.777156    6224 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ffa4b03-7acf-4a5c-9045-59f171bb9bfa-tls-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:52:27.777328 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:27.777171    6224 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ffa4b03-7acf-4a5c-9045-59f171bb9bfa-dshm\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:52:27.974078 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:27.973995    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-8fef73-656cd4fdb-st5tl" podUID="1edc6428-e830-42ed-868b-bdb7079e0c33" containerName="payload-processing" containerID="cri-o://aa9a43ab9c21ba0788735277a5dd16d77dab644b59870016a797d2465391dd03" gracePeriod=25
Oct 30 07:52:28.008341 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:28.008301    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8fef73-99b947487-lf4k2" podUID="0df77ed5-4c16-40d9-9f1c-6dba3150a7e6" containerName="payload-pre-processing" containerID="cri-o://fc3630ab64d4fa5cbb0cd0e4c9d4d2515c46aa50e703a1a6e91dcc798a30a377" gracePeriod=25
Oct 30 07:52:28.567351 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:28.567317    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f480c6/test-model-f480c6-kserve-7bbd97ff6f-75xld"
Oct 30 07:52:28.626916 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:28.626887    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f480c6/test-model-f480c6-kserve-7bbd97ff6f-75xld"]
Oct 30 07:52:28.639759 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:28.639730    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-f480c6/test-model-f480c6-kserve-7bbd97ff6f-75xld"]
Oct 30 07:52:29.505130 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:29.505091    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ffa4b03-7acf-4a5c-9045-59f171bb9bfa" path="/var/lib/kubelet/pods/2ffa4b03-7acf-4a5c-9045-59f171bb9bfa/volumes"
Oct 30 07:52:31.860905 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:52:31.860862    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:52:32.015198 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:32.015163    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8fef73-99b947487-lf4k2" podUID="0df77ed5-4c16-40d9-9f1c-6dba3150a7e6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 07:52:32.015397 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:32.015226    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-8fef73-656cd4fdb-st5tl" podUID="1edc6428-e830-42ed-868b-bdb7079e0c33" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 07:52:42.014505 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:42.014470    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-8fef73-656cd4fdb-st5tl" podUID="1edc6428-e830-42ed-868b-bdb7079e0c33" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 07:52:42.014973 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:42.014824    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8fef73-99b947487-lf4k2" podUID="0df77ed5-4c16-40d9-9f1c-6dba3150a7e6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 07:52:48.299108 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:48.299077    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8fef73-656cd4fdb-st5tl"
Oct 30 07:52:48.344750 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:48.344721    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1edc6428-e830-42ed-868b-bdb7079e0c33-kube-api-access-btrpr\" (UniqueName: \"kubernetes.io/projected/1edc6428-e830-42ed-868b-bdb7079e0c33-kube-api-access-btrpr\") pod \"1edc6428-e830-42ed-868b-bdb7079e0c33\" (UID: \"1edc6428-e830-42ed-868b-bdb7079e0c33\") "
Oct 30 07:52:48.345020 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:48.344997    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1edc6428-e830-42ed-868b-bdb7079e0c33-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1edc6428-e830-42ed-868b-bdb7079e0c33-plugins-config-volume\") pod \"1edc6428-e830-42ed-868b-bdb7079e0c33\" (UID: \"1edc6428-e830-42ed-868b-bdb7079e0c33\") "
Oct 30 07:52:48.345368 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:48.345332    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1edc6428-e830-42ed-868b-bdb7079e0c33-plugins-config-volume" pod "1edc6428-e830-42ed-868b-bdb7079e0c33" (UID: "1edc6428-e830-42ed-868b-bdb7079e0c33"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:52:48.345614 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:48.345596    6224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1edc6428-e830-42ed-868b-bdb7079e0c33-plugins-config-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:52:48.346929 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:48.346908    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1edc6428-e830-42ed-868b-bdb7079e0c33-kube-api-access-btrpr" pod "1edc6428-e830-42ed-868b-bdb7079e0c33" (UID: "1edc6428-e830-42ed-868b-bdb7079e0c33"). InnerVolumeSpecName "kube-api-access-btrpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:52:48.353323 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:48.353294    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8fef73-99b947487-lf4k2"
Oct 30 07:52:48.446046 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:48.446003    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0df77ed5-4c16-40d9-9f1c-6dba3150a7e6-kube-api-access-slhhj\" (UniqueName: \"kubernetes.io/projected/0df77ed5-4c16-40d9-9f1c-6dba3150a7e6-kube-api-access-slhhj\") pod \"0df77ed5-4c16-40d9-9f1c-6dba3150a7e6\" (UID: \"0df77ed5-4c16-40d9-9f1c-6dba3150a7e6\") "
Oct 30 07:52:48.446245 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:48.446166    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0df77ed5-4c16-40d9-9f1c-6dba3150a7e6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0df77ed5-4c16-40d9-9f1c-6dba3150a7e6-plugins-config-volume\") pod \"0df77ed5-4c16-40d9-9f1c-6dba3150a7e6\" (UID: \"0df77ed5-4c16-40d9-9f1c-6dba3150a7e6\") "
Oct 30 07:52:48.446618 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:48.446587    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0df77ed5-4c16-40d9-9f1c-6dba3150a7e6-plugins-config-volume" pod "0df77ed5-4c16-40d9-9f1c-6dba3150a7e6" (UID: "0df77ed5-4c16-40d9-9f1c-6dba3150a7e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:52:48.446759 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:48.446738    6224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0df77ed5-4c16-40d9-9f1c-6dba3150a7e6-plugins-config-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:52:48.446852 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:48.446759    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btrpr\" (UniqueName: \"kubernetes.io/projected/1edc6428-e830-42ed-868b-bdb7079e0c33-kube-api-access-btrpr\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:52:48.448282 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:48.448261    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0df77ed5-4c16-40d9-9f1c-6dba3150a7e6-kube-api-access-slhhj" pod "0df77ed5-4c16-40d9-9f1c-6dba3150a7e6" (UID: "0df77ed5-4c16-40d9-9f1c-6dba3150a7e6"). InnerVolumeSpecName "kube-api-access-slhhj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:52:48.548511 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:48.548475    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slhhj\" (UniqueName: \"kubernetes.io/projected/0df77ed5-4c16-40d9-9f1c-6dba3150a7e6-kube-api-access-slhhj\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:52:48.716570 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:48.716533    6224 generic.go:356] "Generic (PLEG): container finished" podID="1edc6428-e830-42ed-868b-bdb7079e0c33" containerID="aa9a43ab9c21ba0788735277a5dd16d77dab644b59870016a797d2465391dd03" exitCode=0
Oct 30 07:52:48.716779 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:48.716592    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8fef73-656cd4fdb-st5tl"
Oct 30 07:52:48.716779 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:48.716596    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8fef73-656cd4fdb-st5tl" event={"ID":"1edc6428-e830-42ed-868b-bdb7079e0c33","Type":"ContainerDied","Data":"aa9a43ab9c21ba0788735277a5dd16d77dab644b59870016a797d2465391dd03"}
Oct 30 07:52:48.716779 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:48.716655    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8fef73-656cd4fdb-st5tl" event={"ID":"1edc6428-e830-42ed-868b-bdb7079e0c33","Type":"ContainerDied","Data":"d2535108df63eed5787745cf5b20aca3283526b17ad7f460bf69d1191fa7d469"}
Oct 30 07:52:48.716779 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:48.716688    6224 scope.go:122] "RemoveContainer" containerID="aa9a43ab9c21ba0788735277a5dd16d77dab644b59870016a797d2465391dd03"
Oct 30 07:52:48.718444 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:48.718421    6224 generic.go:356] "Generic (PLEG): container finished" podID="0df77ed5-4c16-40d9-9f1c-6dba3150a7e6" containerID="fc3630ab64d4fa5cbb0cd0e4c9d4d2515c46aa50e703a1a6e91dcc798a30a377" exitCode=0
Oct 30 07:52:48.718670 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:48.718470    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8fef73-99b947487-lf4k2" event={"ID":"0df77ed5-4c16-40d9-9f1c-6dba3150a7e6","Type":"ContainerDied","Data":"fc3630ab64d4fa5cbb0cd0e4c9d4d2515c46aa50e703a1a6e91dcc798a30a377"}
Oct 30 07:52:48.718670 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:48.718479    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8fef73-99b947487-lf4k2"
Oct 30 07:52:48.718670 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:48.718507    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8fef73-99b947487-lf4k2" event={"ID":"0df77ed5-4c16-40d9-9f1c-6dba3150a7e6","Type":"ContainerDied","Data":"93a4b2b264a0092f178af267a606f7ff1c1a270953d07993d12fb355eba125a0"}
Oct 30 07:52:48.728507 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:48.728479    6224 scope.go:122] "RemoveContainer" containerID="aa9a43ab9c21ba0788735277a5dd16d77dab644b59870016a797d2465391dd03"
Oct 30 07:52:48.728787 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:52:48.728766    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa9a43ab9c21ba0788735277a5dd16d77dab644b59870016a797d2465391dd03\": container with ID starting with aa9a43ab9c21ba0788735277a5dd16d77dab644b59870016a797d2465391dd03 not found: ID does not exist" containerID="aa9a43ab9c21ba0788735277a5dd16d77dab644b59870016a797d2465391dd03"
Oct 30 07:52:48.728921 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:48.728794    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa9a43ab9c21ba0788735277a5dd16d77dab644b59870016a797d2465391dd03"} err="failed to get container status \"aa9a43ab9c21ba0788735277a5dd16d77dab644b59870016a797d2465391dd03\": rpc error: code = NotFound desc = could not find container \"aa9a43ab9c21ba0788735277a5dd16d77dab644b59870016a797d2465391dd03\": container with ID starting with aa9a43ab9c21ba0788735277a5dd16d77dab644b59870016a797d2465391dd03 not found: ID does not exist"
Oct 30 07:52:48.728921 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:48.728828    6224 scope.go:122] "RemoveContainer" containerID="fc3630ab64d4fa5cbb0cd0e4c9d4d2515c46aa50e703a1a6e91dcc798a30a377"
Oct 30 07:52:48.739109 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:48.739089    6224 scope.go:122] "RemoveContainer" containerID="fc3630ab64d4fa5cbb0cd0e4c9d4d2515c46aa50e703a1a6e91dcc798a30a377"
Oct 30 07:52:48.739381 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:52:48.739361    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc3630ab64d4fa5cbb0cd0e4c9d4d2515c46aa50e703a1a6e91dcc798a30a377\": container with ID starting with fc3630ab64d4fa5cbb0cd0e4c9d4d2515c46aa50e703a1a6e91dcc798a30a377 not found: ID does not exist" containerID="fc3630ab64d4fa5cbb0cd0e4c9d4d2515c46aa50e703a1a6e91dcc798a30a377"
Oct 30 07:52:48.739445 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:48.739390    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc3630ab64d4fa5cbb0cd0e4c9d4d2515c46aa50e703a1a6e91dcc798a30a377"} err="failed to get container status \"fc3630ab64d4fa5cbb0cd0e4c9d4d2515c46aa50e703a1a6e91dcc798a30a377\": rpc error: code = NotFound desc = could not find container \"fc3630ab64d4fa5cbb0cd0e4c9d4d2515c46aa50e703a1a6e91dcc798a30a377\": container with ID starting with fc3630ab64d4fa5cbb0cd0e4c9d4d2515c46aa50e703a1a6e91dcc798a30a377 not found: ID does not exist"
Oct 30 07:52:48.781142 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:48.781104    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8fef73-656cd4fdb-st5tl"]
Oct 30 07:52:48.786227 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:48.786202    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8fef73-656cd4fdb-st5tl"]
Oct 30 07:52:48.847848 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:48.847763    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8fef73-99b947487-lf4k2"]
Oct 30 07:52:48.851263 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:48.851241    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8fef73-99b947487-lf4k2"]
Oct 30 07:52:49.506057 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:49.506025    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0df77ed5-4c16-40d9-9f1c-6dba3150a7e6" path="/var/lib/kubelet/pods/0df77ed5-4c16-40d9-9f1c-6dba3150a7e6/volumes"
Oct 30 07:52:49.506478 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:49.506467    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1edc6428-e830-42ed-868b-bdb7079e0c33" path="/var/lib/kubelet/pods/1edc6428-e830-42ed-868b-bdb7079e0c33/volumes"
Oct 30 07:52:59.416773 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:52:59.416738    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:53:01.857873 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:53:01.857834    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:53:15.840649 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:15.840572    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-54xv2"]
Oct 30 07:53:15.841365 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:15.841346    6224 cpu_manager.go:414] "containerMap: removing container" podUID="0df77ed5-4c16-40d9-9f1c-6dba3150a7e6" containerName="payload-pre-processing"
Oct 30 07:53:15.841420 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:15.841366    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0df77ed5-4c16-40d9-9f1c-6dba3150a7e6" containerName="payload-pre-processing"
Oct 30 07:53:15.841420 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:15.841382    6224 cpu_manager.go:414] "containerMap: removing container" podUID="2ffa4b03-7acf-4a5c-9045-59f171bb9bfa" containerName="main"
Oct 30 07:53:15.841420 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:15.841389    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ffa4b03-7acf-4a5c-9045-59f171bb9bfa" containerName="main"
Oct 30 07:53:15.841420 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:15.841413    6224 cpu_manager.go:414] "containerMap: removing container" podUID="0f2070a8-8f2b-4673-a431-b3640f77d332" containerName="istio-proxy"
Oct 30 07:53:15.841420 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:15.841418    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f2070a8-8f2b-4673-a431-b3640f77d332" containerName="istio-proxy"
Oct 30 07:53:15.841589 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:15.841426    6224 cpu_manager.go:414] "containerMap: removing container" podUID="1edc6428-e830-42ed-868b-bdb7079e0c33" containerName="payload-processing"
Oct 30 07:53:15.841589 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:15.841431    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1edc6428-e830-42ed-868b-bdb7079e0c33" containerName="payload-processing"
Oct 30 07:53:15.841589 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:15.841444    6224 cpu_manager.go:414] "containerMap: removing container" podUID="efa670c6-14aa-4c5e-be1c-9976b974bdff" containerName="maas-api"
Oct 30 07:53:15.841589 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:15.841449    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efa670c6-14aa-4c5e-be1c-9976b974bdff" containerName="maas-api"
Oct 30 07:53:15.841589 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:15.841459    6224 cpu_manager.go:414] "containerMap: removing container" podUID="0a0e028c-e9eb-4260-bfa3-37104c26af0e" containerName="istio-proxy"
Oct 30 07:53:15.841589 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:15.841463    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a0e028c-e9eb-4260-bfa3-37104c26af0e" containerName="istio-proxy"
Oct 30 07:53:15.841589 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:15.841471    6224 cpu_manager.go:414] "containerMap: removing container" podUID="02fa3182-1c4a-418e-882b-08db6f71a0a7" containerName="istio-proxy"
Oct 30 07:53:15.841589 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:15.841475    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02fa3182-1c4a-418e-882b-08db6f71a0a7" containerName="istio-proxy"
Oct 30 07:53:15.841589 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:15.841485    6224 cpu_manager.go:414] "containerMap: removing container" podUID="68dd39f6-f527-431d-94d4-52a6101d08de" containerName="istio-proxy"
Oct 30 07:53:15.841589 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:15.841489    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68dd39f6-f527-431d-94d4-52a6101d08de" containerName="istio-proxy"
Oct 30 07:53:15.841589 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:15.841577    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f2070a8-8f2b-4673-a431-b3640f77d332" containerName="istio-proxy"
Oct 30 07:53:15.841589 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:15.841588    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="1edc6428-e830-42ed-868b-bdb7079e0c33" containerName="payload-processing"
Oct 30 07:53:15.841978 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:15.841597    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="efa670c6-14aa-4c5e-be1c-9976b974bdff" containerName="maas-api"
Oct 30 07:53:15.841978 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:15.841605    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a0e028c-e9eb-4260-bfa3-37104c26af0e" containerName="istio-proxy"
Oct 30 07:53:15.841978 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:15.841613    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="02fa3182-1c4a-418e-882b-08db6f71a0a7" containerName="istio-proxy"
Oct 30 07:53:15.841978 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:15.841623    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="68dd39f6-f527-431d-94d4-52a6101d08de" containerName="istio-proxy"
Oct 30 07:53:15.841978 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:15.841635    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="0df77ed5-4c16-40d9-9f1c-6dba3150a7e6" containerName="payload-pre-processing"
Oct 30 07:53:15.841978 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:15.841642    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ffa4b03-7acf-4a5c-9045-59f171bb9bfa" containerName="main"
Oct 30 07:53:15.845594 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:15.845578    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-54xv2"
Oct 30 07:53:15.888933 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:15.888900    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-54xv2"]
Oct 30 07:53:15.911714 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:15.911675    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3586a2-80e5-44b7-8c23-bcf26c3a8147-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-54xv2\" (UID: \"bb3586a2-80e5-44b7-8c23-bcf26c3a8147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-54xv2"
Oct 30 07:53:15.911877 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:15.911741    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb3586a2-80e5-44b7-8c23-bcf26c3a8147-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-54xv2\" (UID: \"bb3586a2-80e5-44b7-8c23-bcf26c3a8147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-54xv2"
Oct 30 07:53:15.911877 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:15.911786    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb3586a2-80e5-44b7-8c23-bcf26c3a8147-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-54xv2\" (UID: \"bb3586a2-80e5-44b7-8c23-bcf26c3a8147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-54xv2"
Oct 30 07:53:15.911996 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:15.911946    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdrb5\" (UniqueName: \"kubernetes.io/projected/bb3586a2-80e5-44b7-8c23-bcf26c3a8147-kube-api-access-mdrb5\") pod \"maas-default-gateway-openshift-default-6dcf757899-54xv2\" (UID: \"bb3586a2-80e5-44b7-8c23-bcf26c3a8147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-54xv2"
Oct 30 07:53:15.912079 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:15.912066    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb3586a2-80e5-44b7-8c23-bcf26c3a8147-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-54xv2\" (UID: \"bb3586a2-80e5-44b7-8c23-bcf26c3a8147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-54xv2"
Oct 30 07:53:15.912129 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:15.912088    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb3586a2-80e5-44b7-8c23-bcf26c3a8147-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-54xv2\" (UID: \"bb3586a2-80e5-44b7-8c23-bcf26c3a8147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-54xv2"
Oct 30 07:53:15.912129 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:15.912124    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3586a2-80e5-44b7-8c23-bcf26c3a8147-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-54xv2\" (UID: \"bb3586a2-80e5-44b7-8c23-bcf26c3a8147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-54xv2"
Oct 30 07:53:15.912297 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:15.912277    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb3586a2-80e5-44b7-8c23-bcf26c3a8147-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-54xv2\" (UID: \"bb3586a2-80e5-44b7-8c23-bcf26c3a8147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-54xv2"
Oct 30 07:53:15.912388 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:15.912372    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb3586a2-80e5-44b7-8c23-bcf26c3a8147-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-54xv2\" (UID: \"bb3586a2-80e5-44b7-8c23-bcf26c3a8147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-54xv2"
Oct 30 07:53:16.014201 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:16.014172    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb3586a2-80e5-44b7-8c23-bcf26c3a8147-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-54xv2\" (UID: \"bb3586a2-80e5-44b7-8c23-bcf26c3a8147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-54xv2"
Oct 30 07:53:16.014392 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:16.014213    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3586a2-80e5-44b7-8c23-bcf26c3a8147-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-54xv2\" (UID: \"bb3586a2-80e5-44b7-8c23-bcf26c3a8147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-54xv2"
Oct 30 07:53:16.014392 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:16.014255    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb3586a2-80e5-44b7-8c23-bcf26c3a8147-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-54xv2\" (UID: \"bb3586a2-80e5-44b7-8c23-bcf26c3a8147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-54xv2"
Oct 30 07:53:16.014392 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:16.014298    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb3586a2-80e5-44b7-8c23-bcf26c3a8147-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-54xv2\" (UID: \"bb3586a2-80e5-44b7-8c23-bcf26c3a8147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-54xv2"
Oct 30 07:53:16.014392 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:16.014347    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3586a2-80e5-44b7-8c23-bcf26c3a8147-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-54xv2\" (UID: \"bb3586a2-80e5-44b7-8c23-bcf26c3a8147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-54xv2"
Oct 30 07:53:16.014392 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:16.014373    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb3586a2-80e5-44b7-8c23-bcf26c3a8147-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-54xv2\" (UID: \"bb3586a2-80e5-44b7-8c23-bcf26c3a8147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-54xv2"
Oct 30 07:53:16.014392 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:16.014387    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb3586a2-80e5-44b7-8c23-bcf26c3a8147-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-54xv2\" (UID: \"bb3586a2-80e5-44b7-8c23-bcf26c3a8147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-54xv2"
Oct 30 07:53:16.014749 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:16.014439    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdrb5\" (UniqueName: \"kubernetes.io/projected/bb3586a2-80e5-44b7-8c23-bcf26c3a8147-kube-api-access-mdrb5\") pod \"maas-default-gateway-openshift-default-6dcf757899-54xv2\" (UID: \"bb3586a2-80e5-44b7-8c23-bcf26c3a8147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-54xv2"
Oct 30 07:53:16.014749 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:16.014563    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb3586a2-80e5-44b7-8c23-bcf26c3a8147-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-54xv2\" (UID: \"bb3586a2-80e5-44b7-8c23-bcf26c3a8147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-54xv2"
Oct 30 07:53:16.014749 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:16.014649    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb3586a2-80e5-44b7-8c23-bcf26c3a8147-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-54xv2\" (UID: \"bb3586a2-80e5-44b7-8c23-bcf26c3a8147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-54xv2"
Oct 30 07:53:16.014749 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:16.014695    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3586a2-80e5-44b7-8c23-bcf26c3a8147-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-54xv2\" (UID: \"bb3586a2-80e5-44b7-8c23-bcf26c3a8147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-54xv2"
Oct 30 07:53:16.015066 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:16.015027    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3586a2-80e5-44b7-8c23-bcf26c3a8147-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-54xv2\" (UID: \"bb3586a2-80e5-44b7-8c23-bcf26c3a8147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-54xv2"
Oct 30 07:53:16.015189 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:16.015108    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb3586a2-80e5-44b7-8c23-bcf26c3a8147-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-54xv2\" (UID: \"bb3586a2-80e5-44b7-8c23-bcf26c3a8147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-54xv2"
Oct 30 07:53:16.015482 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:16.015457    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb3586a2-80e5-44b7-8c23-bcf26c3a8147-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-54xv2\" (UID: \"bb3586a2-80e5-44b7-8c23-bcf26c3a8147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-54xv2"
Oct 30 07:53:16.016922 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:16.016901    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb3586a2-80e5-44b7-8c23-bcf26c3a8147-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-54xv2\" (UID: \"bb3586a2-80e5-44b7-8c23-bcf26c3a8147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-54xv2"
Oct 30 07:53:16.017298 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:16.017277    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb3586a2-80e5-44b7-8c23-bcf26c3a8147-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-54xv2\" (UID: \"bb3586a2-80e5-44b7-8c23-bcf26c3a8147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-54xv2"
Oct 30 07:53:16.050889 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:16.050866    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdrb5\" (UniqueName: \"kubernetes.io/projected/bb3586a2-80e5-44b7-8c23-bcf26c3a8147-kube-api-access-mdrb5\") pod \"maas-default-gateway-openshift-default-6dcf757899-54xv2\" (UID: \"bb3586a2-80e5-44b7-8c23-bcf26c3a8147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-54xv2"
Oct 30 07:53:16.053004 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:16.052761    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb3586a2-80e5-44b7-8c23-bcf26c3a8147-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-54xv2\" (UID: \"bb3586a2-80e5-44b7-8c23-bcf26c3a8147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-54xv2"
Oct 30 07:53:16.160282 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:16.160209    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-54xv2"
Oct 30 07:53:16.493424 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:16.493392    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-54xv2"]
Oct 30 07:53:16.493636 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:53:16.493613    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb3586a2_80e5_44b7_8c23_bcf26c3a8147.slice/crio-84e2629232b6d8285e9ca27c5fc21c6687011f36738799c5e37194d5a39074dd WatchSource:0}: Error finding container 84e2629232b6d8285e9ca27c5fc21c6687011f36738799c5e37194d5a39074dd: Status 404 returned error can't find the container with id 84e2629232b6d8285e9ca27c5fc21c6687011f36738799c5e37194d5a39074dd
Oct 30 07:53:16.496635 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:16.496602    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:53:16.496723 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:16.496674    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:53:16.496723 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:16.496702    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:53:16.939602 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:16.939563    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-54xv2" event={"ID":"bb3586a2-80e5-44b7-8c23-bcf26c3a8147","Type":"ContainerStarted","Data":"479382ec07779630bc6b07691d6c931ab608ab847fdf9b998e282c6c9c9f5617"}
Oct 30 07:53:16.940052 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:16.939634    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-54xv2" event={"ID":"bb3586a2-80e5-44b7-8c23-bcf26c3a8147","Type":"ContainerStarted","Data":"84e2629232b6d8285e9ca27c5fc21c6687011f36738799c5e37194d5a39074dd"}
Oct 30 07:53:17.014642 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:17.014588    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-54xv2" podStartSLOduration=2.014574386 podStartE2EDuration="2.014574386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:53:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:53:16.998785934 +0000 UTC m=+3355.608192908" watchObservedRunningTime="2026-10-30 07:53:17.014574386 +0000 UTC m=+3355.623981356"
Oct 30 07:53:17.162247 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:17.162201    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-54xv2"
Oct 30 07:53:17.164473 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:17.164415    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-54xv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body=
Oct 30 07:53:17.164638 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:17.164484    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-54xv2" podUID="bb3586a2-80e5-44b7-8c23-bcf26c3a8147" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused"
Oct 30 07:53:18.160954 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:18.160917    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-54xv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body=
Oct 30 07:53:18.161359 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:18.160970    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-54xv2" podUID="bb3586a2-80e5-44b7-8c23-bcf26c3a8147" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused"
Oct 30 07:53:19.160651 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:19.160619    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-54xv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body=
Oct 30 07:53:19.160848 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:19.160682    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-54xv2" podUID="bb3586a2-80e5-44b7-8c23-bcf26c3a8147" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused"
Oct 30 07:53:20.160605 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:20.160571    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-54xv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body=
Oct 30 07:53:20.161138 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:20.160629    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-54xv2" podUID="bb3586a2-80e5-44b7-8c23-bcf26c3a8147" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused"
Oct 30 07:53:20.820863 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:20.810870    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28"]
Oct 30 07:53:20.820863 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:20.811463    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28" podUID="aa7e606a-884a-4553-943a-15fd296e0a24" containerName="istio-proxy" containerID="cri-o://f7ba6292111a6248b2dc76056f36da9806cfffe69d84620fab4a714f8ba7c3e7" gracePeriod=30
Oct 30 07:53:21.160740 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:21.160657    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-54xv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body=
Oct 30 07:53:21.160740 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:21.160718    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-54xv2" podUID="bb3586a2-80e5-44b7-8c23-bcf26c3a8147" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused"
Oct 30 07:53:22.160775 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:22.160731    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-54xv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body=
Oct 30 07:53:22.161202 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:22.160828    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-54xv2" podUID="bb3586a2-80e5-44b7-8c23-bcf26c3a8147" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused"
Oct 30 07:53:23.161217 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:23.161175    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-54xv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body=
Oct 30 07:53:23.161689 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:23.161251    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-54xv2" podUID="bb3586a2-80e5-44b7-8c23-bcf26c3a8147" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused"
Oct 30 07:53:23.943921 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:23.943890    6224 scope.go:122] "RemoveContainer" containerID="3449da77d4786fc38c0f9ae2bd5ffe75c5c1a5133510244a7989b4c984ca8974"
Oct 30 07:53:24.160980 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:24.160947    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-54xv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body=
Oct 30 07:53:24.161151 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:24.160997    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-54xv2" podUID="bb3586a2-80e5-44b7-8c23-bcf26c3a8147" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused"
Oct 30 07:53:25.166293 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:25.166265    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-54xv2"
Oct 30 07:53:25.166901 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:25.166792    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-54xv2"
Oct 30 07:53:25.167525 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:25.167504    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-54xv2"
Oct 30 07:53:26.052095 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:26.052049    6224 generic.go:356] "Generic (PLEG): container finished" podID="aa7e606a-884a-4553-943a-15fd296e0a24" containerID="f7ba6292111a6248b2dc76056f36da9806cfffe69d84620fab4a714f8ba7c3e7" exitCode=0
Oct 30 07:53:26.052261 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:26.052119    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28" event={"ID":"aa7e606a-884a-4553-943a-15fd296e0a24","Type":"ContainerDied","Data":"f7ba6292111a6248b2dc76056f36da9806cfffe69d84620fab4a714f8ba7c3e7"}
Oct 30 07:53:26.104458 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:26.104396    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28"
Oct 30 07:53:26.177790 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:26.177747    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa7e606a-884a-4553-943a-15fd296e0a24-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa7e606a-884a-4553-943a-15fd296e0a24-istiod-ca-cert\") pod \"aa7e606a-884a-4553-943a-15fd296e0a24\" (UID: \"aa7e606a-884a-4553-943a-15fd296e0a24\") "
Oct 30 07:53:26.178283 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:26.177850    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa7e606a-884a-4553-943a-15fd296e0a24-istio-token\" (UniqueName: \"kubernetes.io/projected/aa7e606a-884a-4553-943a-15fd296e0a24-istio-token\") pod \"aa7e606a-884a-4553-943a-15fd296e0a24\" (UID: \"aa7e606a-884a-4553-943a-15fd296e0a24\") "
Oct 30 07:53:26.178283 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:26.177934    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa7e606a-884a-4553-943a-15fd296e0a24-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa7e606a-884a-4553-943a-15fd296e0a24-credential-socket\") pod \"aa7e606a-884a-4553-943a-15fd296e0a24\" (UID: \"aa7e606a-884a-4553-943a-15fd296e0a24\") "
Oct 30 07:53:26.178283 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:26.177985    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa7e606a-884a-4553-943a-15fd296e0a24-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa7e606a-884a-4553-943a-15fd296e0a24-istio-podinfo\") pod \"aa7e606a-884a-4553-943a-15fd296e0a24\" (UID: \"aa7e606a-884a-4553-943a-15fd296e0a24\") "
Oct 30 07:53:26.178283 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:26.178054    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa7e606a-884a-4553-943a-15fd296e0a24-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa7e606a-884a-4553-943a-15fd296e0a24-workload-socket\") pod \"aa7e606a-884a-4553-943a-15fd296e0a24\" (UID: \"aa7e606a-884a-4553-943a-15fd296e0a24\") "
Oct 30 07:53:26.178283 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:26.178127    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa7e606a-884a-4553-943a-15fd296e0a24-kube-api-access-zpt7g\" (UniqueName: \"kubernetes.io/projected/aa7e606a-884a-4553-943a-15fd296e0a24-kube-api-access-zpt7g\") pod \"aa7e606a-884a-4553-943a-15fd296e0a24\" (UID: \"aa7e606a-884a-4553-943a-15fd296e0a24\") "
Oct 30 07:53:26.178283 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:26.178196    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa7e606a-884a-4553-943a-15fd296e0a24-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa7e606a-884a-4553-943a-15fd296e0a24-workload-certs\") pod \"aa7e606a-884a-4553-943a-15fd296e0a24\" (UID: \"aa7e606a-884a-4553-943a-15fd296e0a24\") "
Oct 30 07:53:26.178283 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:26.178206    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa7e606a-884a-4553-943a-15fd296e0a24-istiod-ca-cert" pod "aa7e606a-884a-4553-943a-15fd296e0a24" (UID: "aa7e606a-884a-4553-943a-15fd296e0a24"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:53:26.178283 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:26.178230    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa7e606a-884a-4553-943a-15fd296e0a24-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa7e606a-884a-4553-943a-15fd296e0a24-istio-envoy\") pod \"aa7e606a-884a-4553-943a-15fd296e0a24\" (UID: \"aa7e606a-884a-4553-943a-15fd296e0a24\") "
Oct 30 07:53:26.178283 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:26.178259    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa7e606a-884a-4553-943a-15fd296e0a24-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa7e606a-884a-4553-943a-15fd296e0a24-istio-data\") pod \"aa7e606a-884a-4553-943a-15fd296e0a24\" (UID: \"aa7e606a-884a-4553-943a-15fd296e0a24\") "
Oct 30 07:53:26.178840 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:26.178763    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa7e606a-884a-4553-943a-15fd296e0a24-workload-socket" pod "aa7e606a-884a-4553-943a-15fd296e0a24" (UID: "aa7e606a-884a-4553-943a-15fd296e0a24"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:53:26.179061 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:26.179038    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa7e606a-884a-4553-943a-15fd296e0a24-istio-data" pod "aa7e606a-884a-4553-943a-15fd296e0a24" (UID: "aa7e606a-884a-4553-943a-15fd296e0a24"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:53:26.179061 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:26.179025    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa7e606a-884a-4553-943a-15fd296e0a24-workload-certs" pod "aa7e606a-884a-4553-943a-15fd296e0a24" (UID: "aa7e606a-884a-4553-943a-15fd296e0a24"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:53:26.179230 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:26.179114    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa7e606a-884a-4553-943a-15fd296e0a24-credential-socket" pod "aa7e606a-884a-4553-943a-15fd296e0a24" (UID: "aa7e606a-884a-4553-943a-15fd296e0a24"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:53:26.179230 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:26.179148    6224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa7e606a-884a-4553-943a-15fd296e0a24-istiod-ca-cert\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:53:26.179230 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:26.179169    6224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa7e606a-884a-4553-943a-15fd296e0a24-workload-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:53:26.181100 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:26.181074    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa7e606a-884a-4553-943a-15fd296e0a24-kube-api-access-zpt7g" pod "aa7e606a-884a-4553-943a-15fd296e0a24" (UID: "aa7e606a-884a-4553-943a-15fd296e0a24"). InnerVolumeSpecName "kube-api-access-zpt7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:53:26.192849 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:26.181968    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa7e606a-884a-4553-943a-15fd296e0a24-istio-envoy" pod "aa7e606a-884a-4553-943a-15fd296e0a24" (UID: "aa7e606a-884a-4553-943a-15fd296e0a24"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:53:26.192849 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:26.186153    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa7e606a-884a-4553-943a-15fd296e0a24-istio-token" pod "aa7e606a-884a-4553-943a-15fd296e0a24" (UID: "aa7e606a-884a-4553-943a-15fd296e0a24"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:53:26.194931 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:26.194900    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa7e606a-884a-4553-943a-15fd296e0a24-istio-podinfo" pod "aa7e606a-884a-4553-943a-15fd296e0a24" (UID: "aa7e606a-884a-4553-943a-15fd296e0a24"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:53:26.280409 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:26.280367    6224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa7e606a-884a-4553-943a-15fd296e0a24-credential-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:53:26.280409 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:26.280407    6224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa7e606a-884a-4553-943a-15fd296e0a24-istio-podinfo\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:53:26.280409 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:26.280419    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpt7g\" (UniqueName: \"kubernetes.io/projected/aa7e606a-884a-4553-943a-15fd296e0a24-kube-api-access-zpt7g\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:53:26.280624 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:26.280431    6224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa7e606a-884a-4553-943a-15fd296e0a24-workload-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:53:26.280624 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:26.280439    6224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa7e606a-884a-4553-943a-15fd296e0a24-istio-envoy\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:53:26.280624 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:26.280447    6224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa7e606a-884a-4553-943a-15fd296e0a24-istio-data\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:53:26.280624 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:26.280455    6224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa7e606a-884a-4553-943a-15fd296e0a24-istio-token\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:53:27.060975 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:27.060941    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28"
Oct 30 07:53:27.060975 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:27.060966    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28" event={"ID":"aa7e606a-884a-4553-943a-15fd296e0a24","Type":"ContainerDied","Data":"29938bd754a68c906bd84e1caca5d2f2ada0cb65d1327ed7d0cd52071491856a"}
Oct 30 07:53:27.061223 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:27.061025    6224 scope.go:122] "RemoveContainer" containerID="f7ba6292111a6248b2dc76056f36da9806cfffe69d84620fab4a714f8ba7c3e7"
Oct 30 07:53:27.141590 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:27.141551    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28"]
Oct 30 07:53:27.155441 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:27.155412    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-8fef73-openshift-default-8494c86c4f-k7v28"]
Oct 30 07:53:27.505545 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:27.505462    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa7e606a-884a-4553-943a-15fd296e0a24" path="/var/lib/kubelet/pods/aa7e606a-884a-4553-943a-15fd296e0a24/volumes"
Oct 30 07:53:30.882840 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:30.882773    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mn4cr"]
Oct 30 07:53:30.883487 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:30.883470    6224 cpu_manager.go:414] "containerMap: removing container" podUID="aa7e606a-884a-4553-943a-15fd296e0a24" containerName="istio-proxy"
Oct 30 07:53:30.883487 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:30.883485    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa7e606a-884a-4553-943a-15fd296e0a24" containerName="istio-proxy"
Oct 30 07:53:30.883632 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:30.883594    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa7e606a-884a-4553-943a-15fd296e0a24" containerName="istio-proxy"
Oct 30 07:53:30.888693 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:30.888673    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mn4cr"
Oct 30 07:53:30.926943 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:30.926915    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mn4cr"]
Oct 30 07:53:30.939862 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:30.939825    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5747b4f9-c446-4172-9e98-cea9f7489f2f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-mn4cr\" (UID: \"5747b4f9-c446-4172-9e98-cea9f7489f2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mn4cr"
Oct 30 07:53:30.940015 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:30.939887    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5747b4f9-c446-4172-9e98-cea9f7489f2f-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-mn4cr\" (UID: \"5747b4f9-c446-4172-9e98-cea9f7489f2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mn4cr"
Oct 30 07:53:30.940015 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:30.939916    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkwpj\" (UniqueName: \"kubernetes.io/projected/5747b4f9-c446-4172-9e98-cea9f7489f2f-kube-api-access-fkwpj\") pod \"maas-default-gateway-openshift-default-6dcf757899-mn4cr\" (UID: \"5747b4f9-c446-4172-9e98-cea9f7489f2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mn4cr"
Oct 30 07:53:30.940015 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:30.939966    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5747b4f9-c446-4172-9e98-cea9f7489f2f-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-mn4cr\" (UID: \"5747b4f9-c446-4172-9e98-cea9f7489f2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mn4cr"
Oct 30 07:53:30.940183 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:30.940015    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5747b4f9-c446-4172-9e98-cea9f7489f2f-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-mn4cr\" (UID: \"5747b4f9-c446-4172-9e98-cea9f7489f2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mn4cr"
Oct 30 07:53:30.940183 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:30.940038    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5747b4f9-c446-4172-9e98-cea9f7489f2f-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-mn4cr\" (UID: \"5747b4f9-c446-4172-9e98-cea9f7489f2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mn4cr"
Oct 30 07:53:30.940183 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:30.940054    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5747b4f9-c446-4172-9e98-cea9f7489f2f-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-mn4cr\" (UID: \"5747b4f9-c446-4172-9e98-cea9f7489f2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mn4cr"
Oct 30 07:53:30.940183 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:30.940092    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5747b4f9-c446-4172-9e98-cea9f7489f2f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-mn4cr\" (UID: \"5747b4f9-c446-4172-9e98-cea9f7489f2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mn4cr"
Oct 30 07:53:30.940183 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:30.940138    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5747b4f9-c446-4172-9e98-cea9f7489f2f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-mn4cr\" (UID: \"5747b4f9-c446-4172-9e98-cea9f7489f2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mn4cr"
Oct 30 07:53:31.041654 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:31.041626    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5747b4f9-c446-4172-9e98-cea9f7489f2f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-mn4cr\" (UID: \"5747b4f9-c446-4172-9e98-cea9f7489f2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mn4cr"
Oct 30 07:53:31.041839 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:31.041686    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5747b4f9-c446-4172-9e98-cea9f7489f2f-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-mn4cr\" (UID: \"5747b4f9-c446-4172-9e98-cea9f7489f2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mn4cr"
Oct 30 07:53:31.041839 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:31.041723    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkwpj\" (UniqueName: \"kubernetes.io/projected/5747b4f9-c446-4172-9e98-cea9f7489f2f-kube-api-access-fkwpj\") pod \"maas-default-gateway-openshift-default-6dcf757899-mn4cr\" (UID: \"5747b4f9-c446-4172-9e98-cea9f7489f2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mn4cr"
Oct 30 07:53:31.041839 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:31.041761    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5747b4f9-c446-4172-9e98-cea9f7489f2f-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-mn4cr\" (UID: \"5747b4f9-c446-4172-9e98-cea9f7489f2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mn4cr"
Oct 30 07:53:31.041839 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:31.041825    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5747b4f9-c446-4172-9e98-cea9f7489f2f-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-mn4cr\" (UID: \"5747b4f9-c446-4172-9e98-cea9f7489f2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mn4cr"
Oct 30 07:53:31.042074 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:31.041861    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5747b4f9-c446-4172-9e98-cea9f7489f2f-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-mn4cr\" (UID: \"5747b4f9-c446-4172-9e98-cea9f7489f2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mn4cr"
Oct 30 07:53:31.042074 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:31.042057    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5747b4f9-c446-4172-9e98-cea9f7489f2f-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-mn4cr\" (UID: \"5747b4f9-c446-4172-9e98-cea9f7489f2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mn4cr"
Oct 30 07:53:31.042188 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:31.042156    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5747b4f9-c446-4172-9e98-cea9f7489f2f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-mn4cr\" (UID: \"5747b4f9-c446-4172-9e98-cea9f7489f2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mn4cr"
Oct 30 07:53:31.042245 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:31.042180    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5747b4f9-c446-4172-9e98-cea9f7489f2f-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-mn4cr\" (UID: \"5747b4f9-c446-4172-9e98-cea9f7489f2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mn4cr"
Oct 30 07:53:31.042245 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:31.042228    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5747b4f9-c446-4172-9e98-cea9f7489f2f-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-mn4cr\" (UID: \"5747b4f9-c446-4172-9e98-cea9f7489f2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mn4cr"
Oct 30 07:53:31.042361 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:31.042259    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5747b4f9-c446-4172-9e98-cea9f7489f2f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-mn4cr\" (UID: \"5747b4f9-c446-4172-9e98-cea9f7489f2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mn4cr"
Oct 30 07:53:31.042361 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:31.042274    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5747b4f9-c446-4172-9e98-cea9f7489f2f-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-mn4cr\" (UID: \"5747b4f9-c446-4172-9e98-cea9f7489f2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mn4cr"
Oct 30 07:53:31.042667 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:31.042644    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5747b4f9-c446-4172-9e98-cea9f7489f2f-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-mn4cr\" (UID: \"5747b4f9-c446-4172-9e98-cea9f7489f2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mn4cr"
Oct 30 07:53:31.042973 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:31.042950    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5747b4f9-c446-4172-9e98-cea9f7489f2f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-mn4cr\" (UID: \"5747b4f9-c446-4172-9e98-cea9f7489f2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mn4cr"
Oct 30 07:53:31.044209 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:31.044187    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5747b4f9-c446-4172-9e98-cea9f7489f2f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-mn4cr\" (UID: \"5747b4f9-c446-4172-9e98-cea9f7489f2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mn4cr"
Oct 30 07:53:31.044389 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:31.044364    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5747b4f9-c446-4172-9e98-cea9f7489f2f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-mn4cr\" (UID: \"5747b4f9-c446-4172-9e98-cea9f7489f2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mn4cr"
Oct 30 07:53:31.070068 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:31.070036    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5747b4f9-c446-4172-9e98-cea9f7489f2f-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-mn4cr\" (UID: \"5747b4f9-c446-4172-9e98-cea9f7489f2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mn4cr"
Oct 30 07:53:31.070179 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:31.070159    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkwpj\" (UniqueName: \"kubernetes.io/projected/5747b4f9-c446-4172-9e98-cea9f7489f2f-kube-api-access-fkwpj\") pod \"maas-default-gateway-openshift-default-6dcf757899-mn4cr\" (UID: \"5747b4f9-c446-4172-9e98-cea9f7489f2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mn4cr"
Oct 30 07:53:31.203591 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:31.203522    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mn4cr"
Oct 30 07:53:31.531185 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:53:31.531142    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5747b4f9_c446_4172_9e98_cea9f7489f2f.slice/crio-23f29004bcaf5b58880392789496652f4090fef76dfd941af701537258760c08 WatchSource:0}: Error finding container 23f29004bcaf5b58880392789496652f4090fef76dfd941af701537258760c08: Status 404 returned error can't find the container with id 23f29004bcaf5b58880392789496652f4090fef76dfd941af701537258760c08
Oct 30 07:53:31.533974 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:31.533949    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mn4cr"]
Oct 30 07:53:31.534349 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:31.534319    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:53:31.534397 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:31.534380    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:53:31.534435 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:31.534410    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:53:31.867250 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:53:31.861876    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:53:32.102027 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:32.101990    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mn4cr" event={"ID":"5747b4f9-c446-4172-9e98-cea9f7489f2f","Type":"ContainerStarted","Data":"eb6baacd206507e426943b8df3d65b1fdf6f43cdc9bbced3cd49f433751dfb7a"}
Oct 30 07:53:32.102431 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:32.102052    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mn4cr" event={"ID":"5747b4f9-c446-4172-9e98-cea9f7489f2f","Type":"ContainerStarted","Data":"23f29004bcaf5b58880392789496652f4090fef76dfd941af701537258760c08"}
Oct 30 07:53:32.205436 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:32.205344    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mn4cr"
Oct 30 07:53:32.207645 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:32.207618    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-mn4cr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body=
Oct 30 07:53:32.207789 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:32.207670    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mn4cr" podUID="5747b4f9-c446-4172-9e98-cea9f7489f2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused"
Oct 30 07:53:33.204191 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:33.204157    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-mn4cr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body=
Oct 30 07:53:33.204675 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:33.204210    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mn4cr" podUID="5747b4f9-c446-4172-9e98-cea9f7489f2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused"
Oct 30 07:53:34.204897 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:34.204860    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-mn4cr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body=
Oct 30 07:53:34.205358 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:34.204929    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mn4cr" podUID="5747b4f9-c446-4172-9e98-cea9f7489f2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused"
Oct 30 07:53:35.204732 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:35.204688    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-mn4cr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body=
Oct 30 07:53:35.204963 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:35.204751    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mn4cr" podUID="5747b4f9-c446-4172-9e98-cea9f7489f2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused"
Oct 30 07:53:36.204388 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:36.204353    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-mn4cr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body=
Oct 30 07:53:36.204583 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:36.204439    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mn4cr" podUID="5747b4f9-c446-4172-9e98-cea9f7489f2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused"
Oct 30 07:53:37.204678 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:37.204645    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-mn4cr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body=
Oct 30 07:53:37.205070 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:37.204702    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mn4cr" podUID="5747b4f9-c446-4172-9e98-cea9f7489f2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused"
Oct 30 07:53:38.204580 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:38.204541    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-mn4cr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body=
Oct 30 07:53:38.204768 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:38.204593    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mn4cr" podUID="5747b4f9-c446-4172-9e98-cea9f7489f2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused"
Oct 30 07:53:40.027678 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:40.027641    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mn4cr"
Oct 30 07:53:40.028267 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:40.028244    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mn4cr"
Oct 30 07:53:40.029348 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:40.029324    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mn4cr"
Oct 30 07:53:40.118178 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:53:40.118122    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mn4cr" podStartSLOduration=10.118102874 podStartE2EDuration="10.118102874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:53:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:53:32.181188117 +0000 UTC m=+3370.790595088" watchObservedRunningTime="2026-10-30 07:53:40.118102874 +0000 UTC m=+3378.727509843"
Oct 30 07:54:01.859355 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:54:01.859310    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:54:09.318398 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:09.318350    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-91c8c2/test-model-91c8c2-kserve-78669df767-s5qb6"]
Oct 30 07:54:16.983204 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:16.983168    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:54:23.933987 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:23.933947    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-8c0e05/test-model-8c0e05-kserve-6797bf4db9-jdlmp"]
Oct 30 07:54:24.351609 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:24.351570    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-91c8c2/test-model-91c8c2-kserve-78669df767-s5qb6" podUID="0632075e-3c5c-4ba9-b79d-541825da2a3e" containerName="main" containerID="cri-o://0faea9cb300cf84ef0c36fab6090ea8563b00bfebaf74ba3a50bee9c4e8fc05a" gracePeriod=45
Oct 30 07:54:24.531766 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:24.531722    6224 generic.go:356] "Generic (PLEG): container finished" podID="0632075e-3c5c-4ba9-b79d-541825da2a3e" containerID="0faea9cb300cf84ef0c36fab6090ea8563b00bfebaf74ba3a50bee9c4e8fc05a" exitCode=0
Oct 30 07:54:24.531987 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:24.531935    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-91c8c2/test-model-91c8c2-kserve-78669df767-s5qb6" event={"ID":"0632075e-3c5c-4ba9-b79d-541825da2a3e","Type":"ContainerDied","Data":"0faea9cb300cf84ef0c36fab6090ea8563b00bfebaf74ba3a50bee9c4e8fc05a"}
Oct 30 07:54:24.655896 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:24.655869    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-91c8c2/test-model-91c8c2-kserve-78669df767-s5qb6"
Oct 30 07:54:24.724718 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:24.724682    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0632075e-3c5c-4ba9-b79d-541825da2a3e-tls-certs\" (UniqueName: \"kubernetes.io/secret/0632075e-3c5c-4ba9-b79d-541825da2a3e-tls-certs\") pod \"0632075e-3c5c-4ba9-b79d-541825da2a3e\" (UID: \"0632075e-3c5c-4ba9-b79d-541825da2a3e\") "
Oct 30 07:54:24.724953 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:24.724733    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0632075e-3c5c-4ba9-b79d-541825da2a3e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0632075e-3c5c-4ba9-b79d-541825da2a3e-dshm\") pod \"0632075e-3c5c-4ba9-b79d-541825da2a3e\" (UID: \"0632075e-3c5c-4ba9-b79d-541825da2a3e\") "
Oct 30 07:54:24.724953 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:24.724868    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0632075e-3c5c-4ba9-b79d-541825da2a3e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0632075e-3c5c-4ba9-b79d-541825da2a3e-model-cache\") pod \"0632075e-3c5c-4ba9-b79d-541825da2a3e\" (UID: \"0632075e-3c5c-4ba9-b79d-541825da2a3e\") "
Oct 30 07:54:24.725099 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:24.724989    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0632075e-3c5c-4ba9-b79d-541825da2a3e-kube-api-access-bcspp\" (UniqueName: \"kubernetes.io/projected/0632075e-3c5c-4ba9-b79d-541825da2a3e-kube-api-access-bcspp\") pod \"0632075e-3c5c-4ba9-b79d-541825da2a3e\" (UID: \"0632075e-3c5c-4ba9-b79d-541825da2a3e\") "
Oct 30 07:54:24.725099 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:24.725062    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0632075e-3c5c-4ba9-b79d-541825da2a3e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0632075e-3c5c-4ba9-b79d-541825da2a3e-tmp-dir\") pod \"0632075e-3c5c-4ba9-b79d-541825da2a3e\" (UID: \"0632075e-3c5c-4ba9-b79d-541825da2a3e\") "
Oct 30 07:54:24.725262 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:24.725136    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0632075e-3c5c-4ba9-b79d-541825da2a3e-home\" (UniqueName: \"kubernetes.io/empty-dir/0632075e-3c5c-4ba9-b79d-541825da2a3e-home\") pod \"0632075e-3c5c-4ba9-b79d-541825da2a3e\" (UID: \"0632075e-3c5c-4ba9-b79d-541825da2a3e\") "
Oct 30 07:54:24.725933 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:24.725841    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0632075e-3c5c-4ba9-b79d-541825da2a3e-model-cache" pod "0632075e-3c5c-4ba9-b79d-541825da2a3e" (UID: "0632075e-3c5c-4ba9-b79d-541825da2a3e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:54:24.725933 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:24.725909    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0632075e-3c5c-4ba9-b79d-541825da2a3e-tmp-dir" pod "0632075e-3c5c-4ba9-b79d-541825da2a3e" (UID: "0632075e-3c5c-4ba9-b79d-541825da2a3e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:54:24.726147 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:24.725997    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0632075e-3c5c-4ba9-b79d-541825da2a3e-home" pod "0632075e-3c5c-4ba9-b79d-541825da2a3e" (UID: "0632075e-3c5c-4ba9-b79d-541825da2a3e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:54:24.727790 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:24.727767    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0632075e-3c5c-4ba9-b79d-541825da2a3e-dshm" pod "0632075e-3c5c-4ba9-b79d-541825da2a3e" (UID: "0632075e-3c5c-4ba9-b79d-541825da2a3e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:54:24.727981 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:24.727941    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0632075e-3c5c-4ba9-b79d-541825da2a3e-tls-certs" pod "0632075e-3c5c-4ba9-b79d-541825da2a3e" (UID: "0632075e-3c5c-4ba9-b79d-541825da2a3e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:54:24.728664 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:24.728630    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0632075e-3c5c-4ba9-b79d-541825da2a3e-kube-api-access-bcspp" pod "0632075e-3c5c-4ba9-b79d-541825da2a3e" (UID: "0632075e-3c5c-4ba9-b79d-541825da2a3e"). InnerVolumeSpecName "kube-api-access-bcspp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:54:24.827019 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:24.826987    6224 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0632075e-3c5c-4ba9-b79d-541825da2a3e-tls-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:54:24.827019 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:24.827013    6224 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0632075e-3c5c-4ba9-b79d-541825da2a3e-dshm\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:54:24.827019 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:24.827022    6224 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0632075e-3c5c-4ba9-b79d-541825da2a3e-model-cache\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:54:24.827296 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:24.827031    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcspp\" (UniqueName: \"kubernetes.io/projected/0632075e-3c5c-4ba9-b79d-541825da2a3e-kube-api-access-bcspp\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:54:24.827296 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:24.827042    6224 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0632075e-3c5c-4ba9-b79d-541825da2a3e-tmp-dir\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:54:24.827296 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:24.827054    6224 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0632075e-3c5c-4ba9-b79d-541825da2a3e-home\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:54:25.540624 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:25.540597    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-91c8c2/test-model-91c8c2-kserve-78669df767-s5qb6"
Oct 30 07:54:25.541072 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:25.540633    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-91c8c2/test-model-91c8c2-kserve-78669df767-s5qb6" event={"ID":"0632075e-3c5c-4ba9-b79d-541825da2a3e","Type":"ContainerDied","Data":"4b9ec3612f10275d1cda938e16648a28f6099bfb8f7c2186b186140415898427"}
Oct 30 07:54:25.541072 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:25.540682    6224 scope.go:122] "RemoveContainer" containerID="0faea9cb300cf84ef0c36fab6090ea8563b00bfebaf74ba3a50bee9c4e8fc05a"
Oct 30 07:54:25.601834 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:25.601773    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-91c8c2/test-model-91c8c2-kserve-78669df767-s5qb6"]
Oct 30 07:54:25.615066 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:25.615039    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-91c8c2/test-model-91c8c2-kserve-78669df767-s5qb6"]
Oct 30 07:54:27.505151 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:27.505111    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0632075e-3c5c-4ba9-b79d-541825da2a3e" path="/var/lib/kubelet/pods/0632075e-3c5c-4ba9-b79d-541825da2a3e/volumes"
Oct 30 07:54:31.859480 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:54:31.859419    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:54:32.471334 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:32.471303    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2"]
Oct 30 07:54:32.471539 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:32.471516    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2" podUID="e1871ac3-4279-4cd5-8fc2-5891514a0dff" containerName="istio-proxy" containerID="cri-o://0115fda8f7fa3c133561ba9c4b1fe1d6cbbff866a9cd8d71d13b0c17d8c083df" gracePeriod=30
Oct 30 07:54:32.535860 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:32.535795    6224 patch_prober.go:29] interesting pod/e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 07:54:32.536038 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:32.535881    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2" podUID="e1871ac3-4279-4cd5-8fc2-5891514a0dff" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 07:54:34.670132 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:34.670101    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-91c8c2-67cf6564dd-z4qql"]
Oct 30 07:54:34.704943 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:34.704912    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-91c8c2-5b7b6b7c59-zlzl6"]
Oct 30 07:54:37.651440 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:37.651405    6224 generic.go:356] "Generic (PLEG): container finished" podID="e1871ac3-4279-4cd5-8fc2-5891514a0dff" containerID="0115fda8f7fa3c133561ba9c4b1fe1d6cbbff866a9cd8d71d13b0c17d8c083df" exitCode=0
Oct 30 07:54:37.651920 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:37.651678    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2" event={"ID":"e1871ac3-4279-4cd5-8fc2-5891514a0dff","Type":"ContainerDied","Data":"0115fda8f7fa3c133561ba9c4b1fe1d6cbbff866a9cd8d71d13b0c17d8c083df"}
Oct 30 07:54:37.743007 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:37.742944    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2"
Oct 30 07:54:37.903729 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:37.903690    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1871ac3-4279-4cd5-8fc2-5891514a0dff-istio-token\" (UniqueName: \"kubernetes.io/projected/e1871ac3-4279-4cd5-8fc2-5891514a0dff-istio-token\") pod \"e1871ac3-4279-4cd5-8fc2-5891514a0dff\" (UID: \"e1871ac3-4279-4cd5-8fc2-5891514a0dff\") "
Oct 30 07:54:37.903729 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:37.903735    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1871ac3-4279-4cd5-8fc2-5891514a0dff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1871ac3-4279-4cd5-8fc2-5891514a0dff-istio-data\") pod \"e1871ac3-4279-4cd5-8fc2-5891514a0dff\" (UID: \"e1871ac3-4279-4cd5-8fc2-5891514a0dff\") "
Oct 30 07:54:37.903998 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:37.903851    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1871ac3-4279-4cd5-8fc2-5891514a0dff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1871ac3-4279-4cd5-8fc2-5891514a0dff-istiod-ca-cert\") pod \"e1871ac3-4279-4cd5-8fc2-5891514a0dff\" (UID: \"e1871ac3-4279-4cd5-8fc2-5891514a0dff\") "
Oct 30 07:54:37.903998 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:37.903904    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e1871ac3-4279-4cd5-8fc2-5891514a0dff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1871ac3-4279-4cd5-8fc2-5891514a0dff-istio-podinfo\") pod \"e1871ac3-4279-4cd5-8fc2-5891514a0dff\" (UID: \"e1871ac3-4279-4cd5-8fc2-5891514a0dff\") "
Oct 30 07:54:37.903998 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:37.903927    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1871ac3-4279-4cd5-8fc2-5891514a0dff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1871ac3-4279-4cd5-8fc2-5891514a0dff-istio-envoy\") pod \"e1871ac3-4279-4cd5-8fc2-5891514a0dff\" (UID: \"e1871ac3-4279-4cd5-8fc2-5891514a0dff\") "
Oct 30 07:54:37.904153 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:37.904139    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1871ac3-4279-4cd5-8fc2-5891514a0dff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1871ac3-4279-4cd5-8fc2-5891514a0dff-workload-socket\") pod \"e1871ac3-4279-4cd5-8fc2-5891514a0dff\" (UID: \"e1871ac3-4279-4cd5-8fc2-5891514a0dff\") "
Oct 30 07:54:37.904221 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:37.904162    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1871ac3-4279-4cd5-8fc2-5891514a0dff-istio-data" pod "e1871ac3-4279-4cd5-8fc2-5891514a0dff" (UID: "e1871ac3-4279-4cd5-8fc2-5891514a0dff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:54:37.904221 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:37.904175    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1871ac3-4279-4cd5-8fc2-5891514a0dff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1871ac3-4279-4cd5-8fc2-5891514a0dff-workload-certs\") pod \"e1871ac3-4279-4cd5-8fc2-5891514a0dff\" (UID: \"e1871ac3-4279-4cd5-8fc2-5891514a0dff\") "
Oct 30 07:54:37.904328 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:37.904262    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1871ac3-4279-4cd5-8fc2-5891514a0dff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1871ac3-4279-4cd5-8fc2-5891514a0dff-credential-socket\") pod \"e1871ac3-4279-4cd5-8fc2-5891514a0dff\" (UID: \"e1871ac3-4279-4cd5-8fc2-5891514a0dff\") "
Oct 30 07:54:37.904328 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:37.904300    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1871ac3-4279-4cd5-8fc2-5891514a0dff-kube-api-access-g4m88\" (UniqueName: \"kubernetes.io/projected/e1871ac3-4279-4cd5-8fc2-5891514a0dff-kube-api-access-g4m88\") pod \"e1871ac3-4279-4cd5-8fc2-5891514a0dff\" (UID: \"e1871ac3-4279-4cd5-8fc2-5891514a0dff\") "
Oct 30 07:54:37.904447 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:37.904372    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1871ac3-4279-4cd5-8fc2-5891514a0dff-workload-certs" pod "e1871ac3-4279-4cd5-8fc2-5891514a0dff" (UID: "e1871ac3-4279-4cd5-8fc2-5891514a0dff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:54:37.904447 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:37.904391    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1871ac3-4279-4cd5-8fc2-5891514a0dff-istiod-ca-cert" pod "e1871ac3-4279-4cd5-8fc2-5891514a0dff" (UID: "e1871ac3-4279-4cd5-8fc2-5891514a0dff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:54:37.904564 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:37.904460    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1871ac3-4279-4cd5-8fc2-5891514a0dff-workload-socket" pod "e1871ac3-4279-4cd5-8fc2-5891514a0dff" (UID: "e1871ac3-4279-4cd5-8fc2-5891514a0dff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:54:37.904628 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:37.904577    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1871ac3-4279-4cd5-8fc2-5891514a0dff-credential-socket" pod "e1871ac3-4279-4cd5-8fc2-5891514a0dff" (UID: "e1871ac3-4279-4cd5-8fc2-5891514a0dff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:54:37.904959 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:37.904939    6224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1871ac3-4279-4cd5-8fc2-5891514a0dff-workload-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:54:37.904959 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:37.904958    6224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1871ac3-4279-4cd5-8fc2-5891514a0dff-workload-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:54:37.905157 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:37.904967    6224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1871ac3-4279-4cd5-8fc2-5891514a0dff-credential-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:54:37.905157 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:37.904978    6224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1871ac3-4279-4cd5-8fc2-5891514a0dff-istio-data\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:54:37.905157 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:37.904987    6224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1871ac3-4279-4cd5-8fc2-5891514a0dff-istiod-ca-cert\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:54:37.906099 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:37.906077    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1871ac3-4279-4cd5-8fc2-5891514a0dff-istio-token" pod "e1871ac3-4279-4cd5-8fc2-5891514a0dff" (UID: "e1871ac3-4279-4cd5-8fc2-5891514a0dff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:54:37.906356 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:37.906342    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1871ac3-4279-4cd5-8fc2-5891514a0dff-istio-envoy" pod "e1871ac3-4279-4cd5-8fc2-5891514a0dff" (UID: "e1871ac3-4279-4cd5-8fc2-5891514a0dff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:54:37.906520 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:37.906506    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e1871ac3-4279-4cd5-8fc2-5891514a0dff-istio-podinfo" pod "e1871ac3-4279-4cd5-8fc2-5891514a0dff" (UID: "e1871ac3-4279-4cd5-8fc2-5891514a0dff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:54:37.906580 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:37.906566    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1871ac3-4279-4cd5-8fc2-5891514a0dff-kube-api-access-g4m88" pod "e1871ac3-4279-4cd5-8fc2-5891514a0dff" (UID: "e1871ac3-4279-4cd5-8fc2-5891514a0dff"). InnerVolumeSpecName "kube-api-access-g4m88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:54:38.005881 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:38.005779    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4m88\" (UniqueName: \"kubernetes.io/projected/e1871ac3-4279-4cd5-8fc2-5891514a0dff-kube-api-access-g4m88\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:54:38.005881 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:38.005840    6224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1871ac3-4279-4cd5-8fc2-5891514a0dff-istio-token\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:54:38.005881 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:38.005851    6224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1871ac3-4279-4cd5-8fc2-5891514a0dff-istio-podinfo\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:54:38.005881 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:38.005859    6224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1871ac3-4279-4cd5-8fc2-5891514a0dff-istio-envoy\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:54:38.662507 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:38.662477    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2"
Oct 30 07:54:38.663041 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:38.662499    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2" event={"ID":"e1871ac3-4279-4cd5-8fc2-5891514a0dff","Type":"ContainerDied","Data":"4f0a4a8282bad04c39bc121b8a036d821cb4c5b07e9adbc42bb5149a1a85d129"}
Oct 30 07:54:38.663041 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:38.662574    6224 scope.go:122] "RemoveContainer" containerID="0115fda8f7fa3c133561ba9c4b1fe1d6cbbff866a9cd8d71d13b0c17d8c083df"
Oct 30 07:54:38.734815 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:38.734778    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2"]
Oct 30 07:54:38.748882 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:38.748860    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bab4c-openshift-default-df548679c-gzwb2"]
Oct 30 07:54:38.964336 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:38.964251    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-8c0e05/test-model-8c0e05-kserve-6797bf4db9-jdlmp" podUID="75c9ba4f-dca7-4227-902f-97b2b8a262c6" containerName="main" containerID="cri-o://ffd9821fcb24f875151738e52a0e8c693be6bc4f78e5a026540b5c45804a8630" gracePeriod=45
Oct 30 07:54:39.015959 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:39.015929    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-8c0e05/test-model-8c0e05-kserve-6797bf4db9-jdlmp" podUID="75c9ba4f-dca7-4227-902f-97b2b8a262c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.205:8000/ready\": dial tcp 10.134.0.205:8000: connect: connection refused"
Oct 30 07:54:39.232749 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:39.232695    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8c0e05/test-model-8c0e05-kserve-6797bf4db9-jdlmp"
Oct 30 07:54:39.321071 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:39.321041    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75c9ba4f-dca7-4227-902f-97b2b8a262c6-home\" (UniqueName: \"kubernetes.io/empty-dir/75c9ba4f-dca7-4227-902f-97b2b8a262c6-home\") pod \"75c9ba4f-dca7-4227-902f-97b2b8a262c6\" (UID: \"75c9ba4f-dca7-4227-902f-97b2b8a262c6\") "
Oct 30 07:54:39.321250 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:39.321079    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75c9ba4f-dca7-4227-902f-97b2b8a262c6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/75c9ba4f-dca7-4227-902f-97b2b8a262c6-dshm\") pod \"75c9ba4f-dca7-4227-902f-97b2b8a262c6\" (UID: \"75c9ba4f-dca7-4227-902f-97b2b8a262c6\") "
Oct 30 07:54:39.321250 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:39.321123    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75c9ba4f-dca7-4227-902f-97b2b8a262c6-kube-api-access-vb7hx\" (UniqueName: \"kubernetes.io/projected/75c9ba4f-dca7-4227-902f-97b2b8a262c6-kube-api-access-vb7hx\") pod \"75c9ba4f-dca7-4227-902f-97b2b8a262c6\" (UID: \"75c9ba4f-dca7-4227-902f-97b2b8a262c6\") "
Oct 30 07:54:39.321250 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:39.321166    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75c9ba4f-dca7-4227-902f-97b2b8a262c6-tls-certs\" (UniqueName: \"kubernetes.io/secret/75c9ba4f-dca7-4227-902f-97b2b8a262c6-tls-certs\") pod \"75c9ba4f-dca7-4227-902f-97b2b8a262c6\" (UID: \"75c9ba4f-dca7-4227-902f-97b2b8a262c6\") "
Oct 30 07:54:39.321439 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:39.321279    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75c9ba4f-dca7-4227-902f-97b2b8a262c6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75c9ba4f-dca7-4227-902f-97b2b8a262c6-model-cache\") pod \"75c9ba4f-dca7-4227-902f-97b2b8a262c6\" (UID: \"75c9ba4f-dca7-4227-902f-97b2b8a262c6\") "
Oct 30 07:54:39.321439 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:39.321310    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75c9ba4f-dca7-4227-902f-97b2b8a262c6-home" pod "75c9ba4f-dca7-4227-902f-97b2b8a262c6" (UID: "75c9ba4f-dca7-4227-902f-97b2b8a262c6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:54:39.321439 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:39.321345    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75c9ba4f-dca7-4227-902f-97b2b8a262c6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75c9ba4f-dca7-4227-902f-97b2b8a262c6-tmp-dir\") pod \"75c9ba4f-dca7-4227-902f-97b2b8a262c6\" (UID: \"75c9ba4f-dca7-4227-902f-97b2b8a262c6\") "
Oct 30 07:54:39.321617 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:39.321549    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75c9ba4f-dca7-4227-902f-97b2b8a262c6-model-cache" pod "75c9ba4f-dca7-4227-902f-97b2b8a262c6" (UID: "75c9ba4f-dca7-4227-902f-97b2b8a262c6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:54:39.321680 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:39.321624    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75c9ba4f-dca7-4227-902f-97b2b8a262c6-tmp-dir" pod "75c9ba4f-dca7-4227-902f-97b2b8a262c6" (UID: "75c9ba4f-dca7-4227-902f-97b2b8a262c6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:54:39.321942 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:39.321921    6224 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75c9ba4f-dca7-4227-902f-97b2b8a262c6-home\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:54:39.321999 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:39.321949    6224 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75c9ba4f-dca7-4227-902f-97b2b8a262c6-model-cache\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:54:39.321999 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:39.321965    6224 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75c9ba4f-dca7-4227-902f-97b2b8a262c6-tmp-dir\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:54:39.323457 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:39.323434    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75c9ba4f-dca7-4227-902f-97b2b8a262c6-dshm" pod "75c9ba4f-dca7-4227-902f-97b2b8a262c6" (UID: "75c9ba4f-dca7-4227-902f-97b2b8a262c6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:54:39.323565 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:39.323506    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75c9ba4f-dca7-4227-902f-97b2b8a262c6-tls-certs" pod "75c9ba4f-dca7-4227-902f-97b2b8a262c6" (UID: "75c9ba4f-dca7-4227-902f-97b2b8a262c6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:54:39.323626 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:39.323611    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75c9ba4f-dca7-4227-902f-97b2b8a262c6-kube-api-access-vb7hx" pod "75c9ba4f-dca7-4227-902f-97b2b8a262c6" (UID: "75c9ba4f-dca7-4227-902f-97b2b8a262c6"). InnerVolumeSpecName "kube-api-access-vb7hx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:54:39.423629 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:39.423598    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb7hx\" (UniqueName: \"kubernetes.io/projected/75c9ba4f-dca7-4227-902f-97b2b8a262c6-kube-api-access-vb7hx\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:54:39.423629 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:39.423626    6224 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75c9ba4f-dca7-4227-902f-97b2b8a262c6-tls-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:54:39.423767 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:39.423637    6224 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75c9ba4f-dca7-4227-902f-97b2b8a262c6-dshm\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:54:39.504558 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:39.504490    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1871ac3-4279-4cd5-8fc2-5891514a0dff" path="/var/lib/kubelet/pods/e1871ac3-4279-4cd5-8fc2-5891514a0dff/volumes"
Oct 30 07:54:39.670533 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:39.670492    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-91c8c2-67cf6564dd-z4qql" podUID="5a38806b-e470-45b3-a949-c457876d72e4" containerName="payload-processing" containerID="cri-o://c66018fdc4671b66bf676cfa94e39c14644065cb4a0bb321aaa2f7bd53bc5336" gracePeriod=25
Oct 30 07:54:39.672233 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:39.672205    6224 generic.go:356] "Generic (PLEG): container finished" podID="75c9ba4f-dca7-4227-902f-97b2b8a262c6" containerID="ffd9821fcb24f875151738e52a0e8c693be6bc4f78e5a026540b5c45804a8630" exitCode=0
Oct 30 07:54:39.672349 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:39.672245    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8c0e05/test-model-8c0e05-kserve-6797bf4db9-jdlmp"
Oct 30 07:54:39.672349 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:39.672245    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8c0e05/test-model-8c0e05-kserve-6797bf4db9-jdlmp" event={"ID":"75c9ba4f-dca7-4227-902f-97b2b8a262c6","Type":"ContainerDied","Data":"ffd9821fcb24f875151738e52a0e8c693be6bc4f78e5a026540b5c45804a8630"}
Oct 30 07:54:39.672425 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:39.672363    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8c0e05/test-model-8c0e05-kserve-6797bf4db9-jdlmp" event={"ID":"75c9ba4f-dca7-4227-902f-97b2b8a262c6","Type":"ContainerDied","Data":"2158b214fd6c11f766605055b8ae457deec8d3409bfc3bb36be89ac74e1a77e3"}
Oct 30 07:54:39.672425 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:39.672387    6224 scope.go:122] "RemoveContainer" containerID="ffd9821fcb24f875151738e52a0e8c693be6bc4f78e5a026540b5c45804a8630"
Oct 30 07:54:39.682962 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:39.682939    6224 scope.go:122] "RemoveContainer" containerID="ffd9821fcb24f875151738e52a0e8c693be6bc4f78e5a026540b5c45804a8630"
Oct 30 07:54:39.683246 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:54:39.683224    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffd9821fcb24f875151738e52a0e8c693be6bc4f78e5a026540b5c45804a8630\": container with ID starting with ffd9821fcb24f875151738e52a0e8c693be6bc4f78e5a026540b5c45804a8630 not found: ID does not exist" containerID="ffd9821fcb24f875151738e52a0e8c693be6bc4f78e5a026540b5c45804a8630"
Oct 30 07:54:39.683307 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:39.683254    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffd9821fcb24f875151738e52a0e8c693be6bc4f78e5a026540b5c45804a8630"} err="failed to get container status \"ffd9821fcb24f875151738e52a0e8c693be6bc4f78e5a026540b5c45804a8630\": rpc error: code = NotFound desc = could not find container \"ffd9821fcb24f875151738e52a0e8c693be6bc4f78e5a026540b5c45804a8630\": container with ID starting with ffd9821fcb24f875151738e52a0e8c693be6bc4f78e5a026540b5c45804a8630 not found: ID does not exist"
Oct 30 07:54:39.705234 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:39.705207    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-91c8c2-5b7b6b7c59-zlzl6" podUID="6626da66-f9b9-4838-8f9e-02ece551f087" containerName="payload-pre-processing" containerID="cri-o://3fbccbd3744945d22b4935d7d83748a23d2ad3f5f846d858d5ede75f7f2ac62e" gracePeriod=25
Oct 30 07:54:39.730356 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:39.730334    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-8c0e05/test-model-8c0e05-kserve-6797bf4db9-jdlmp"]
Oct 30 07:54:39.743986 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:39.743964    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-8c0e05/test-model-8c0e05-kserve-6797bf4db9-jdlmp"]
Oct 30 07:54:39.937189 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:39.937161    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-91c8c2-67cf6564dd-z4qql"
Oct 30 07:54:40.037295 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:40.037260    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a38806b-e470-45b3-a949-c457876d72e4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a38806b-e470-45b3-a949-c457876d72e4-plugins-config-volume\") pod \"5a38806b-e470-45b3-a949-c457876d72e4\" (UID: \"5a38806b-e470-45b3-a949-c457876d72e4\") "
Oct 30 07:54:40.037487 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:40.037430    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a38806b-e470-45b3-a949-c457876d72e4-kube-api-access-5tk9d\" (UniqueName: \"kubernetes.io/projected/5a38806b-e470-45b3-a949-c457876d72e4-kube-api-access-5tk9d\") pod \"5a38806b-e470-45b3-a949-c457876d72e4\" (UID: \"5a38806b-e470-45b3-a949-c457876d72e4\") "
Oct 30 07:54:40.037765 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:40.037738    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a38806b-e470-45b3-a949-c457876d72e4-plugins-config-volume" pod "5a38806b-e470-45b3-a949-c457876d72e4" (UID: "5a38806b-e470-45b3-a949-c457876d72e4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:54:40.038094 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:40.038069    6224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a38806b-e470-45b3-a949-c457876d72e4-plugins-config-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:54:40.039789 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:40.039728    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a38806b-e470-45b3-a949-c457876d72e4-kube-api-access-5tk9d" pod "5a38806b-e470-45b3-a949-c457876d72e4" (UID: "5a38806b-e470-45b3-a949-c457876d72e4"). InnerVolumeSpecName "kube-api-access-5tk9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:54:40.139777 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:40.139750    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tk9d\" (UniqueName: \"kubernetes.io/projected/5a38806b-e470-45b3-a949-c457876d72e4-kube-api-access-5tk9d\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:54:40.682243 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:40.682204    6224 generic.go:356] "Generic (PLEG): container finished" podID="5a38806b-e470-45b3-a949-c457876d72e4" containerID="c66018fdc4671b66bf676cfa94e39c14644065cb4a0bb321aaa2f7bd53bc5336" exitCode=0
Oct 30 07:54:40.682825 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:40.682247    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-91c8c2-67cf6564dd-z4qql"
Oct 30 07:54:40.682825 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:40.682294    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-91c8c2-67cf6564dd-z4qql" event={"ID":"5a38806b-e470-45b3-a949-c457876d72e4","Type":"ContainerDied","Data":"c66018fdc4671b66bf676cfa94e39c14644065cb4a0bb321aaa2f7bd53bc5336"}
Oct 30 07:54:40.682825 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:40.682364    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-91c8c2-67cf6564dd-z4qql" event={"ID":"5a38806b-e470-45b3-a949-c457876d72e4","Type":"ContainerDied","Data":"b8a3bd7dd66bbf9e9b20178dd3c8bb9c3e49f79e7240a1905c2b6189221a041a"}
Oct 30 07:54:40.682825 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:40.682401    6224 scope.go:122] "RemoveContainer" containerID="c66018fdc4671b66bf676cfa94e39c14644065cb4a0bb321aaa2f7bd53bc5336"
Oct 30 07:54:40.693947 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:40.693929    6224 scope.go:122] "RemoveContainer" containerID="c66018fdc4671b66bf676cfa94e39c14644065cb4a0bb321aaa2f7bd53bc5336"
Oct 30 07:54:40.694218 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:54:40.694201    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c66018fdc4671b66bf676cfa94e39c14644065cb4a0bb321aaa2f7bd53bc5336\": container with ID starting with c66018fdc4671b66bf676cfa94e39c14644065cb4a0bb321aaa2f7bd53bc5336 not found: ID does not exist" containerID="c66018fdc4671b66bf676cfa94e39c14644065cb4a0bb321aaa2f7bd53bc5336"
Oct 30 07:54:40.694260 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:40.694224    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c66018fdc4671b66bf676cfa94e39c14644065cb4a0bb321aaa2f7bd53bc5336"} err="failed to get container status \"c66018fdc4671b66bf676cfa94e39c14644065cb4a0bb321aaa2f7bd53bc5336\": rpc error: code = NotFound desc = could not find container \"c66018fdc4671b66bf676cfa94e39c14644065cb4a0bb321aaa2f7bd53bc5336\": container with ID starting with c66018fdc4671b66bf676cfa94e39c14644065cb4a0bb321aaa2f7bd53bc5336 not found: ID does not exist"
Oct 30 07:54:40.740584 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:40.740557    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-91c8c2-67cf6564dd-z4qql"]
Oct 30 07:54:40.754000 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:40.753978    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-91c8c2-67cf6564dd-z4qql"]
Oct 30 07:54:41.504073 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:41.504039    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a38806b-e470-45b3-a949-c457876d72e4" path="/var/lib/kubelet/pods/5a38806b-e470-45b3-a949-c457876d72e4/volumes"
Oct 30 07:54:41.504504 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:41.504490    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75c9ba4f-dca7-4227-902f-97b2b8a262c6" path="/var/lib/kubelet/pods/75c9ba4f-dca7-4227-902f-97b2b8a262c6/volumes"
Oct 30 07:54:46.117460 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:46.117425    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hpbmz"]
Oct 30 07:54:46.118141 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:46.118124    6224 cpu_manager.go:414] "containerMap: removing container" podUID="75c9ba4f-dca7-4227-902f-97b2b8a262c6" containerName="main"
Oct 30 07:54:46.118141 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:46.118141    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75c9ba4f-dca7-4227-902f-97b2b8a262c6" containerName="main"
Oct 30 07:54:46.118228 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:46.118158    6224 cpu_manager.go:414] "containerMap: removing container" podUID="e1871ac3-4279-4cd5-8fc2-5891514a0dff" containerName="istio-proxy"
Oct 30 07:54:46.118228 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:46.118163    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1871ac3-4279-4cd5-8fc2-5891514a0dff" containerName="istio-proxy"
Oct 30 07:54:46.118228 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:46.118174    6224 cpu_manager.go:414] "containerMap: removing container" podUID="5a38806b-e470-45b3-a949-c457876d72e4" containerName="payload-processing"
Oct 30 07:54:46.118228 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:46.118179    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a38806b-e470-45b3-a949-c457876d72e4" containerName="payload-processing"
Oct 30 07:54:46.118228 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:46.118188    6224 cpu_manager.go:414] "containerMap: removing container" podUID="0632075e-3c5c-4ba9-b79d-541825da2a3e" containerName="main"
Oct 30 07:54:46.118228 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:46.118193    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0632075e-3c5c-4ba9-b79d-541825da2a3e" containerName="main"
Oct 30 07:54:46.118434 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:46.118288    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="75c9ba4f-dca7-4227-902f-97b2b8a262c6" containerName="main"
Oct 30 07:54:46.118434 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:46.118296    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1871ac3-4279-4cd5-8fc2-5891514a0dff" containerName="istio-proxy"
Oct 30 07:54:46.118434 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:46.118305    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a38806b-e470-45b3-a949-c457876d72e4" containerName="payload-processing"
Oct 30 07:54:46.118434 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:46.118312    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="0632075e-3c5c-4ba9-b79d-541825da2a3e" containerName="main"
Oct 30 07:54:46.123431 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:46.123406    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hpbmz"
Oct 30 07:54:46.159999 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:46.159965    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hpbmz"]
Oct 30 07:54:46.214230 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:46.214195    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/522040eb-5fce-4b49-9856-410c83cffd8f-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-hpbmz\" (UID: \"522040eb-5fce-4b49-9856-410c83cffd8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hpbmz"
Oct 30 07:54:46.214383 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:46.214236    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/522040eb-5fce-4b49-9856-410c83cffd8f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-hpbmz\" (UID: \"522040eb-5fce-4b49-9856-410c83cffd8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hpbmz"
Oct 30 07:54:46.214383 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:46.214262    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/522040eb-5fce-4b49-9856-410c83cffd8f-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-hpbmz\" (UID: \"522040eb-5fce-4b49-9856-410c83cffd8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hpbmz"
Oct 30 07:54:46.214465 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:46.214395    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkspn\" (UniqueName: \"kubernetes.io/projected/522040eb-5fce-4b49-9856-410c83cffd8f-kube-api-access-tkspn\") pod \"maas-default-gateway-openshift-default-6dcf757899-hpbmz\" (UID: \"522040eb-5fce-4b49-9856-410c83cffd8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hpbmz"
Oct 30 07:54:46.214465 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:46.214436    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/522040eb-5fce-4b49-9856-410c83cffd8f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-hpbmz\" (UID: \"522040eb-5fce-4b49-9856-410c83cffd8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hpbmz"
Oct 30 07:54:46.214465 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:46.214454    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/522040eb-5fce-4b49-9856-410c83cffd8f-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-hpbmz\" (UID: \"522040eb-5fce-4b49-9856-410c83cffd8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hpbmz"
Oct 30 07:54:46.214599 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:46.214573    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/522040eb-5fce-4b49-9856-410c83cffd8f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-hpbmz\" (UID: \"522040eb-5fce-4b49-9856-410c83cffd8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hpbmz"
Oct 30 07:54:46.214688 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:46.214674    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/522040eb-5fce-4b49-9856-410c83cffd8f-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-hpbmz\" (UID: \"522040eb-5fce-4b49-9856-410c83cffd8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hpbmz"
Oct 30 07:54:46.214734 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:46.214719    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/522040eb-5fce-4b49-9856-410c83cffd8f-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-hpbmz\" (UID: \"522040eb-5fce-4b49-9856-410c83cffd8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hpbmz"
Oct 30 07:54:46.315600 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:46.315563    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkspn\" (UniqueName: \"kubernetes.io/projected/522040eb-5fce-4b49-9856-410c83cffd8f-kube-api-access-tkspn\") pod \"maas-default-gateway-openshift-default-6dcf757899-hpbmz\" (UID: \"522040eb-5fce-4b49-9856-410c83cffd8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hpbmz"
Oct 30 07:54:46.315784 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:46.315617    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/522040eb-5fce-4b49-9856-410c83cffd8f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-hpbmz\" (UID: \"522040eb-5fce-4b49-9856-410c83cffd8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hpbmz"
Oct 30 07:54:46.315784 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:46.315637    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/522040eb-5fce-4b49-9856-410c83cffd8f-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-hpbmz\" (UID: \"522040eb-5fce-4b49-9856-410c83cffd8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hpbmz"
Oct 30 07:54:46.315784 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:46.315670    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/522040eb-5fce-4b49-9856-410c83cffd8f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-hpbmz\" (UID: \"522040eb-5fce-4b49-9856-410c83cffd8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hpbmz"
Oct 30 07:54:46.315784 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:46.315739    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/522040eb-5fce-4b49-9856-410c83cffd8f-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-hpbmz\" (UID: \"522040eb-5fce-4b49-9856-410c83cffd8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hpbmz"
Oct 30 07:54:46.315784 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:46.315784    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/522040eb-5fce-4b49-9856-410c83cffd8f-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-hpbmz\" (UID: \"522040eb-5fce-4b49-9856-410c83cffd8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hpbmz"
Oct 30 07:54:46.316302 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:46.315863    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/522040eb-5fce-4b49-9856-410c83cffd8f-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-hpbmz\" (UID: \"522040eb-5fce-4b49-9856-410c83cffd8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hpbmz"
Oct 30 07:54:46.316302 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:46.315893    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/522040eb-5fce-4b49-9856-410c83cffd8f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-hpbmz\" (UID: \"522040eb-5fce-4b49-9856-410c83cffd8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hpbmz"
Oct 30 07:54:46.316302 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:46.316130    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/522040eb-5fce-4b49-9856-410c83cffd8f-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-hpbmz\" (UID: \"522040eb-5fce-4b49-9856-410c83cffd8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hpbmz"
Oct 30 07:54:46.316302 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:46.316202    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/522040eb-5fce-4b49-9856-410c83cffd8f-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-hpbmz\" (UID: \"522040eb-5fce-4b49-9856-410c83cffd8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hpbmz"
Oct 30 07:54:46.316302 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:46.316251    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/522040eb-5fce-4b49-9856-410c83cffd8f-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-hpbmz\" (UID: \"522040eb-5fce-4b49-9856-410c83cffd8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hpbmz"
Oct 30 07:54:46.316302 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:46.316266    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/522040eb-5fce-4b49-9856-410c83cffd8f-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-hpbmz\" (UID: \"522040eb-5fce-4b49-9856-410c83cffd8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hpbmz"
Oct 30 07:54:46.316526 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:46.316468    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/522040eb-5fce-4b49-9856-410c83cffd8f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-hpbmz\" (UID: \"522040eb-5fce-4b49-9856-410c83cffd8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hpbmz"
Oct 30 07:54:46.316526 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:46.316503    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/522040eb-5fce-4b49-9856-410c83cffd8f-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-hpbmz\" (UID: \"522040eb-5fce-4b49-9856-410c83cffd8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hpbmz"
Oct 30 07:54:46.318172 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:46.318151    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/522040eb-5fce-4b49-9856-410c83cffd8f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-hpbmz\" (UID: \"522040eb-5fce-4b49-9856-410c83cffd8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hpbmz"
Oct 30 07:54:46.318404 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:46.318384    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/522040eb-5fce-4b49-9856-410c83cffd8f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-hpbmz\" (UID: \"522040eb-5fce-4b49-9856-410c83cffd8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hpbmz"
Oct 30 07:54:46.341146 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:46.341111    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkspn\" (UniqueName: \"kubernetes.io/projected/522040eb-5fce-4b49-9856-410c83cffd8f-kube-api-access-tkspn\") pod \"maas-default-gateway-openshift-default-6dcf757899-hpbmz\" (UID: \"522040eb-5fce-4b49-9856-410c83cffd8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hpbmz"
Oct 30 07:54:46.341378 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:46.341359    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/522040eb-5fce-4b49-9856-410c83cffd8f-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-hpbmz\" (UID: \"522040eb-5fce-4b49-9856-410c83cffd8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hpbmz"
Oct 30 07:54:46.437057 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:46.436974    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hpbmz"
Oct 30 07:54:46.762133 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:54:46.762098    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod522040eb_5fce_4b49_9856_410c83cffd8f.slice/crio-884862cf8c20ec3e042285f005223b7b71c2343f50e5b1f5c843eacfda93c571 WatchSource:0}: Error finding container 884862cf8c20ec3e042285f005223b7b71c2343f50e5b1f5c843eacfda93c571: Status 404 returned error can't find the container with id 884862cf8c20ec3e042285f005223b7b71c2343f50e5b1f5c843eacfda93c571
Oct 30 07:54:46.762992 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:46.762962    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hpbmz"]
Oct 30 07:54:46.764910 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:46.764881    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:54:46.764996 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:46.764946    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:54:46.764996 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:46.764974    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:54:47.079107 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:47.079070    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-91c8c2-5b7b6b7c59-zlzl6" podUID="6626da66-f9b9-4838-8f9e-02ece551f087" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 07:54:47.745275 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:47.745215    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hpbmz" event={"ID":"522040eb-5fce-4b49-9856-410c83cffd8f","Type":"ContainerStarted","Data":"6acbf2763ed95634cb7c9b74684e73272afdd2fd7735ee2e1cb32e493d6392c9"}
Oct 30 07:54:47.745802 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:47.745293    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hpbmz" event={"ID":"522040eb-5fce-4b49-9856-410c83cffd8f","Type":"ContainerStarted","Data":"884862cf8c20ec3e042285f005223b7b71c2343f50e5b1f5c843eacfda93c571"}
Oct 30 07:54:47.826390 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:47.826332    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hpbmz" podStartSLOduration=1.8263198090000001 podStartE2EDuration="1.826319809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:54:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:54:47.824599707 +0000 UTC m=+3446.434006676" watchObservedRunningTime="2026-10-30 07:54:47.826319809 +0000 UTC m=+3446.435726779"
Oct 30 07:54:48.437790 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:48.437760    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hpbmz"
Oct 30 07:54:48.439676 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:48.439647    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-hpbmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body=
Oct 30 07:54:48.439851 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:48.439711    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hpbmz" podUID="522040eb-5fce-4b49-9856-410c83cffd8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused"
Oct 30 07:54:49.437552 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:49.437515    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-hpbmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body=
Oct 30 07:54:49.438015 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:49.437570    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hpbmz" podUID="522040eb-5fce-4b49-9856-410c83cffd8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused"
Oct 30 07:54:50.438162 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:50.438123    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-hpbmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body=
Oct 30 07:54:50.438650 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:50.438189    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hpbmz" podUID="522040eb-5fce-4b49-9856-410c83cffd8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused"
Oct 30 07:54:51.437472 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:51.437435    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-hpbmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body=
Oct 30 07:54:51.437659 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:51.437498    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hpbmz" podUID="522040eb-5fce-4b49-9856-410c83cffd8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused"
Oct 30 07:54:52.437673 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:52.437635    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-hpbmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body=
Oct 30 07:54:52.438080 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:52.437706    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hpbmz" podUID="522040eb-5fce-4b49-9856-410c83cffd8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused"
Oct 30 07:54:53.438350 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:53.438306    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-hpbmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body=
Oct 30 07:54:53.438791 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:53.438388    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hpbmz" podUID="522040eb-5fce-4b49-9856-410c83cffd8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused"
Oct 30 07:54:55.180213 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:55.180181    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hpbmz"
Oct 30 07:54:55.180718 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:55.180615    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hpbmz"
Oct 30 07:54:55.181587 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:55.181561    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hpbmz"
Oct 30 07:54:57.078681 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:57.078642    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-91c8c2-5b7b6b7c59-zlzl6" podUID="6626da66-f9b9-4838-8f9e-02ece551f087" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 07:54:59.868313 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:59.868276    6224 generic.go:356] "Generic (PLEG): container finished" podID="6626da66-f9b9-4838-8f9e-02ece551f087" containerID="3fbccbd3744945d22b4935d7d83748a23d2ad3f5f846d858d5ede75f7f2ac62e" exitCode=0
Oct 30 07:54:59.868796 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:59.868497    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-91c8c2-5b7b6b7c59-zlzl6" event={"ID":"6626da66-f9b9-4838-8f9e-02ece551f087","Type":"ContainerDied","Data":"3fbccbd3744945d22b4935d7d83748a23d2ad3f5f846d858d5ede75f7f2ac62e"}
Oct 30 07:54:59.986839 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:54:59.986795    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-91c8c2-5b7b6b7c59-zlzl6"
Oct 30 07:55:00.120699 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:55:00.120661    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6626da66-f9b9-4838-8f9e-02ece551f087-kube-api-access-kg2j6\" (UniqueName: \"kubernetes.io/projected/6626da66-f9b9-4838-8f9e-02ece551f087-kube-api-access-kg2j6\") pod \"6626da66-f9b9-4838-8f9e-02ece551f087\" (UID: \"6626da66-f9b9-4838-8f9e-02ece551f087\") "
Oct 30 07:55:00.120699 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:55:00.120703    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6626da66-f9b9-4838-8f9e-02ece551f087-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6626da66-f9b9-4838-8f9e-02ece551f087-plugins-config-volume\") pod \"6626da66-f9b9-4838-8f9e-02ece551f087\" (UID: \"6626da66-f9b9-4838-8f9e-02ece551f087\") "
Oct 30 07:55:00.121102 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:55:00.121078    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6626da66-f9b9-4838-8f9e-02ece551f087-plugins-config-volume" pod "6626da66-f9b9-4838-8f9e-02ece551f087" (UID: "6626da66-f9b9-4838-8f9e-02ece551f087"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:55:00.121249 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:55:00.121235    6224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6626da66-f9b9-4838-8f9e-02ece551f087-plugins-config-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:55:00.122784 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:55:00.122763    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6626da66-f9b9-4838-8f9e-02ece551f087-kube-api-access-kg2j6" pod "6626da66-f9b9-4838-8f9e-02ece551f087" (UID: "6626da66-f9b9-4838-8f9e-02ece551f087"). InnerVolumeSpecName "kube-api-access-kg2j6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:55:00.223003 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:55:00.222939    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kg2j6\" (UniqueName: \"kubernetes.io/projected/6626da66-f9b9-4838-8f9e-02ece551f087-kube-api-access-kg2j6\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:55:00.885947 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:55:00.885870    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-91c8c2-5b7b6b7c59-zlzl6" event={"ID":"6626da66-f9b9-4838-8f9e-02ece551f087","Type":"ContainerDied","Data":"d54a8a74bd770ee46baf9bc759ef2e99dcf98e1fabcca87c2216cdd0c2ca0507"}
Oct 30 07:55:00.886476 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:55:00.885967    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-91c8c2-5b7b6b7c59-zlzl6"
Oct 30 07:55:00.886658 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:55:00.886471    6224 scope.go:122] "RemoveContainer" containerID="3fbccbd3744945d22b4935d7d83748a23d2ad3f5f846d858d5ede75f7f2ac62e"
Oct 30 07:55:00.946071 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:55:00.946041    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-91c8c2-5b7b6b7c59-zlzl6"]
Oct 30 07:55:00.959125 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:55:00.959096    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-91c8c2-5b7b6b7c59-zlzl6"]
Oct 30 07:55:01.504817 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:55:01.504782    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6626da66-f9b9-4838-8f9e-02ece551f087" path="/var/lib/kubelet/pods/6626da66-f9b9-4838-8f9e-02ece551f087/volumes"
Oct 30 07:55:01.860974 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:55:01.860936    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:55:07.301875 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:55:07.301837    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:55:31.858635 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:55:31.858591    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:55:40.252044 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:55:40.252003    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8c0e05-594d698444-8sp4q"]
Oct 30 07:55:40.289732 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:55:40.289702    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8c0e05-697c896c55-l22m2"]
Oct 30 07:55:42.994866 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:55:42.994832    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-0e9723/test-model-0e9723-kserve-67974fd467-jpgfq"]
Oct 30 07:55:45.253252 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:55:45.253199    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-8c0e05-594d698444-8sp4q" podUID="5b4d7b70-3af7-41dd-97f4-b62acb3215d8" containerName="payload-processing" containerID="cri-o://ba3022eed8ed84acb0232dc553407ef2952194729841c61244e02ee2d234f29f" gracePeriod=25
Oct 30 07:55:45.290491 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:55:45.290443    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8c0e05-697c896c55-l22m2" podUID="eb72e294-0efe-4aef-80e7-240321871b92" containerName="payload-pre-processing" containerID="cri-o://da82dbbb437e1c93722bd2a97d702aa47b4fad49fde2532e8c9fa41ecc1f32e6" gracePeriod=25
Oct 30 07:55:45.536496 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:55:45.536470    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8c0e05-594d698444-8sp4q"
Oct 30 07:55:45.616090 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:55:45.616048    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b4d7b70-3af7-41dd-97f4-b62acb3215d8-kube-api-access-jzn7r\" (UniqueName: \"kubernetes.io/projected/5b4d7b70-3af7-41dd-97f4-b62acb3215d8-kube-api-access-jzn7r\") pod \"5b4d7b70-3af7-41dd-97f4-b62acb3215d8\" (UID: \"5b4d7b70-3af7-41dd-97f4-b62acb3215d8\") "
Oct 30 07:55:45.616291 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:55:45.616265    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b4d7b70-3af7-41dd-97f4-b62acb3215d8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b4d7b70-3af7-41dd-97f4-b62acb3215d8-plugins-config-volume\") pod \"5b4d7b70-3af7-41dd-97f4-b62acb3215d8\" (UID: \"5b4d7b70-3af7-41dd-97f4-b62acb3215d8\") "
Oct 30 07:55:45.617016 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:55:45.616985    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b4d7b70-3af7-41dd-97f4-b62acb3215d8-plugins-config-volume" pod "5b4d7b70-3af7-41dd-97f4-b62acb3215d8" (UID: "5b4d7b70-3af7-41dd-97f4-b62acb3215d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:55:45.618558 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:55:45.618533    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b4d7b70-3af7-41dd-97f4-b62acb3215d8-kube-api-access-jzn7r" pod "5b4d7b70-3af7-41dd-97f4-b62acb3215d8" (UID: "5b4d7b70-3af7-41dd-97f4-b62acb3215d8"). InnerVolumeSpecName "kube-api-access-jzn7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:55:45.718101 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:55:45.718074    6224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b4d7b70-3af7-41dd-97f4-b62acb3215d8-plugins-config-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:55:45.718101 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:55:45.718098    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzn7r\" (UniqueName: \"kubernetes.io/projected/5b4d7b70-3af7-41dd-97f4-b62acb3215d8-kube-api-access-jzn7r\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:55:46.232235 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:55:46.232200    6224 generic.go:356] "Generic (PLEG): container finished" podID="5b4d7b70-3af7-41dd-97f4-b62acb3215d8" containerID="ba3022eed8ed84acb0232dc553407ef2952194729841c61244e02ee2d234f29f" exitCode=0
Oct 30 07:55:46.232447 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:55:46.232245    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8c0e05-594d698444-8sp4q"
Oct 30 07:55:46.232447 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:55:46.232255    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8c0e05-594d698444-8sp4q" event={"ID":"5b4d7b70-3af7-41dd-97f4-b62acb3215d8","Type":"ContainerDied","Data":"ba3022eed8ed84acb0232dc553407ef2952194729841c61244e02ee2d234f29f"}
Oct 30 07:55:46.232447 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:55:46.232303    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8c0e05-594d698444-8sp4q" event={"ID":"5b4d7b70-3af7-41dd-97f4-b62acb3215d8","Type":"ContainerDied","Data":"a09bf6f48893251287cdfb91f542879d4fd29407d0a0f91a90f0019279d348f6"}
Oct 30 07:55:46.232447 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:55:46.232327    6224 scope.go:122] "RemoveContainer" containerID="ba3022eed8ed84acb0232dc553407ef2952194729841c61244e02ee2d234f29f"
Oct 30 07:55:46.253984 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:55:46.253954    6224 scope.go:122] "RemoveContainer" containerID="ba3022eed8ed84acb0232dc553407ef2952194729841c61244e02ee2d234f29f"
Oct 30 07:55:46.256546 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:55:46.256520    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba3022eed8ed84acb0232dc553407ef2952194729841c61244e02ee2d234f29f\": container with ID starting with ba3022eed8ed84acb0232dc553407ef2952194729841c61244e02ee2d234f29f not found: ID does not exist" containerID="ba3022eed8ed84acb0232dc553407ef2952194729841c61244e02ee2d234f29f"
Oct 30 07:55:46.256546 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:55:46.256556    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba3022eed8ed84acb0232dc553407ef2952194729841c61244e02ee2d234f29f"} err="failed to get container status \"ba3022eed8ed84acb0232dc553407ef2952194729841c61244e02ee2d234f29f\": rpc error: code = NotFound desc = could not find container \"ba3022eed8ed84acb0232dc553407ef2952194729841c61244e02ee2d234f29f\": container with ID starting with ba3022eed8ed84acb0232dc553407ef2952194729841c61244e02ee2d234f29f not found: ID does not exist"
Oct 30 07:55:46.294021 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:55:46.293982    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8c0e05-594d698444-8sp4q"]
Oct 30 07:55:46.306823 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:55:46.306784    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8c0e05-594d698444-8sp4q"]
Oct 30 07:55:47.505349 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:55:47.505318    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b4d7b70-3af7-41dd-97f4-b62acb3215d8" path="/var/lib/kubelet/pods/5b4d7b70-3af7-41dd-97f4-b62acb3215d8/volumes"
Oct 30 07:55:54.498117 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:55:54.498080    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8c0e05-697c896c55-l22m2" podUID="eb72e294-0efe-4aef-80e7-240321871b92" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 07:55:58.025493 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:55:58.025456    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-0e9723/test-model-0e9723-kserve-67974fd467-jpgfq" podUID="57d482ac-c4e1-4a09-8d52-ed12f04c0b3b" containerName="main" containerID="cri-o://4f17fdd7e7eca1bc8d2f40cfc205be4b80d8e2376442cc3be7ae2225ab8e4b3f" gracePeriod=45
Oct 30 07:55:58.172238 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:55:58.172195    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-0e9723/test-model-0e9723-kserve-67974fd467-jpgfq" podUID="57d482ac-c4e1-4a09-8d52-ed12f04c0b3b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.208:8000/ready\": dial tcp 10.134.0.208:8000: connect: connection refused"
Oct 30 07:55:58.317634 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:55:58.317604    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0e9723/test-model-0e9723-kserve-67974fd467-jpgfq"
Oct 30 07:55:58.320023 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:55:58.319996    6224 generic.go:356] "Generic (PLEG): container finished" podID="57d482ac-c4e1-4a09-8d52-ed12f04c0b3b" containerID="4f17fdd7e7eca1bc8d2f40cfc205be4b80d8e2376442cc3be7ae2225ab8e4b3f" exitCode=0
Oct 30 07:55:58.320140 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:55:58.320043    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0e9723/test-model-0e9723-kserve-67974fd467-jpgfq"
Oct 30 07:55:58.320140 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:55:58.320078    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0e9723/test-model-0e9723-kserve-67974fd467-jpgfq" event={"ID":"57d482ac-c4e1-4a09-8d52-ed12f04c0b3b","Type":"ContainerDied","Data":"4f17fdd7e7eca1bc8d2f40cfc205be4b80d8e2376442cc3be7ae2225ab8e4b3f"}
Oct 30 07:55:58.320140 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:55:58.320121    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0e9723/test-model-0e9723-kserve-67974fd467-jpgfq" event={"ID":"57d482ac-c4e1-4a09-8d52-ed12f04c0b3b","Type":"ContainerDied","Data":"021420b63bf78ae4726a2ad5b6f2908b7f3245c468fece9682301862daa77b36"}
Oct 30 07:55:58.320282 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:55:58.320141    6224 scope.go:122] "RemoveContainer" containerID="4f17fdd7e7eca1bc8d2f40cfc205be4b80d8e2376442cc3be7ae2225ab8e4b3f"
Oct 30 07:55:58.330196 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:55:58.330179    6224 scope.go:122] "RemoveContainer" containerID="4f17fdd7e7eca1bc8d2f40cfc205be4b80d8e2376442cc3be7ae2225ab8e4b3f"
Oct 30 07:55:58.330476 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:55:58.330454    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f17fdd7e7eca1bc8d2f40cfc205be4b80d8e2376442cc3be7ae2225ab8e4b3f\": container with ID starting with 4f17fdd7e7eca1bc8d2f40cfc205be4b80d8e2376442cc3be7ae2225ab8e4b3f not found: ID does not exist" containerID="4f17fdd7e7eca1bc8d2f40cfc205be4b80d8e2376442cc3be7ae2225ab8e4b3f"
Oct 30 07:55:58.330529 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:55:58.330487    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f17fdd7e7eca1bc8d2f40cfc205be4b80d8e2376442cc3be7ae2225ab8e4b3f"} err="failed to get container status \"4f17fdd7e7eca1bc8d2f40cfc205be4b80d8e2376442cc3be7ae2225ab8e4b3f\": rpc error: code = NotFound desc = could not find container \"4f17fdd7e7eca1bc8d2f40cfc205be4b80d8e2376442cc3be7ae2225ab8e4b3f\": container with ID starting with 4f17fdd7e7eca1bc8d2f40cfc205be4b80d8e2376442cc3be7ae2225ab8e4b3f not found: ID does not exist"
Oct 30 07:55:58.375195 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:55:58.375170    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57d482ac-c4e1-4a09-8d52-ed12f04c0b3b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/57d482ac-c4e1-4a09-8d52-ed12f04c0b3b-dshm\") pod \"57d482ac-c4e1-4a09-8d52-ed12f04c0b3b\" (UID: \"57d482ac-c4e1-4a09-8d52-ed12f04c0b3b\") "
Oct 30 07:55:58.375348 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:55:58.375227    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57d482ac-c4e1-4a09-8d52-ed12f04c0b3b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57d482ac-c4e1-4a09-8d52-ed12f04c0b3b-model-cache\") pod \"57d482ac-c4e1-4a09-8d52-ed12f04c0b3b\" (UID: \"57d482ac-c4e1-4a09-8d52-ed12f04c0b3b\") "
Oct 30 07:55:58.375348 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:55:58.375291    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57d482ac-c4e1-4a09-8d52-ed12f04c0b3b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57d482ac-c4e1-4a09-8d52-ed12f04c0b3b-tmp-dir\") pod \"57d482ac-c4e1-4a09-8d52-ed12f04c0b3b\" (UID: \"57d482ac-c4e1-4a09-8d52-ed12f04c0b3b\") "
Oct 30 07:55:58.375495 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:55:58.375477    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57d482ac-c4e1-4a09-8d52-ed12f04c0b3b-home\" (UniqueName: \"kubernetes.io/empty-dir/57d482ac-c4e1-4a09-8d52-ed12f04c0b3b-home\") pod \"57d482ac-c4e1-4a09-8d52-ed12f04c0b3b\" (UID: \"57d482ac-c4e1-4a09-8d52-ed12f04c0b3b\") "
Oct 30 07:55:58.375583 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:55:58.375563    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57d482ac-c4e1-4a09-8d52-ed12f04c0b3b-model-cache" pod "57d482ac-c4e1-4a09-8d52-ed12f04c0b3b" (UID: "57d482ac-c4e1-4a09-8d52-ed12f04c0b3b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:55:58.375712 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:55:58.375691    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57d482ac-c4e1-4a09-8d52-ed12f04c0b3b-kube-api-access-dd55h\" (UniqueName: \"kubernetes.io/projected/57d482ac-c4e1-4a09-8d52-ed12f04c0b3b-kube-api-access-dd55h\") pod \"57d482ac-c4e1-4a09-8d52-ed12f04c0b3b\" (UID: \"57d482ac-c4e1-4a09-8d52-ed12f04c0b3b\") "
Oct 30 07:55:58.376400 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:55:58.375837    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57d482ac-c4e1-4a09-8d52-ed12f04c0b3b-home" pod "57d482ac-c4e1-4a09-8d52-ed12f04c0b3b" (UID: "57d482ac-c4e1-4a09-8d52-ed12f04c0b3b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:55:58.376400 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:55:58.375870    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57d482ac-c4e1-4a09-8d52-ed12f04c0b3b-tls-certs\" (UniqueName: \"kubernetes.io/secret/57d482ac-c4e1-4a09-8d52-ed12f04c0b3b-tls-certs\") pod \"57d482ac-c4e1-4a09-8d52-ed12f04c0b3b\" (UID: \"57d482ac-c4e1-4a09-8d52-ed12f04c0b3b\") "
Oct 30 07:55:58.376400 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:55:58.376278    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57d482ac-c4e1-4a09-8d52-ed12f04c0b3b-tmp-dir" pod "57d482ac-c4e1-4a09-8d52-ed12f04c0b3b" (UID: "57d482ac-c4e1-4a09-8d52-ed12f04c0b3b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:55:58.376636 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:55:58.376613    6224 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57d482ac-c4e1-4a09-8d52-ed12f04c0b3b-tmp-dir\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:55:58.376710 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:55:58.376639    6224 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57d482ac-c4e1-4a09-8d52-ed12f04c0b3b-home\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:55:58.376710 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:55:58.376653    6224 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57d482ac-c4e1-4a09-8d52-ed12f04c0b3b-model-cache\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:55:58.378553 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:55:58.378127    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57d482ac-c4e1-4a09-8d52-ed12f04c0b3b-dshm" pod "57d482ac-c4e1-4a09-8d52-ed12f04c0b3b" (UID: "57d482ac-c4e1-4a09-8d52-ed12f04c0b3b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:55:58.378553 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:55:58.378376    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57d482ac-c4e1-4a09-8d52-ed12f04c0b3b-kube-api-access-dd55h" pod "57d482ac-c4e1-4a09-8d52-ed12f04c0b3b" (UID: "57d482ac-c4e1-4a09-8d52-ed12f04c0b3b"). InnerVolumeSpecName "kube-api-access-dd55h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:55:58.378909 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:55:58.378888    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57d482ac-c4e1-4a09-8d52-ed12f04c0b3b-tls-certs" pod "57d482ac-c4e1-4a09-8d52-ed12f04c0b3b" (UID: "57d482ac-c4e1-4a09-8d52-ed12f04c0b3b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:55:58.477715 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:55:58.477684    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dd55h\" (UniqueName: \"kubernetes.io/projected/57d482ac-c4e1-4a09-8d52-ed12f04c0b3b-kube-api-access-dd55h\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:55:58.477715 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:55:58.477711    6224 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57d482ac-c4e1-4a09-8d52-ed12f04c0b3b-tls-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:55:58.477956 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:55:58.477723    6224 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57d482ac-c4e1-4a09-8d52-ed12f04c0b3b-dshm\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:55:58.672652 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:55:58.672622    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-0e9723/test-model-0e9723-kserve-67974fd467-jpgfq"]
Oct 30 07:55:58.676961 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:55:58.676936    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-0e9723/test-model-0e9723-kserve-67974fd467-jpgfq"]
Oct 30 07:55:59.505011 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:55:59.504979    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57d482ac-c4e1-4a09-8d52-ed12f04c0b3b" path="/var/lib/kubelet/pods/57d482ac-c4e1-4a09-8d52-ed12f04c0b3b/volumes"
Oct 30 07:56:01.857857 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:56:01.857801    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:56:04.497649 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:04.497557    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8c0e05-697c896c55-l22m2" podUID="eb72e294-0efe-4aef-80e7-240321871b92" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 07:56:04.995694 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:04.995649    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-8514a3/test-model-8514a3-kserve-564c455ff5-sxd7p"]
Oct 30 07:56:05.612568 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:05.612540    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8c0e05-697c896c55-l22m2"
Oct 30 07:56:05.670717 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:05.670687    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb72e294-0efe-4aef-80e7-240321871b92-kube-api-access-f78gh\" (UniqueName: \"kubernetes.io/projected/eb72e294-0efe-4aef-80e7-240321871b92-kube-api-access-f78gh\") pod \"eb72e294-0efe-4aef-80e7-240321871b92\" (UID: \"eb72e294-0efe-4aef-80e7-240321871b92\") "
Oct 30 07:56:05.670923 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:05.670740    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb72e294-0efe-4aef-80e7-240321871b92-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb72e294-0efe-4aef-80e7-240321871b92-plugins-config-volume\") pod \"eb72e294-0efe-4aef-80e7-240321871b92\" (UID: \"eb72e294-0efe-4aef-80e7-240321871b92\") "
Oct 30 07:56:05.671457 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:05.671429    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb72e294-0efe-4aef-80e7-240321871b92-plugins-config-volume" pod "eb72e294-0efe-4aef-80e7-240321871b92" (UID: "eb72e294-0efe-4aef-80e7-240321871b92"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:56:05.673031 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:05.673004    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb72e294-0efe-4aef-80e7-240321871b92-kube-api-access-f78gh" pod "eb72e294-0efe-4aef-80e7-240321871b92" (UID: "eb72e294-0efe-4aef-80e7-240321871b92"). InnerVolumeSpecName "kube-api-access-f78gh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:56:05.772719 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:05.772683    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f78gh\" (UniqueName: \"kubernetes.io/projected/eb72e294-0efe-4aef-80e7-240321871b92-kube-api-access-f78gh\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:56:05.772719 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:05.772718    6224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb72e294-0efe-4aef-80e7-240321871b92-plugins-config-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:56:06.379702 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:06.379661    6224 generic.go:356] "Generic (PLEG): container finished" podID="eb72e294-0efe-4aef-80e7-240321871b92" containerID="da82dbbb437e1c93722bd2a97d702aa47b4fad49fde2532e8c9fa41ecc1f32e6" exitCode=0
Oct 30 07:56:06.379918 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:06.379707    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8c0e05-697c896c55-l22m2"
Oct 30 07:56:06.379918 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:06.379748    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8c0e05-697c896c55-l22m2" event={"ID":"eb72e294-0efe-4aef-80e7-240321871b92","Type":"ContainerDied","Data":"da82dbbb437e1c93722bd2a97d702aa47b4fad49fde2532e8c9fa41ecc1f32e6"}
Oct 30 07:56:06.379918 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:06.379867    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8c0e05-697c896c55-l22m2" event={"ID":"eb72e294-0efe-4aef-80e7-240321871b92","Type":"ContainerDied","Data":"bd4d3c38a0f964b4c0222b70a0e27b204f061c6ccd726bae9343b7043db9084d"}
Oct 30 07:56:06.379918 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:06.379900    6224 scope.go:122] "RemoveContainer" containerID="da82dbbb437e1c93722bd2a97d702aa47b4fad49fde2532e8c9fa41ecc1f32e6"
Oct 30 07:56:06.391229 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:06.391212    6224 scope.go:122] "RemoveContainer" containerID="da82dbbb437e1c93722bd2a97d702aa47b4fad49fde2532e8c9fa41ecc1f32e6"
Oct 30 07:56:06.391468 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:56:06.391448    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da82dbbb437e1c93722bd2a97d702aa47b4fad49fde2532e8c9fa41ecc1f32e6\": container with ID starting with da82dbbb437e1c93722bd2a97d702aa47b4fad49fde2532e8c9fa41ecc1f32e6 not found: ID does not exist" containerID="da82dbbb437e1c93722bd2a97d702aa47b4fad49fde2532e8c9fa41ecc1f32e6"
Oct 30 07:56:06.391537 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:06.391479    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da82dbbb437e1c93722bd2a97d702aa47b4fad49fde2532e8c9fa41ecc1f32e6"} err="failed to get container status \"da82dbbb437e1c93722bd2a97d702aa47b4fad49fde2532e8c9fa41ecc1f32e6\": rpc error: code = NotFound desc = could not find container \"da82dbbb437e1c93722bd2a97d702aa47b4fad49fde2532e8c9fa41ecc1f32e6\": container with ID starting with da82dbbb437e1c93722bd2a97d702aa47b4fad49fde2532e8c9fa41ecc1f32e6 not found: ID does not exist"
Oct 30 07:56:06.441195 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:06.441164    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8c0e05-697c896c55-l22m2"]
Oct 30 07:56:06.445960 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:06.445937    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8c0e05-697c896c55-l22m2"]
Oct 30 07:56:07.505927 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:07.505890    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb72e294-0efe-4aef-80e7-240321871b92" path="/var/lib/kubelet/pods/eb72e294-0efe-4aef-80e7-240321871b92/volumes"
Oct 30 07:56:15.857936 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:15.857903    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0e9723-54cb7ddd4-jnzdg"]
Oct 30 07:56:20.020917 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:20.020877    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-8514a3/test-model-8514a3-kserve-564c455ff5-sxd7p" podUID="12cd7103-6458-49fd-b23b-160a32f26b87" containerName="main" containerID="cri-o://791be768c7c4c64ad632a45cd2492d3a76f790814fd2b1a7315c7653fd4e1b8b" gracePeriod=45
Oct 30 07:56:20.292790 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:20.292766    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8514a3/test-model-8514a3-kserve-564c455ff5-sxd7p"
Oct 30 07:56:20.456530 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:20.456501    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12cd7103-6458-49fd-b23b-160a32f26b87-tls-certs\" (UniqueName: \"kubernetes.io/secret/12cd7103-6458-49fd-b23b-160a32f26b87-tls-certs\") pod \"12cd7103-6458-49fd-b23b-160a32f26b87\" (UID: \"12cd7103-6458-49fd-b23b-160a32f26b87\") "
Oct 30 07:56:20.456701 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:20.456544    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12cd7103-6458-49fd-b23b-160a32f26b87-kube-api-access-2z85s\" (UniqueName: \"kubernetes.io/projected/12cd7103-6458-49fd-b23b-160a32f26b87-kube-api-access-2z85s\") pod \"12cd7103-6458-49fd-b23b-160a32f26b87\" (UID: \"12cd7103-6458-49fd-b23b-160a32f26b87\") "
Oct 30 07:56:20.456701 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:20.456621    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12cd7103-6458-49fd-b23b-160a32f26b87-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12cd7103-6458-49fd-b23b-160a32f26b87-model-cache\") pod \"12cd7103-6458-49fd-b23b-160a32f26b87\" (UID: \"12cd7103-6458-49fd-b23b-160a32f26b87\") "
Oct 30 07:56:20.456701 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:20.456638    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12cd7103-6458-49fd-b23b-160a32f26b87-home\" (UniqueName: \"kubernetes.io/empty-dir/12cd7103-6458-49fd-b23b-160a32f26b87-home\") pod \"12cd7103-6458-49fd-b23b-160a32f26b87\" (UID: \"12cd7103-6458-49fd-b23b-160a32f26b87\") "
Oct 30 07:56:20.456701 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:20.456668    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12cd7103-6458-49fd-b23b-160a32f26b87-dshm\" (UniqueName: \"kubernetes.io/empty-dir/12cd7103-6458-49fd-b23b-160a32f26b87-dshm\") pod \"12cd7103-6458-49fd-b23b-160a32f26b87\" (UID: \"12cd7103-6458-49fd-b23b-160a32f26b87\") "
Oct 30 07:56:20.457003 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:20.456739    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12cd7103-6458-49fd-b23b-160a32f26b87-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12cd7103-6458-49fd-b23b-160a32f26b87-tmp-dir\") pod \"12cd7103-6458-49fd-b23b-160a32f26b87\" (UID: \"12cd7103-6458-49fd-b23b-160a32f26b87\") "
Oct 30 07:56:20.457003 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:20.456881    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12cd7103-6458-49fd-b23b-160a32f26b87-model-cache" pod "12cd7103-6458-49fd-b23b-160a32f26b87" (UID: "12cd7103-6458-49fd-b23b-160a32f26b87"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:56:20.457003 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:20.456953    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12cd7103-6458-49fd-b23b-160a32f26b87-home" pod "12cd7103-6458-49fd-b23b-160a32f26b87" (UID: "12cd7103-6458-49fd-b23b-160a32f26b87"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:56:20.457171 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:20.457120    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12cd7103-6458-49fd-b23b-160a32f26b87-tmp-dir" pod "12cd7103-6458-49fd-b23b-160a32f26b87" (UID: "12cd7103-6458-49fd-b23b-160a32f26b87"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:56:20.457382 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:20.457362    6224 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12cd7103-6458-49fd-b23b-160a32f26b87-model-cache\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:56:20.457473 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:20.457385    6224 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12cd7103-6458-49fd-b23b-160a32f26b87-home\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:56:20.457473 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:20.457399    6224 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12cd7103-6458-49fd-b23b-160a32f26b87-tmp-dir\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:56:20.458919 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:20.458899    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12cd7103-6458-49fd-b23b-160a32f26b87-tls-certs" pod "12cd7103-6458-49fd-b23b-160a32f26b87" (UID: "12cd7103-6458-49fd-b23b-160a32f26b87"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:56:20.458998 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:20.458920    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12cd7103-6458-49fd-b23b-160a32f26b87-dshm" pod "12cd7103-6458-49fd-b23b-160a32f26b87" (UID: "12cd7103-6458-49fd-b23b-160a32f26b87"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:56:20.459039 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:20.459018    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12cd7103-6458-49fd-b23b-160a32f26b87-kube-api-access-2z85s" pod "12cd7103-6458-49fd-b23b-160a32f26b87" (UID: "12cd7103-6458-49fd-b23b-160a32f26b87"). InnerVolumeSpecName "kube-api-access-2z85s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:56:20.485736 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:20.485707    6224 generic.go:356] "Generic (PLEG): container finished" podID="12cd7103-6458-49fd-b23b-160a32f26b87" containerID="791be768c7c4c64ad632a45cd2492d3a76f790814fd2b1a7315c7653fd4e1b8b" exitCode=0
Oct 30 07:56:20.485884 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:20.485750    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8514a3/test-model-8514a3-kserve-564c455ff5-sxd7p"
Oct 30 07:56:20.485884 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:20.485795    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8514a3/test-model-8514a3-kserve-564c455ff5-sxd7p" event={"ID":"12cd7103-6458-49fd-b23b-160a32f26b87","Type":"ContainerDied","Data":"791be768c7c4c64ad632a45cd2492d3a76f790814fd2b1a7315c7653fd4e1b8b"}
Oct 30 07:56:20.485884 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:20.485861    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8514a3/test-model-8514a3-kserve-564c455ff5-sxd7p" event={"ID":"12cd7103-6458-49fd-b23b-160a32f26b87","Type":"ContainerDied","Data":"7f6cc5fd63f87926d9909832ffb8d74c901c73f82ce774d36a0573905ce5048f"}
Oct 30 07:56:20.486070 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:20.485884    6224 scope.go:122] "RemoveContainer" containerID="791be768c7c4c64ad632a45cd2492d3a76f790814fd2b1a7315c7653fd4e1b8b"
Oct 30 07:56:20.498226 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:20.498203    6224 scope.go:122] "RemoveContainer" containerID="791be768c7c4c64ad632a45cd2492d3a76f790814fd2b1a7315c7653fd4e1b8b"
Oct 30 07:56:20.501445 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:56:20.501422    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"791be768c7c4c64ad632a45cd2492d3a76f790814fd2b1a7315c7653fd4e1b8b\": container with ID starting with 791be768c7c4c64ad632a45cd2492d3a76f790814fd2b1a7315c7653fd4e1b8b not found: ID does not exist" containerID="791be768c7c4c64ad632a45cd2492d3a76f790814fd2b1a7315c7653fd4e1b8b"
Oct 30 07:56:20.501533 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:20.501455    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"791be768c7c4c64ad632a45cd2492d3a76f790814fd2b1a7315c7653fd4e1b8b"} err="failed to get container status \"791be768c7c4c64ad632a45cd2492d3a76f790814fd2b1a7315c7653fd4e1b8b\": rpc error: code = NotFound desc = could not find container \"791be768c7c4c64ad632a45cd2492d3a76f790814fd2b1a7315c7653fd4e1b8b\": container with ID starting with 791be768c7c4c64ad632a45cd2492d3a76f790814fd2b1a7315c7653fd4e1b8b not found: ID does not exist"
Oct 30 07:56:20.545870 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:20.545846    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-8514a3/test-model-8514a3-kserve-564c455ff5-sxd7p"]
Oct 30 07:56:20.549647 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:20.549624    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-8514a3/test-model-8514a3-kserve-564c455ff5-sxd7p"]
Oct 30 07:56:20.558240 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:20.558215    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z85s\" (UniqueName: \"kubernetes.io/projected/12cd7103-6458-49fd-b23b-160a32f26b87-kube-api-access-2z85s\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:56:20.558240 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:20.558239    6224 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12cd7103-6458-49fd-b23b-160a32f26b87-dshm\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:56:20.558418 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:20.558250    6224 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12cd7103-6458-49fd-b23b-160a32f26b87-tls-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:56:20.859272 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:20.859181    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-0e9723-54cb7ddd4-jnzdg" podUID="46b5c1df-3984-4df3-8d60-8f4bca4228f4" containerName="payload-processing" containerID="cri-o://b20f1aa38d12029b8acf62f9f5e3604167b20739971cec79b96bf8baae6de039" gracePeriod=25
Oct 30 07:56:21.517076 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:21.517004    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12cd7103-6458-49fd-b23b-160a32f26b87" path="/var/lib/kubelet/pods/12cd7103-6458-49fd-b23b-160a32f26b87/volumes"
Oct 30 07:56:25.504968 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:25.504936    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-0e9723-54cb7ddd4-jnzdg" podUID="46b5c1df-3984-4df3-8d60-8f4bca4228f4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 07:56:26.156732 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:26.156698    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:56:31.858793 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:56:31.858753    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:56:35.511752 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:35.511710    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-0e9723-54cb7ddd4-jnzdg" podUID="46b5c1df-3984-4df3-8d60-8f4bca4228f4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 07:56:36.189653 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:36.189612    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ccac6f/test-model-ccac6f-kserve-7c9865d8cf-9sw9m"]
Oct 30 07:56:41.137484 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:41.137456    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0e9723-54cb7ddd4-jnzdg"
Oct 30 07:56:41.237300 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:41.236185    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46b5c1df-3984-4df3-8d60-8f4bca4228f4-kube-api-access-q6br7\" (UniqueName: \"kubernetes.io/projected/46b5c1df-3984-4df3-8d60-8f4bca4228f4-kube-api-access-q6br7\") pod \"46b5c1df-3984-4df3-8d60-8f4bca4228f4\" (UID: \"46b5c1df-3984-4df3-8d60-8f4bca4228f4\") "
Oct 30 07:56:41.237300 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:41.236237    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46b5c1df-3984-4df3-8d60-8f4bca4228f4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46b5c1df-3984-4df3-8d60-8f4bca4228f4-plugins-config-volume\") pod \"46b5c1df-3984-4df3-8d60-8f4bca4228f4\" (UID: \"46b5c1df-3984-4df3-8d60-8f4bca4228f4\") "
Oct 30 07:56:41.238575 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:41.238524    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46b5c1df-3984-4df3-8d60-8f4bca4228f4-plugins-config-volume" pod "46b5c1df-3984-4df3-8d60-8f4bca4228f4" (UID: "46b5c1df-3984-4df3-8d60-8f4bca4228f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:56:41.242233 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:41.242205    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46b5c1df-3984-4df3-8d60-8f4bca4228f4-kube-api-access-q6br7" pod "46b5c1df-3984-4df3-8d60-8f4bca4228f4" (UID: "46b5c1df-3984-4df3-8d60-8f4bca4228f4"). InnerVolumeSpecName "kube-api-access-q6br7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:56:41.338967 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:41.338931    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6br7\" (UniqueName: \"kubernetes.io/projected/46b5c1df-3984-4df3-8d60-8f4bca4228f4-kube-api-access-q6br7\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:56:41.338967 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:41.338959    6224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46b5c1df-3984-4df3-8d60-8f4bca4228f4-plugins-config-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:56:41.644985 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:41.644950    6224 generic.go:356] "Generic (PLEG): container finished" podID="46b5c1df-3984-4df3-8d60-8f4bca4228f4" containerID="b20f1aa38d12029b8acf62f9f5e3604167b20739971cec79b96bf8baae6de039" exitCode=0
Oct 30 07:56:41.645204 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:41.644994    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0e9723-54cb7ddd4-jnzdg"
Oct 30 07:56:41.645204 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:41.645043    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0e9723-54cb7ddd4-jnzdg" event={"ID":"46b5c1df-3984-4df3-8d60-8f4bca4228f4","Type":"ContainerDied","Data":"b20f1aa38d12029b8acf62f9f5e3604167b20739971cec79b96bf8baae6de039"}
Oct 30 07:56:41.645204 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:41.645095    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0e9723-54cb7ddd4-jnzdg" event={"ID":"46b5c1df-3984-4df3-8d60-8f4bca4228f4","Type":"ContainerDied","Data":"f17ff19544e35106c031fdfbfb93123bd6b22c34e6c516159753fa18410878c7"}
Oct 30 07:56:41.645204 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:41.645126    6224 scope.go:122] "RemoveContainer" containerID="b20f1aa38d12029b8acf62f9f5e3604167b20739971cec79b96bf8baae6de039"
Oct 30 07:56:41.659681 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:41.659665    6224 scope.go:122] "RemoveContainer" containerID="b20f1aa38d12029b8acf62f9f5e3604167b20739971cec79b96bf8baae6de039"
Oct 30 07:56:41.660018 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:56:41.659996    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b20f1aa38d12029b8acf62f9f5e3604167b20739971cec79b96bf8baae6de039\": container with ID starting with b20f1aa38d12029b8acf62f9f5e3604167b20739971cec79b96bf8baae6de039 not found: ID does not exist" containerID="b20f1aa38d12029b8acf62f9f5e3604167b20739971cec79b96bf8baae6de039"
Oct 30 07:56:41.660099 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:41.660025    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b20f1aa38d12029b8acf62f9f5e3604167b20739971cec79b96bf8baae6de039"} err="failed to get container status \"b20f1aa38d12029b8acf62f9f5e3604167b20739971cec79b96bf8baae6de039\": rpc error: code = NotFound desc = could not find container \"b20f1aa38d12029b8acf62f9f5e3604167b20739971cec79b96bf8baae6de039\": container with ID starting with b20f1aa38d12029b8acf62f9f5e3604167b20739971cec79b96bf8baae6de039 not found: ID does not exist"
Oct 30 07:56:41.703820 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:41.703787    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0e9723-54cb7ddd4-jnzdg"]
Oct 30 07:56:41.717574 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:41.717546    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0e9723-54cb7ddd4-jnzdg"]
Oct 30 07:56:43.505560 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:43.505522    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46b5c1df-3984-4df3-8d60-8f4bca4228f4" path="/var/lib/kubelet/pods/46b5c1df-3984-4df3-8d60-8f4bca4228f4/volumes"
Oct 30 07:56:44.345206 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:44.345173    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8514a3-774fb4b48c-b547d"]
Oct 30 07:56:49.345619 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:49.345575    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8514a3-774fb4b48c-b547d" podUID="9d6723d6-a9d5-4a9f-a6d3-edc4dca51a7f" containerName="payload-pre-processing" containerID="cri-o://3d52c6fd801512578394c8f4e4841cfebfdcdbb4d02cc9faf5a25eac75a5f794" gracePeriod=25
Oct 30 07:56:51.213885 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:51.213842    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-ccac6f/test-model-ccac6f-kserve-7c9865d8cf-9sw9m" podUID="5c021874-306b-45db-b32a-e3953625ecc8" containerName="main" containerID="cri-o://fcfb792cb95ac074f5681d3b5b240b3afa30f3e06a1242647492f4fd1ac9885e" gracePeriod=45
Oct 30 07:56:51.488800 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:51.488775    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ccac6f/test-model-ccac6f-kserve-7c9865d8cf-9sw9m"
Oct 30 07:56:51.569733 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:51.569701    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c021874-306b-45db-b32a-e3953625ecc8-kube-api-access-ld8cj\" (UniqueName: \"kubernetes.io/projected/5c021874-306b-45db-b32a-e3953625ecc8-kube-api-access-ld8cj\") pod \"5c021874-306b-45db-b32a-e3953625ecc8\" (UID: \"5c021874-306b-45db-b32a-e3953625ecc8\") "
Oct 30 07:56:51.569938 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:51.569834    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c021874-306b-45db-b32a-e3953625ecc8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c021874-306b-45db-b32a-e3953625ecc8-tmp-dir\") pod \"5c021874-306b-45db-b32a-e3953625ecc8\" (UID: \"5c021874-306b-45db-b32a-e3953625ecc8\") "
Oct 30 07:56:51.569938 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:51.569873    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c021874-306b-45db-b32a-e3953625ecc8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c021874-306b-45db-b32a-e3953625ecc8-dshm\") pod \"5c021874-306b-45db-b32a-e3953625ecc8\" (UID: \"5c021874-306b-45db-b32a-e3953625ecc8\") "
Oct 30 07:56:51.570067 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:51.569964    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c021874-306b-45db-b32a-e3953625ecc8-tls-certs\" (UniqueName: \"kubernetes.io/secret/5c021874-306b-45db-b32a-e3953625ecc8-tls-certs\") pod \"5c021874-306b-45db-b32a-e3953625ecc8\" (UID: \"5c021874-306b-45db-b32a-e3953625ecc8\") "
Oct 30 07:56:51.570067 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:51.570049    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c021874-306b-45db-b32a-e3953625ecc8-tmp-dir" pod "5c021874-306b-45db-b32a-e3953625ecc8" (UID: "5c021874-306b-45db-b32a-e3953625ecc8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:56:51.570178 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:51.570091    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c021874-306b-45db-b32a-e3953625ecc8-home\" (UniqueName: \"kubernetes.io/empty-dir/5c021874-306b-45db-b32a-e3953625ecc8-home\") pod \"5c021874-306b-45db-b32a-e3953625ecc8\" (UID: \"5c021874-306b-45db-b32a-e3953625ecc8\") "
Oct 30 07:56:51.570178 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:51.570168    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c021874-306b-45db-b32a-e3953625ecc8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c021874-306b-45db-b32a-e3953625ecc8-model-cache\") pod \"5c021874-306b-45db-b32a-e3953625ecc8\" (UID: \"5c021874-306b-45db-b32a-e3953625ecc8\") "
Oct 30 07:56:51.570689 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:51.570374    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c021874-306b-45db-b32a-e3953625ecc8-home" pod "5c021874-306b-45db-b32a-e3953625ecc8" (UID: "5c021874-306b-45db-b32a-e3953625ecc8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:56:51.570689 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:51.570609    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c021874-306b-45db-b32a-e3953625ecc8-model-cache" pod "5c021874-306b-45db-b32a-e3953625ecc8" (UID: "5c021874-306b-45db-b32a-e3953625ecc8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:56:51.570942 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:51.570719    6224 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c021874-306b-45db-b32a-e3953625ecc8-model-cache\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:56:51.570942 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:51.570744    6224 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c021874-306b-45db-b32a-e3953625ecc8-tmp-dir\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:56:51.570942 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:51.570757    6224 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c021874-306b-45db-b32a-e3953625ecc8-home\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:56:51.572340 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:51.572314    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c021874-306b-45db-b32a-e3953625ecc8-kube-api-access-ld8cj" pod "5c021874-306b-45db-b32a-e3953625ecc8" (UID: "5c021874-306b-45db-b32a-e3953625ecc8"). InnerVolumeSpecName "kube-api-access-ld8cj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:56:51.572453 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:51.572372    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c021874-306b-45db-b32a-e3953625ecc8-dshm" pod "5c021874-306b-45db-b32a-e3953625ecc8" (UID: "5c021874-306b-45db-b32a-e3953625ecc8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:56:51.572453 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:51.572402    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c021874-306b-45db-b32a-e3953625ecc8-tls-certs" pod "5c021874-306b-45db-b32a-e3953625ecc8" (UID: "5c021874-306b-45db-b32a-e3953625ecc8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:56:51.672336 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:51.672308    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ld8cj\" (UniqueName: \"kubernetes.io/projected/5c021874-306b-45db-b32a-e3953625ecc8-kube-api-access-ld8cj\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:56:51.672336 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:51.672335    6224 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c021874-306b-45db-b32a-e3953625ecc8-dshm\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:56:51.672517 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:51.672346    6224 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c021874-306b-45db-b32a-e3953625ecc8-tls-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:56:51.732175 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:51.732106    6224 generic.go:356] "Generic (PLEG): container finished" podID="5c021874-306b-45db-b32a-e3953625ecc8" containerID="fcfb792cb95ac074f5681d3b5b240b3afa30f3e06a1242647492f4fd1ac9885e" exitCode=0
Oct 30 07:56:51.732175 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:51.732149    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ccac6f/test-model-ccac6f-kserve-7c9865d8cf-9sw9m"
Oct 30 07:56:51.732327 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:51.732192    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ccac6f/test-model-ccac6f-kserve-7c9865d8cf-9sw9m" event={"ID":"5c021874-306b-45db-b32a-e3953625ecc8","Type":"ContainerDied","Data":"fcfb792cb95ac074f5681d3b5b240b3afa30f3e06a1242647492f4fd1ac9885e"}
Oct 30 07:56:51.732327 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:51.732231    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ccac6f/test-model-ccac6f-kserve-7c9865d8cf-9sw9m" event={"ID":"5c021874-306b-45db-b32a-e3953625ecc8","Type":"ContainerDied","Data":"a532ca0c8893af1dba5cff0b3070cfc78f0501318a64655c7b87b629fb64a0fb"}
Oct 30 07:56:51.732327 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:51.732248    6224 scope.go:122] "RemoveContainer" containerID="fcfb792cb95ac074f5681d3b5b240b3afa30f3e06a1242647492f4fd1ac9885e"
Oct 30 07:56:51.743394 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:51.743377    6224 scope.go:122] "RemoveContainer" containerID="fcfb792cb95ac074f5681d3b5b240b3afa30f3e06a1242647492f4fd1ac9885e"
Oct 30 07:56:51.743676 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:56:51.743658    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcfb792cb95ac074f5681d3b5b240b3afa30f3e06a1242647492f4fd1ac9885e\": container with ID starting with fcfb792cb95ac074f5681d3b5b240b3afa30f3e06a1242647492f4fd1ac9885e not found: ID does not exist" containerID="fcfb792cb95ac074f5681d3b5b240b3afa30f3e06a1242647492f4fd1ac9885e"
Oct 30 07:56:51.743722 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:51.743686    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcfb792cb95ac074f5681d3b5b240b3afa30f3e06a1242647492f4fd1ac9885e"} err="failed to get container status \"fcfb792cb95ac074f5681d3b5b240b3afa30f3e06a1242647492f4fd1ac9885e\": rpc error: code = NotFound desc = could not find container \"fcfb792cb95ac074f5681d3b5b240b3afa30f3e06a1242647492f4fd1ac9885e\": container with ID starting with fcfb792cb95ac074f5681d3b5b240b3afa30f3e06a1242647492f4fd1ac9885e not found: ID does not exist"
Oct 30 07:56:51.797213 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:51.797183    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ccac6f/test-model-ccac6f-kserve-7c9865d8cf-9sw9m"]
Oct 30 07:56:51.806240 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:51.806217    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-ccac6f/test-model-ccac6f-kserve-7c9865d8cf-9sw9m"]
Oct 30 07:56:53.509632 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:53.509592    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c021874-306b-45db-b32a-e3953625ecc8" path="/var/lib/kubelet/pods/5c021874-306b-45db-b32a-e3953625ecc8/volumes"
Oct 30 07:56:53.769548 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:53.769450    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8514a3-774fb4b48c-b547d" podUID="9d6723d6-a9d5-4a9f-a6d3-edc4dca51a7f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 07:56:55.014832 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:55.011468    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-896ca4/test-model-896ca4-kserve-654ff764b9-lgqsk"]
Oct 30 07:56:55.359127 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:55.357919    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw" podUID="0597696e-26b8-4ef8-8697-64d49b2b9398" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:56:55.409954 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:55.409891    6224 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-2bab4c/distinct-w4-2bab4c-kserve-6868d767c9-j7wlm" podUID="11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": context deadline exceeded"
Oct 30 07:56:55.465988 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:55.465236    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2bab4c/premium-w4-2bab4c-kserve-665887fc68-c6dgf" podUID="f9463476-fcf5-4d66-bfd2-b85ce4ed651f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:56:55.717387 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:55.717291    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb6be4/premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t" podUID="7dd5133c-480f-4fa0-9dd1-27662004aae9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:56:55.717990 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:55.717959    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-06b26e/premium-w0-06b26e-kserve-c9dbd9879-b4jlk" podUID="1230dcae-e1cd-460d-86fc-bd40c8427b5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 30 07:56:55.719474 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:55.719445    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb6be4/simulator-w2-bb6be4-kserve-8485bbf954-dnbbz" podUID="6370f7e2-d6c8-4fb5-8198-32b38837ca31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Oct 30 07:56:55.865735 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:55.862594    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2bab4c/distinct-w4-2bab4c-kserve-6868d767c9-j7wlm" podUID="11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:56:55.928021 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:55.927961    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-06b26e/distinct-w0-06b26e-kserve-545cf57898-2fhhd" podUID="71294bfa-ac2c-40b7-b889-0527daa07f63" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:56:56.096946 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:56.096902    6224 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-mn4cr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 07:56:56.097557 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:56:56.097523    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mn4cr" podUID="5747b4f9-c446-4172-9e98-cea9f7489f2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:57:01.858711 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:57:01.858677    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:57:03.769309 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:03.769263    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8514a3-774fb4b48c-b547d" podUID="9d6723d6-a9d5-4a9f-a6d3-edc4dca51a7f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 07:57:05.867154 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:05.867120    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ccac6f-8567799696-99jx8"]
Oct 30 07:57:09.614047 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:09.613984    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8514a3-774fb4b48c-b547d"
Oct 30 07:57:09.719178 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:09.719139    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d6723d6-a9d5-4a9f-a6d3-edc4dca51a7f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d6723d6-a9d5-4a9f-a6d3-edc4dca51a7f-plugins-config-volume\") pod \"9d6723d6-a9d5-4a9f-a6d3-edc4dca51a7f\" (UID: \"9d6723d6-a9d5-4a9f-a6d3-edc4dca51a7f\") "
Oct 30 07:57:09.719360 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:09.719243    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d6723d6-a9d5-4a9f-a6d3-edc4dca51a7f-kube-api-access-4bpc2\" (UniqueName: \"kubernetes.io/projected/9d6723d6-a9d5-4a9f-a6d3-edc4dca51a7f-kube-api-access-4bpc2\") pod \"9d6723d6-a9d5-4a9f-a6d3-edc4dca51a7f\" (UID: \"9d6723d6-a9d5-4a9f-a6d3-edc4dca51a7f\") "
Oct 30 07:57:09.719518 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:09.719492    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d6723d6-a9d5-4a9f-a6d3-edc4dca51a7f-plugins-config-volume" pod "9d6723d6-a9d5-4a9f-a6d3-edc4dca51a7f" (UID: "9d6723d6-a9d5-4a9f-a6d3-edc4dca51a7f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:57:09.719855 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:09.719831    6224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d6723d6-a9d5-4a9f-a6d3-edc4dca51a7f-plugins-config-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:57:09.721507 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:09.721487    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d6723d6-a9d5-4a9f-a6d3-edc4dca51a7f-kube-api-access-4bpc2" pod "9d6723d6-a9d5-4a9f-a6d3-edc4dca51a7f" (UID: "9d6723d6-a9d5-4a9f-a6d3-edc4dca51a7f"). InnerVolumeSpecName "kube-api-access-4bpc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:57:09.820860 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:09.820827    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bpc2\" (UniqueName: \"kubernetes.io/projected/9d6723d6-a9d5-4a9f-a6d3-edc4dca51a7f-kube-api-access-4bpc2\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:57:10.068403 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:10.068370    6224 generic.go:356] "Generic (PLEG): container finished" podID="9d6723d6-a9d5-4a9f-a6d3-edc4dca51a7f" containerID="3d52c6fd801512578394c8f4e4841cfebfdcdbb4d02cc9faf5a25eac75a5f794" exitCode=0
Oct 30 07:57:10.068582 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:10.068415    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8514a3-774fb4b48c-b547d"
Oct 30 07:57:10.068582 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:10.068448    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8514a3-774fb4b48c-b547d" event={"ID":"9d6723d6-a9d5-4a9f-a6d3-edc4dca51a7f","Type":"ContainerDied","Data":"3d52c6fd801512578394c8f4e4841cfebfdcdbb4d02cc9faf5a25eac75a5f794"}
Oct 30 07:57:10.068582 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:10.068500    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8514a3-774fb4b48c-b547d" event={"ID":"9d6723d6-a9d5-4a9f-a6d3-edc4dca51a7f","Type":"ContainerDied","Data":"24abd65a4245b86ec8615fdea218e200ad7c6002578405884bbf477096ce515e"}
Oct 30 07:57:10.068582 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:10.068524    6224 scope.go:122] "RemoveContainer" containerID="3d52c6fd801512578394c8f4e4841cfebfdcdbb4d02cc9faf5a25eac75a5f794"
Oct 30 07:57:10.079485 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:10.079465    6224 scope.go:122] "RemoveContainer" containerID="3d52c6fd801512578394c8f4e4841cfebfdcdbb4d02cc9faf5a25eac75a5f794"
Oct 30 07:57:10.079738 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:57:10.079719    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d52c6fd801512578394c8f4e4841cfebfdcdbb4d02cc9faf5a25eac75a5f794\": container with ID starting with 3d52c6fd801512578394c8f4e4841cfebfdcdbb4d02cc9faf5a25eac75a5f794 not found: ID does not exist" containerID="3d52c6fd801512578394c8f4e4841cfebfdcdbb4d02cc9faf5a25eac75a5f794"
Oct 30 07:57:10.079791 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:10.079745    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d52c6fd801512578394c8f4e4841cfebfdcdbb4d02cc9faf5a25eac75a5f794"} err="failed to get container status \"3d52c6fd801512578394c8f4e4841cfebfdcdbb4d02cc9faf5a25eac75a5f794\": rpc error: code = NotFound desc = could not find container \"3d52c6fd801512578394c8f4e4841cfebfdcdbb4d02cc9faf5a25eac75a5f794\": container with ID starting with 3d52c6fd801512578394c8f4e4841cfebfdcdbb4d02cc9faf5a25eac75a5f794 not found: ID does not exist"
Oct 30 07:57:10.126898 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:10.126862    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8514a3-774fb4b48c-b547d"]
Oct 30 07:57:10.141119 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:10.141085    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8514a3-774fb4b48c-b547d"]
Oct 30 07:57:10.817562 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:10.817517    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-896ca4/test-model-896ca4-kserve-654ff764b9-lgqsk" podUID="c7b8da2a-15d6-4beb-ab11-ad80a7f162e8" containerName="main" containerID="cri-o://0f6aebc79ccc2eb2a60246c665823b4b209d4c6a56e9fe07c1af4dabfc6f1b77" gracePeriod=45
Oct 30 07:57:10.867424 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:10.867387    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-ccac6f-8567799696-99jx8" podUID="8be122aa-a9d8-45f4-b4c7-738496b41217" containerName="payload-processing" containerID="cri-o://d0280e3571343e26a6dfb9daa90d7b298304dec5689bc2f05d42f455989e03cf" gracePeriod=25
Oct 30 07:57:10.931744 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:10.931703    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-896ca4/test-model-896ca4-kserve-654ff764b9-lgqsk" podUID="c7b8da2a-15d6-4beb-ab11-ad80a7f162e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.207:8000/ready\": dial tcp 10.134.0.207:8000: connect: connection refused"
Oct 30 07:57:11.082154 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.082083    6224 generic.go:356] "Generic (PLEG): container finished" podID="c7b8da2a-15d6-4beb-ab11-ad80a7f162e8" containerID="0f6aebc79ccc2eb2a60246c665823b4b209d4c6a56e9fe07c1af4dabfc6f1b77" exitCode=0
Oct 30 07:57:11.082154 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.082123    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-896ca4/test-model-896ca4-kserve-654ff764b9-lgqsk" event={"ID":"c7b8da2a-15d6-4beb-ab11-ad80a7f162e8","Type":"ContainerDied","Data":"0f6aebc79ccc2eb2a60246c665823b4b209d4c6a56e9fe07c1af4dabfc6f1b77"}
Oct 30 07:57:11.082390 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.082161    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-896ca4/test-model-896ca4-kserve-654ff764b9-lgqsk" event={"ID":"c7b8da2a-15d6-4beb-ab11-ad80a7f162e8","Type":"ContainerDied","Data":"ab34ac8b8e3f6e597aa538f6732eff6f430e75eb1f799358cb32d31df8d31fb7"}
Oct 30 07:57:11.082390 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.082173    6224 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab34ac8b8e3f6e597aa538f6732eff6f430e75eb1f799358cb32d31df8d31fb7"
Oct 30 07:57:11.083884 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.083863    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-896ca4/test-model-896ca4-kserve-654ff764b9-lgqsk"
Oct 30 07:57:11.238524 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.238484    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7b8da2a-15d6-4beb-ab11-ad80a7f162e8-home\" (UniqueName: \"kubernetes.io/empty-dir/c7b8da2a-15d6-4beb-ab11-ad80a7f162e8-home\") pod \"c7b8da2a-15d6-4beb-ab11-ad80a7f162e8\" (UID: \"c7b8da2a-15d6-4beb-ab11-ad80a7f162e8\") "
Oct 30 07:57:11.238709 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.238562    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7b8da2a-15d6-4beb-ab11-ad80a7f162e8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7b8da2a-15d6-4beb-ab11-ad80a7f162e8-dshm\") pod \"c7b8da2a-15d6-4beb-ab11-ad80a7f162e8\" (UID: \"c7b8da2a-15d6-4beb-ab11-ad80a7f162e8\") "
Oct 30 07:57:11.238754 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.238705    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7b8da2a-15d6-4beb-ab11-ad80a7f162e8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7b8da2a-15d6-4beb-ab11-ad80a7f162e8-tmp-dir\") pod \"c7b8da2a-15d6-4beb-ab11-ad80a7f162e8\" (UID: \"c7b8da2a-15d6-4beb-ab11-ad80a7f162e8\") "
Oct 30 07:57:11.238754 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.238743    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7b8da2a-15d6-4beb-ab11-ad80a7f162e8-tls-certs\" (UniqueName: \"kubernetes.io/secret/c7b8da2a-15d6-4beb-ab11-ad80a7f162e8-tls-certs\") pod \"c7b8da2a-15d6-4beb-ab11-ad80a7f162e8\" (UID: \"c7b8da2a-15d6-4beb-ab11-ad80a7f162e8\") "
Oct 30 07:57:11.238892 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.238776    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7b8da2a-15d6-4beb-ab11-ad80a7f162e8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7b8da2a-15d6-4beb-ab11-ad80a7f162e8-model-cache\") pod \"c7b8da2a-15d6-4beb-ab11-ad80a7f162e8\" (UID: \"c7b8da2a-15d6-4beb-ab11-ad80a7f162e8\") "
Oct 30 07:57:11.238892 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.238822    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7b8da2a-15d6-4beb-ab11-ad80a7f162e8-kube-api-access-pvgqp\" (UniqueName: \"kubernetes.io/projected/c7b8da2a-15d6-4beb-ab11-ad80a7f162e8-kube-api-access-pvgqp\") pod \"c7b8da2a-15d6-4beb-ab11-ad80a7f162e8\" (UID: \"c7b8da2a-15d6-4beb-ab11-ad80a7f162e8\") "
Oct 30 07:57:11.238892 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.238794    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b8da2a-15d6-4beb-ab11-ad80a7f162e8-home" pod "c7b8da2a-15d6-4beb-ab11-ad80a7f162e8" (UID: "c7b8da2a-15d6-4beb-ab11-ad80a7f162e8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:57:11.239078 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.239054    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b8da2a-15d6-4beb-ab11-ad80a7f162e8-model-cache" pod "c7b8da2a-15d6-4beb-ab11-ad80a7f162e8" (UID: "c7b8da2a-15d6-4beb-ab11-ad80a7f162e8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:57:11.239078 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.239066    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b8da2a-15d6-4beb-ab11-ad80a7f162e8-tmp-dir" pod "c7b8da2a-15d6-4beb-ab11-ad80a7f162e8" (UID: "c7b8da2a-15d6-4beb-ab11-ad80a7f162e8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:57:11.239318 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.239299    6224 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7b8da2a-15d6-4beb-ab11-ad80a7f162e8-home\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:57:11.239318 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.239316    6224 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7b8da2a-15d6-4beb-ab11-ad80a7f162e8-tmp-dir\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:57:11.239515 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.239326    6224 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7b8da2a-15d6-4beb-ab11-ad80a7f162e8-model-cache\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:57:11.241075 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.241052    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b8da2a-15d6-4beb-ab11-ad80a7f162e8-dshm" pod "c7b8da2a-15d6-4beb-ab11-ad80a7f162e8" (UID: "c7b8da2a-15d6-4beb-ab11-ad80a7f162e8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:57:11.241195 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.241129    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7b8da2a-15d6-4beb-ab11-ad80a7f162e8-tls-certs" pod "c7b8da2a-15d6-4beb-ab11-ad80a7f162e8" (UID: "c7b8da2a-15d6-4beb-ab11-ad80a7f162e8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:57:11.241257 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.241195    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7b8da2a-15d6-4beb-ab11-ad80a7f162e8-kube-api-access-pvgqp" pod "c7b8da2a-15d6-4beb-ab11-ad80a7f162e8" (UID: "c7b8da2a-15d6-4beb-ab11-ad80a7f162e8"). InnerVolumeSpecName "kube-api-access-pvgqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:57:11.340658 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.340568    6224 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7b8da2a-15d6-4beb-ab11-ad80a7f162e8-tls-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:57:11.340658 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.340607    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvgqp\" (UniqueName: \"kubernetes.io/projected/c7b8da2a-15d6-4beb-ab11-ad80a7f162e8-kube-api-access-pvgqp\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:57:11.340658 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.340620    6224 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7b8da2a-15d6-4beb-ab11-ad80a7f162e8-dshm\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:57:11.517068 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.517027    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d6723d6-a9d5-4a9f-a6d3-edc4dca51a7f" path="/var/lib/kubelet/pods/9d6723d6-a9d5-4a9f-a6d3-edc4dca51a7f/volumes"
Oct 30 07:57:11.517909 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.517883    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj"]
Oct 30 07:57:11.518620 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.518602    6224 cpu_manager.go:414] "containerMap: removing container" podUID="c7b8da2a-15d6-4beb-ab11-ad80a7f162e8" containerName="main"
Oct 30 07:57:11.518680 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.518622    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7b8da2a-15d6-4beb-ab11-ad80a7f162e8" containerName="main"
Oct 30 07:57:11.518680 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.518649    6224 cpu_manager.go:414] "containerMap: removing container" podUID="eb72e294-0efe-4aef-80e7-240321871b92" containerName="payload-pre-processing"
Oct 30 07:57:11.518680 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.518654    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb72e294-0efe-4aef-80e7-240321871b92" containerName="payload-pre-processing"
Oct 30 07:57:11.518680 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.518661    6224 cpu_manager.go:414] "containerMap: removing container" podUID="5c021874-306b-45db-b32a-e3953625ecc8" containerName="main"
Oct 30 07:57:11.518680 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.518667    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c021874-306b-45db-b32a-e3953625ecc8" containerName="main"
Oct 30 07:57:11.518680 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.518677    6224 cpu_manager.go:414] "containerMap: removing container" podUID="12cd7103-6458-49fd-b23b-160a32f26b87" containerName="main"
Oct 30 07:57:11.518680 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.518682    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12cd7103-6458-49fd-b23b-160a32f26b87" containerName="main"
Oct 30 07:57:11.518949 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.518694    6224 cpu_manager.go:414] "containerMap: removing container" podUID="5b4d7b70-3af7-41dd-97f4-b62acb3215d8" containerName="payload-processing"
Oct 30 07:57:11.518949 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.518698    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b4d7b70-3af7-41dd-97f4-b62acb3215d8" containerName="payload-processing"
Oct 30 07:57:11.518949 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.518714    6224 cpu_manager.go:414] "containerMap: removing container" podUID="6626da66-f9b9-4838-8f9e-02ece551f087" containerName="payload-pre-processing"
Oct 30 07:57:11.518949 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.518719    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6626da66-f9b9-4838-8f9e-02ece551f087" containerName="payload-pre-processing"
Oct 30 07:57:11.518949 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.518728    6224 cpu_manager.go:414] "containerMap: removing container" podUID="57d482ac-c4e1-4a09-8d52-ed12f04c0b3b" containerName="main"
Oct 30 07:57:11.518949 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.518733    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57d482ac-c4e1-4a09-8d52-ed12f04c0b3b" containerName="main"
Oct 30 07:57:11.518949 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.518739    6224 cpu_manager.go:414] "containerMap: removing container" podUID="46b5c1df-3984-4df3-8d60-8f4bca4228f4" containerName="payload-processing"
Oct 30 07:57:11.518949 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.518744    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46b5c1df-3984-4df3-8d60-8f4bca4228f4" containerName="payload-processing"
Oct 30 07:57:11.518949 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.518755    6224 cpu_manager.go:414] "containerMap: removing container" podUID="9d6723d6-a9d5-4a9f-a6d3-edc4dca51a7f" containerName="payload-pre-processing"
Oct 30 07:57:11.518949 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.518759    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d6723d6-a9d5-4a9f-a6d3-edc4dca51a7f" containerName="payload-pre-processing"
Oct 30 07:57:11.518949 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.518867    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="6626da66-f9b9-4838-8f9e-02ece551f087" containerName="payload-pre-processing"
Oct 30 07:57:11.518949 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.518875    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="57d482ac-c4e1-4a09-8d52-ed12f04c0b3b" containerName="main"
Oct 30 07:57:11.518949 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.518881    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="46b5c1df-3984-4df3-8d60-8f4bca4228f4" containerName="payload-processing"
Oct 30 07:57:11.518949 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.518887    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d6723d6-a9d5-4a9f-a6d3-edc4dca51a7f" containerName="payload-pre-processing"
Oct 30 07:57:11.518949 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.518896    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7b8da2a-15d6-4beb-ab11-ad80a7f162e8" containerName="main"
Oct 30 07:57:11.518949 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.518906    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb72e294-0efe-4aef-80e7-240321871b92" containerName="payload-pre-processing"
Oct 30 07:57:11.518949 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.518914    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c021874-306b-45db-b32a-e3953625ecc8" containerName="main"
Oct 30 07:57:11.518949 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.518920    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="12cd7103-6458-49fd-b23b-160a32f26b87" containerName="main"
Oct 30 07:57:11.518949 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.518929    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b4d7b70-3af7-41dd-97f4-b62acb3215d8" containerName="payload-processing"
Oct 30 07:57:11.523671 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.523649    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj"
Oct 30 07:57:11.601335 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.601200    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj"]
Oct 30 07:57:11.606943 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.606906    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-083c13ed-openshift-default-dockercfg-7zdrd\""
Oct 30 07:57:11.664843 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.655611    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca0e2ee4-723d-4fc1-b13e-72b13c83d280-workload-socket\") pod \"e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj\" (UID: \"ca0e2ee4-723d-4fc1-b13e-72b13c83d280\") " pod="openshift-ingress/e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj"
Oct 30 07:57:11.664843 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.655665    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca0e2ee4-723d-4fc1-b13e-72b13c83d280-istio-podinfo\") pod \"e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj\" (UID: \"ca0e2ee4-723d-4fc1-b13e-72b13c83d280\") " pod="openshift-ingress/e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj"
Oct 30 07:57:11.664843 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.655713    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca0e2ee4-723d-4fc1-b13e-72b13c83d280-istio-data\") pod \"e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj\" (UID: \"ca0e2ee4-723d-4fc1-b13e-72b13c83d280\") " pod="openshift-ingress/e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj"
Oct 30 07:57:11.664843 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.655777    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca0e2ee4-723d-4fc1-b13e-72b13c83d280-workload-certs\") pod \"e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj\" (UID: \"ca0e2ee4-723d-4fc1-b13e-72b13c83d280\") " pod="openshift-ingress/e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj"
Oct 30 07:57:11.664843 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.655895    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhmmt\" (UniqueName: \"kubernetes.io/projected/ca0e2ee4-723d-4fc1-b13e-72b13c83d280-kube-api-access-vhmmt\") pod \"e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj\" (UID: \"ca0e2ee4-723d-4fc1-b13e-72b13c83d280\") " pod="openshift-ingress/e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj"
Oct 30 07:57:11.664843 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.655967    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca0e2ee4-723d-4fc1-b13e-72b13c83d280-istio-token\") pod \"e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj\" (UID: \"ca0e2ee4-723d-4fc1-b13e-72b13c83d280\") " pod="openshift-ingress/e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj"
Oct 30 07:57:11.664843 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.656032    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca0e2ee4-723d-4fc1-b13e-72b13c83d280-istiod-ca-cert\") pod \"e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj\" (UID: \"ca0e2ee4-723d-4fc1-b13e-72b13c83d280\") " pod="openshift-ingress/e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj"
Oct 30 07:57:11.664843 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.656136    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca0e2ee4-723d-4fc1-b13e-72b13c83d280-istio-envoy\") pod \"e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj\" (UID: \"ca0e2ee4-723d-4fc1-b13e-72b13c83d280\") " pod="openshift-ingress/e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj"
Oct 30 07:57:11.664843 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.656167    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca0e2ee4-723d-4fc1-b13e-72b13c83d280-credential-socket\") pod \"e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj\" (UID: \"ca0e2ee4-723d-4fc1-b13e-72b13c83d280\") " pod="openshift-ingress/e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj"
Oct 30 07:57:11.759853 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.757441    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca0e2ee4-723d-4fc1-b13e-72b13c83d280-istiod-ca-cert\") pod \"e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj\" (UID: \"ca0e2ee4-723d-4fc1-b13e-72b13c83d280\") " pod="openshift-ingress/e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj"
Oct 30 07:57:11.759853 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.757535    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca0e2ee4-723d-4fc1-b13e-72b13c83d280-istio-envoy\") pod \"e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj\" (UID: \"ca0e2ee4-723d-4fc1-b13e-72b13c83d280\") " pod="openshift-ingress/e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj"
Oct 30 07:57:11.759853 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.757567    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca0e2ee4-723d-4fc1-b13e-72b13c83d280-credential-socket\") pod \"e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj\" (UID: \"ca0e2ee4-723d-4fc1-b13e-72b13c83d280\") " pod="openshift-ingress/e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj"
Oct 30 07:57:11.759853 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.757684    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca0e2ee4-723d-4fc1-b13e-72b13c83d280-workload-socket\") pod \"e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj\" (UID: \"ca0e2ee4-723d-4fc1-b13e-72b13c83d280\") " pod="openshift-ingress/e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj"
Oct 30 07:57:11.759853 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.757714    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca0e2ee4-723d-4fc1-b13e-72b13c83d280-istio-podinfo\") pod \"e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj\" (UID: \"ca0e2ee4-723d-4fc1-b13e-72b13c83d280\") " pod="openshift-ingress/e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj"
Oct 30 07:57:11.759853 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.757756    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca0e2ee4-723d-4fc1-b13e-72b13c83d280-istio-data\") pod \"e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj\" (UID: \"ca0e2ee4-723d-4fc1-b13e-72b13c83d280\") " pod="openshift-ingress/e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj"
Oct 30 07:57:11.759853 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.757846    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca0e2ee4-723d-4fc1-b13e-72b13c83d280-workload-certs\") pod \"e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj\" (UID: \"ca0e2ee4-723d-4fc1-b13e-72b13c83d280\") " pod="openshift-ingress/e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj"
Oct 30 07:57:11.759853 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.757907    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhmmt\" (UniqueName: \"kubernetes.io/projected/ca0e2ee4-723d-4fc1-b13e-72b13c83d280-kube-api-access-vhmmt\") pod \"e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj\" (UID: \"ca0e2ee4-723d-4fc1-b13e-72b13c83d280\") " pod="openshift-ingress/e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj"
Oct 30 07:57:11.759853 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.757957    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca0e2ee4-723d-4fc1-b13e-72b13c83d280-istio-token\") pod \"e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj\" (UID: \"ca0e2ee4-723d-4fc1-b13e-72b13c83d280\") " pod="openshift-ingress/e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj"
Oct 30 07:57:11.759853 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.758235    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca0e2ee4-723d-4fc1-b13e-72b13c83d280-workload-socket\") pod \"e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj\" (UID: \"ca0e2ee4-723d-4fc1-b13e-72b13c83d280\") " pod="openshift-ingress/e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj"
Oct 30 07:57:11.759853 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.758307    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca0e2ee4-723d-4fc1-b13e-72b13c83d280-istiod-ca-cert\") pod \"e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj\" (UID: \"ca0e2ee4-723d-4fc1-b13e-72b13c83d280\") " pod="openshift-ingress/e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj"
Oct 30 07:57:11.759853 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.758582    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca0e2ee4-723d-4fc1-b13e-72b13c83d280-workload-certs\") pod \"e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj\" (UID: \"ca0e2ee4-723d-4fc1-b13e-72b13c83d280\") " pod="openshift-ingress/e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj"
Oct 30 07:57:11.762421 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.762391    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca0e2ee4-723d-4fc1-b13e-72b13c83d280-istio-data\") pod \"e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj\" (UID: \"ca0e2ee4-723d-4fc1-b13e-72b13c83d280\") " pod="openshift-ingress/e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj"
Oct 30 07:57:11.763134 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.762991    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca0e2ee4-723d-4fc1-b13e-72b13c83d280-credential-socket\") pod \"e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj\" (UID: \"ca0e2ee4-723d-4fc1-b13e-72b13c83d280\") " pod="openshift-ingress/e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj"
Oct 30 07:57:11.763711 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.763692    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca0e2ee4-723d-4fc1-b13e-72b13c83d280-istio-envoy\") pod \"e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj\" (UID: \"ca0e2ee4-723d-4fc1-b13e-72b13c83d280\") " pod="openshift-ingress/e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj"
Oct 30 07:57:11.765570 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.765521    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca0e2ee4-723d-4fc1-b13e-72b13c83d280-istio-podinfo\") pod \"e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj\" (UID: \"ca0e2ee4-723d-4fc1-b13e-72b13c83d280\") " pod="openshift-ingress/e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj"
Oct 30 07:57:11.784064 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.784040    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca0e2ee4-723d-4fc1-b13e-72b13c83d280-istio-token\") pod \"e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj\" (UID: \"ca0e2ee4-723d-4fc1-b13e-72b13c83d280\") " pod="openshift-ingress/e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj"
Oct 30 07:57:11.787054 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.787028    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhmmt\" (UniqueName: \"kubernetes.io/projected/ca0e2ee4-723d-4fc1-b13e-72b13c83d280-kube-api-access-vhmmt\") pod \"e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj\" (UID: \"ca0e2ee4-723d-4fc1-b13e-72b13c83d280\") " pod="openshift-ingress/e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj"
Oct 30 07:57:11.839881 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:11.839850    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj"
Oct 30 07:57:12.089735 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:12.089696    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-896ca4/test-model-896ca4-kserve-654ff764b9-lgqsk"
Oct 30 07:57:12.150211 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:12.150121    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-896ca4/test-model-896ca4-kserve-654ff764b9-lgqsk"]
Oct 30 07:57:12.153737 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:12.153709    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-896ca4/test-model-896ca4-kserve-654ff764b9-lgqsk"]
Oct 30 07:57:12.167590 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:12.167562    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj"]
Oct 30 07:57:12.169038 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:57:12.169005    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca0e2ee4_723d_4fc1_b13e_72b13c83d280.slice/crio-f3c6151e9bb7712464317e38333daf2aec76a0fba50949ae5c74532c4287b6d5 WatchSource:0}: Error finding container f3c6151e9bb7712464317e38333daf2aec76a0fba50949ae5c74532c4287b6d5: Status 404 returned error can't find the container with id f3c6151e9bb7712464317e38333daf2aec76a0fba50949ae5c74532c4287b6d5
Oct 30 07:57:12.172460 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:12.172420    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:57:12.172545 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:12.172506    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:57:12.172597 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:12.172552    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:57:12.905267 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:12.905230    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-ccac6f-8567799696-99jx8" podUID="8be122aa-a9d8-45f4-b4c7-738496b41217" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 07:57:13.098720 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:13.098686    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj" event={"ID":"ca0e2ee4-723d-4fc1-b13e-72b13c83d280","Type":"ContainerStarted","Data":"5b53486850f289a4bf760dee28e46d37112e535569df409558a611e76a20b4c9"}
Oct 30 07:57:13.098921 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:13.098742    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj" event={"ID":"ca0e2ee4-723d-4fc1-b13e-72b13c83d280","Type":"ContainerStarted","Data":"f3c6151e9bb7712464317e38333daf2aec76a0fba50949ae5c74532c4287b6d5"}
Oct 30 07:57:13.147954 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:13.147906    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj" podStartSLOduration=2.147892785 podStartE2EDuration="2.147892785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:57:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:57:13.146172869 +0000 UTC m=+3591.755579854" watchObservedRunningTime="2026-10-30 07:57:13.147892785 +0000 UTC m=+3591.757299756"
Oct 30 07:57:13.504036 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:13.504000    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7b8da2a-15d6-4beb-ab11-ad80a7f162e8" path="/var/lib/kubelet/pods/c7b8da2a-15d6-4beb-ab11-ad80a7f162e8/volumes"
Oct 30 07:57:13.840158 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:13.840124    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj"
Oct 30 07:57:13.846399 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:13.846370    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj"
Oct 30 07:57:14.105870 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:14.105767    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj"
Oct 30 07:57:14.106833 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:14.106787    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj"
Oct 30 07:57:17.617225 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:17.617190    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-083c13ed-6df68c987f-644fr"]
Oct 30 07:57:17.621886 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:17.621864    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-083c13ed-6df68c987f-644fr"
Oct 30 07:57:17.643565 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:17.643537    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-083c13ed-6df68c987f-644fr"]
Oct 30 07:57:17.652329 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:17.652306    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-083c13ed\""
Oct 30 07:57:17.652476 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:17.652344    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-083c13ed-dockercfg-s58tl\""
Oct 30 07:57:17.772834 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:17.747762    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/271eca48-7f0b-4619-914c-bbf966c47e3d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-083c13ed-6df68c987f-644fr\" (UID: \"271eca48-7f0b-4619-914c-bbf966c47e3d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-083c13ed-6df68c987f-644fr"
Oct 30 07:57:17.825631 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:17.819860    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-083c13ed-64db67c4c4-rbvkp"]
Oct 30 07:57:17.825909 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:17.825874    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5bpv\" (UniqueName: \"kubernetes.io/projected/271eca48-7f0b-4619-914c-bbf966c47e3d-kube-api-access-b5bpv\") pod \"payload-pre-processing-e2e-mt-083c13ed-6df68c987f-644fr\" (UID: \"271eca48-7f0b-4619-914c-bbf966c47e3d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-083c13ed-6df68c987f-644fr"
Oct 30 07:57:17.830989 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:17.830924    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-083c13ed-64db67c4c4-rbvkp"]
Oct 30 07:57:17.831197 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:17.831087    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-083c13ed-64db67c4c4-rbvkp"
Oct 30 07:57:17.927090 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:17.927011    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2eec27f2-9ce9-4269-9c99-0ab3839cf2ee-plugins-config-volume\") pod \"payload-processing-e2e-mt-083c13ed-64db67c4c4-rbvkp\" (UID: \"2eec27f2-9ce9-4269-9c99-0ab3839cf2ee\") " pod="openshift-ingress/payload-processing-e2e-mt-083c13ed-64db67c4c4-rbvkp"
Oct 30 07:57:17.927090 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:17.927063    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8px9\" (UniqueName: \"kubernetes.io/projected/2eec27f2-9ce9-4269-9c99-0ab3839cf2ee-kube-api-access-n8px9\") pod \"payload-processing-e2e-mt-083c13ed-64db67c4c4-rbvkp\" (UID: \"2eec27f2-9ce9-4269-9c99-0ab3839cf2ee\") " pod="openshift-ingress/payload-processing-e2e-mt-083c13ed-64db67c4c4-rbvkp"
Oct 30 07:57:17.927278 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:17.927177    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5bpv\" (UniqueName: \"kubernetes.io/projected/271eca48-7f0b-4619-914c-bbf966c47e3d-kube-api-access-b5bpv\") pod \"payload-pre-processing-e2e-mt-083c13ed-6df68c987f-644fr\" (UID: \"271eca48-7f0b-4619-914c-bbf966c47e3d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-083c13ed-6df68c987f-644fr"
Oct 30 07:57:17.927278 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:17.927243    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/271eca48-7f0b-4619-914c-bbf966c47e3d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-083c13ed-6df68c987f-644fr\" (UID: \"271eca48-7f0b-4619-914c-bbf966c47e3d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-083c13ed-6df68c987f-644fr"
Oct 30 07:57:17.927968 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:17.927940    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/271eca48-7f0b-4619-914c-bbf966c47e3d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-083c13ed-6df68c987f-644fr\" (UID: \"271eca48-7f0b-4619-914c-bbf966c47e3d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-083c13ed-6df68c987f-644fr"
Oct 30 07:57:17.952831 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:17.952792    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5bpv\" (UniqueName: \"kubernetes.io/projected/271eca48-7f0b-4619-914c-bbf966c47e3d-kube-api-access-b5bpv\") pod \"payload-pre-processing-e2e-mt-083c13ed-6df68c987f-644fr\" (UID: \"271eca48-7f0b-4619-914c-bbf966c47e3d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-083c13ed-6df68c987f-644fr"
Oct 30 07:57:18.028036 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:18.028005    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2eec27f2-9ce9-4269-9c99-0ab3839cf2ee-plugins-config-volume\") pod \"payload-processing-e2e-mt-083c13ed-64db67c4c4-rbvkp\" (UID: \"2eec27f2-9ce9-4269-9c99-0ab3839cf2ee\") " pod="openshift-ingress/payload-processing-e2e-mt-083c13ed-64db67c4c4-rbvkp"
Oct 30 07:57:18.028178 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:18.028059    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8px9\" (UniqueName: \"kubernetes.io/projected/2eec27f2-9ce9-4269-9c99-0ab3839cf2ee-kube-api-access-n8px9\") pod \"payload-processing-e2e-mt-083c13ed-64db67c4c4-rbvkp\" (UID: \"2eec27f2-9ce9-4269-9c99-0ab3839cf2ee\") " pod="openshift-ingress/payload-processing-e2e-mt-083c13ed-64db67c4c4-rbvkp"
Oct 30 07:57:18.028750 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:18.028732    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2eec27f2-9ce9-4269-9c99-0ab3839cf2ee-plugins-config-volume\") pod \"payload-processing-e2e-mt-083c13ed-64db67c4c4-rbvkp\" (UID: \"2eec27f2-9ce9-4269-9c99-0ab3839cf2ee\") " pod="openshift-ingress/payload-processing-e2e-mt-083c13ed-64db67c4c4-rbvkp"
Oct 30 07:57:18.052957 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:18.052937    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8px9\" (UniqueName: \"kubernetes.io/projected/2eec27f2-9ce9-4269-9c99-0ab3839cf2ee-kube-api-access-n8px9\") pod \"payload-processing-e2e-mt-083c13ed-64db67c4c4-rbvkp\" (UID: \"2eec27f2-9ce9-4269-9c99-0ab3839cf2ee\") " pod="openshift-ingress/payload-processing-e2e-mt-083c13ed-64db67c4c4-rbvkp"
Oct 30 07:57:18.145878 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:18.145852    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-083c13ed-64db67c4c4-rbvkp"
Oct 30 07:57:18.238368 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:18.238332    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-083c13ed-6df68c987f-644fr"
Oct 30 07:57:18.446069 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:57:18.446038    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eec27f2_9ce9_4269_9c99_0ab3839cf2ee.slice/crio-1337029e4510b4c78f8853e59fa1bee7ad97ad00af24e87db74915d4cd6132ab WatchSource:0}: Error finding container 1337029e4510b4c78f8853e59fa1bee7ad97ad00af24e87db74915d4cd6132ab: Status 404 returned error can't find the container with id 1337029e4510b4c78f8853e59fa1bee7ad97ad00af24e87db74915d4cd6132ab
Oct 30 07:57:18.446754 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:18.446729    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-083c13ed-64db67c4c4-rbvkp"]
Oct 30 07:57:18.540175 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:57:18.540148    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod271eca48_7f0b_4619_914c_bbf966c47e3d.slice/crio-922d3fd99a976fec2af991f9a932c73c15ca048692ae527b503b36daf51dab4d WatchSource:0}: Error finding container 922d3fd99a976fec2af991f9a932c73c15ca048692ae527b503b36daf51dab4d: Status 404 returned error can't find the container with id 922d3fd99a976fec2af991f9a932c73c15ca048692ae527b503b36daf51dab4d
Oct 30 07:57:18.541607 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:18.541580    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-083c13ed-6df68c987f-644fr"]
Oct 30 07:57:19.144837 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:19.144764    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-083c13ed-6df68c987f-644fr" event={"ID":"271eca48-7f0b-4619-914c-bbf966c47e3d","Type":"ContainerStarted","Data":"6ab8903e86645043e11ae482d0bb6be570801cfc459e48268b6c129351c24931"}
Oct 30 07:57:19.144837 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:19.144840    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-083c13ed-6df68c987f-644fr" event={"ID":"271eca48-7f0b-4619-914c-bbf966c47e3d","Type":"ContainerStarted","Data":"922d3fd99a976fec2af991f9a932c73c15ca048692ae527b503b36daf51dab4d"}
Oct 30 07:57:19.145436 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:19.144873    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-083c13ed-6df68c987f-644fr"
Oct 30 07:57:19.146592 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:19.146568    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-083c13ed-64db67c4c4-rbvkp" event={"ID":"2eec27f2-9ce9-4269-9c99-0ab3839cf2ee","Type":"ContainerStarted","Data":"199b35fd7270e1550b9ef7751b243a48345d3a35ed4667ac72639e9758ef3292"}
Oct 30 07:57:19.146717 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:19.146612    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-083c13ed-64db67c4c4-rbvkp"
Oct 30 07:57:19.146717 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:19.146627    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-083c13ed-64db67c4c4-rbvkp" event={"ID":"2eec27f2-9ce9-4269-9c99-0ab3839cf2ee","Type":"ContainerStarted","Data":"1337029e4510b4c78f8853e59fa1bee7ad97ad00af24e87db74915d4cd6132ab"}
Oct 30 07:57:19.189646 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:19.189583    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-083c13ed-6df68c987f-644fr" podStartSLOduration=2.189564003 podStartE2EDuration="2.189564003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:57:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:57:19.188673169 +0000 UTC m=+3597.798080139" watchObservedRunningTime="2026-10-30 07:57:19.189564003 +0000 UTC m=+3597.798970974"
Oct 30 07:57:19.234976 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:19.234919    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-083c13ed-64db67c4c4-rbvkp" podStartSLOduration=2.234905044 podStartE2EDuration="2.234905044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:57:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:57:19.23284562 +0000 UTC m=+3597.842252591" watchObservedRunningTime="2026-10-30 07:57:19.234905044 +0000 UTC m=+3597.844312013"
Oct 30 07:57:22.905401 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:22.905358    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-ccac6f-8567799696-99jx8" podUID="8be122aa-a9d8-45f4-b4c7-738496b41217" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 07:57:24.090118 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:24.090081    6224 scope.go:122] "RemoveContainer" containerID="0f6aebc79ccc2eb2a60246c665823b4b209d4c6a56e9fe07c1af4dabfc6f1b77"
Oct 30 07:57:28.839570 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:28.839536    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:57:30.156602 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:30.156573    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-083c13ed-64db67c4c4-rbvkp"
Oct 30 07:57:30.157096 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:30.156643    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-083c13ed-6df68c987f-644fr"
Oct 30 07:57:31.269611 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:31.269317    6224 generic.go:356] "Generic (PLEG): container finished" podID="8be122aa-a9d8-45f4-b4c7-738496b41217" containerID="d0280e3571343e26a6dfb9daa90d7b298304dec5689bc2f05d42f455989e03cf" exitCode=0
Oct 30 07:57:31.269611 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:31.269483    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ccac6f-8567799696-99jx8" event={"ID":"8be122aa-a9d8-45f4-b4c7-738496b41217","Type":"ContainerDied","Data":"d0280e3571343e26a6dfb9daa90d7b298304dec5689bc2f05d42f455989e03cf"}
Oct 30 07:57:31.416761 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:31.416386    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ccac6f-8567799696-99jx8"
Oct 30 07:57:31.430334 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:31.430299    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8be122aa-a9d8-45f4-b4c7-738496b41217-kube-api-access-q5qb8\" (UniqueName: \"kubernetes.io/projected/8be122aa-a9d8-45f4-b4c7-738496b41217-kube-api-access-q5qb8\") pod \"8be122aa-a9d8-45f4-b4c7-738496b41217\" (UID: \"8be122aa-a9d8-45f4-b4c7-738496b41217\") "
Oct 30 07:57:31.431253 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:31.431221    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8be122aa-a9d8-45f4-b4c7-738496b41217-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8be122aa-a9d8-45f4-b4c7-738496b41217-plugins-config-volume\") pod \"8be122aa-a9d8-45f4-b4c7-738496b41217\" (UID: \"8be122aa-a9d8-45f4-b4c7-738496b41217\") "
Oct 30 07:57:31.432390 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:31.432368    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8be122aa-a9d8-45f4-b4c7-738496b41217-plugins-config-volume" pod "8be122aa-a9d8-45f4-b4c7-738496b41217" (UID: "8be122aa-a9d8-45f4-b4c7-738496b41217"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:57:31.444016 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:31.443970    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8be122aa-a9d8-45f4-b4c7-738496b41217-kube-api-access-q5qb8" pod "8be122aa-a9d8-45f4-b4c7-738496b41217" (UID: "8be122aa-a9d8-45f4-b4c7-738496b41217"). InnerVolumeSpecName "kube-api-access-q5qb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:57:31.537098 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:31.535010    6224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8be122aa-a9d8-45f4-b4c7-738496b41217-plugins-config-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:57:31.537098 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:31.535045    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5qb8\" (UniqueName: \"kubernetes.io/projected/8be122aa-a9d8-45f4-b4c7-738496b41217-kube-api-access-q5qb8\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:57:32.145243 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:57:32.145202    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:57:32.410483 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:32.410434    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ccac6f-8567799696-99jx8" event={"ID":"8be122aa-a9d8-45f4-b4c7-738496b41217","Type":"ContainerDied","Data":"edc4760c7d6238fba7f10e2288e6dad91b1430dd276a4d1b5b093a8ffdd8438f"}
Oct 30 07:57:32.438657 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:32.429292    6224 scope.go:122] "RemoveContainer" containerID="d0280e3571343e26a6dfb9daa90d7b298304dec5689bc2f05d42f455989e03cf"
Oct 30 07:57:32.438657 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:32.411201    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ccac6f-8567799696-99jx8"
Oct 30 07:57:32.612846 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:32.605873    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ccac6f-8567799696-99jx8"]
Oct 30 07:57:32.668886 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:32.662911    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ccac6f-8567799696-99jx8"]
Oct 30 07:57:32.864164 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:32.864058    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-06b26e/distinct-w0-06b26e-kserve-545cf57898-2fhhd" podUID="71294bfa-ac2c-40b7-b889-0527daa07f63" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Oct 30 07:57:32.899950 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:32.899906    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2bab4c/simulator-w4-2bab4c-kserve-6db9444549-7nrk8" podUID="c7b09178-9ac7-492f-890b-5ff3da130981" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:57:32.940879 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:32.936964    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/simulator-w3-2eda2c-kserve-56776b9849-9bqzm" podUID="482e6b7c-2231-423c-b54f-a54e56f16ab7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:57:33.510975 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:33.510930    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8be122aa-a9d8-45f4-b4c7-738496b41217" path="/var/lib/kubelet/pods/8be122aa-a9d8-45f4-b4c7-738496b41217/volumes"
Oct 30 07:57:50.993446 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:50.993404    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-896ca4-7c64c58848-895qc"]
Oct 30 07:57:51.027377 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:51.027341    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-896ca4-54fd8b66c5-52wzf"]
Oct 30 07:57:55.993967 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:55.993925    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-896ca4-7c64c58848-895qc" podUID="78d46aa4-8d9e-4094-9e97-5ef71b817928" containerName="payload-processing" containerID="cri-o://9e8c5b845d5a00b692cd9c84a62668c3f0333422dee9f1603a3c6e98abd248f0" gracePeriod=25
Oct 30 07:57:56.028429 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:57:56.028383    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-896ca4-54fd8b66c5-52wzf" podUID="3f7c9bd0-cb40-4bae-b3b7-9301bc73534d" containerName="payload-pre-processing" containerID="cri-o://7ff7beb3018f0e2dbe12a6c2b6b323a61d2a14b3783a0a7fa0c620bb3b5cea8f" gracePeriod=25
Oct 30 07:58:01.857562 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:58:01.857519    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:58:02.314817 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:02.314776    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-896ca4-54fd8b66c5-52wzf" podUID="3f7c9bd0-cb40-4bae-b3b7-9301bc73534d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 07:58:02.314986 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:02.314796    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-896ca4-7c64c58848-895qc" podUID="78d46aa4-8d9e-4094-9e97-5ef71b817928" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 07:58:11.359081 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:11.359050    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-b5998f/sub-test-model-b5998f-kserve-6bc79cd46c-qbgzz"]
Oct 30 07:58:11.359713 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:11.359700    6224 cpu_manager.go:414] "containerMap: removing container" podUID="8be122aa-a9d8-45f4-b4c7-738496b41217" containerName="payload-processing"
Oct 30 07:58:11.359762 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:11.359715    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8be122aa-a9d8-45f4-b4c7-738496b41217" containerName="payload-processing"
Oct 30 07:58:11.359894 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:11.359882    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="8be122aa-a9d8-45f4-b4c7-738496b41217" containerName="payload-processing"
Oct 30 07:58:11.363554 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:11.363538    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-b5998f/sub-test-model-b5998f-kserve-6bc79cd46c-qbgzz"
Oct 30 07:58:11.383339 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:11.383310    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-b5998f/sub-test-model-b5998f-kserve-6bc79cd46c-qbgzz"]
Oct 30 07:58:11.394167 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:11.394142    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-b5998f\"/\"openshift-service-ca.crt\""
Oct 30 07:58:11.394363 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:11.394220    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-b5998f\"/\"sub-test-model-b5998f-kserve-self-signed-certs\""
Oct 30 07:58:11.395988 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:11.395963    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-b5998f\"/\"kube-root-ca.crt\""
Oct 30 07:58:11.396096 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:11.395983    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-b5998f\"/\"default-dockercfg-9tb9r\""
Oct 30 07:58:11.514940 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:11.514901    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58a0459f-c497-4c48-a55a-f848e61dcbda-tmp-dir\") pod \"sub-test-model-b5998f-kserve-6bc79cd46c-qbgzz\" (UID: \"58a0459f-c497-4c48-a55a-f848e61dcbda\") " pod="ai-tenant-e2e-shared-b-w0-b5998f/sub-test-model-b5998f-kserve-6bc79cd46c-qbgzz"
Oct 30 07:58:11.515144 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:11.515042    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58a0459f-c497-4c48-a55a-f848e61dcbda-dshm\") pod \"sub-test-model-b5998f-kserve-6bc79cd46c-qbgzz\" (UID: \"58a0459f-c497-4c48-a55a-f848e61dcbda\") " pod="ai-tenant-e2e-shared-b-w0-b5998f/sub-test-model-b5998f-kserve-6bc79cd46c-qbgzz"
Oct 30 07:58:11.515144 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:11.515105    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxsft\" (UniqueName: \"kubernetes.io/projected/58a0459f-c497-4c48-a55a-f848e61dcbda-kube-api-access-xxsft\") pod \"sub-test-model-b5998f-kserve-6bc79cd46c-qbgzz\" (UID: \"58a0459f-c497-4c48-a55a-f848e61dcbda\") " pod="ai-tenant-e2e-shared-b-w0-b5998f/sub-test-model-b5998f-kserve-6bc79cd46c-qbgzz"
Oct 30 07:58:11.515235 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:11.515222    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58a0459f-c497-4c48-a55a-f848e61dcbda-home\") pod \"sub-test-model-b5998f-kserve-6bc79cd46c-qbgzz\" (UID: \"58a0459f-c497-4c48-a55a-f848e61dcbda\") " pod="ai-tenant-e2e-shared-b-w0-b5998f/sub-test-model-b5998f-kserve-6bc79cd46c-qbgzz"
Oct 30 07:58:11.515350 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:11.515334    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58a0459f-c497-4c48-a55a-f848e61dcbda-tls-certs\") pod \"sub-test-model-b5998f-kserve-6bc79cd46c-qbgzz\" (UID: \"58a0459f-c497-4c48-a55a-f848e61dcbda\") " pod="ai-tenant-e2e-shared-b-w0-b5998f/sub-test-model-b5998f-kserve-6bc79cd46c-qbgzz"
Oct 30 07:58:11.515410 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:11.515384    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58a0459f-c497-4c48-a55a-f848e61dcbda-model-cache\") pod \"sub-test-model-b5998f-kserve-6bc79cd46c-qbgzz\" (UID: \"58a0459f-c497-4c48-a55a-f848e61dcbda\") " pod="ai-tenant-e2e-shared-b-w0-b5998f/sub-test-model-b5998f-kserve-6bc79cd46c-qbgzz"
Oct 30 07:58:11.616846 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:11.616705    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58a0459f-c497-4c48-a55a-f848e61dcbda-dshm\") pod \"sub-test-model-b5998f-kserve-6bc79cd46c-qbgzz\" (UID: \"58a0459f-c497-4c48-a55a-f848e61dcbda\") " pod="ai-tenant-e2e-shared-b-w0-b5998f/sub-test-model-b5998f-kserve-6bc79cd46c-qbgzz"
Oct 30 07:58:11.617213 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:11.617175    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxsft\" (UniqueName: \"kubernetes.io/projected/58a0459f-c497-4c48-a55a-f848e61dcbda-kube-api-access-xxsft\") pod \"sub-test-model-b5998f-kserve-6bc79cd46c-qbgzz\" (UID: \"58a0459f-c497-4c48-a55a-f848e61dcbda\") " pod="ai-tenant-e2e-shared-b-w0-b5998f/sub-test-model-b5998f-kserve-6bc79cd46c-qbgzz"
Oct 30 07:58:11.617430 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:11.617412    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58a0459f-c497-4c48-a55a-f848e61dcbda-home\") pod \"sub-test-model-b5998f-kserve-6bc79cd46c-qbgzz\" (UID: \"58a0459f-c497-4c48-a55a-f848e61dcbda\") " pod="ai-tenant-e2e-shared-b-w0-b5998f/sub-test-model-b5998f-kserve-6bc79cd46c-qbgzz"
Oct 30 07:58:11.617661 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:11.617638    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58a0459f-c497-4c48-a55a-f848e61dcbda-tls-certs\") pod \"sub-test-model-b5998f-kserve-6bc79cd46c-qbgzz\" (UID: \"58a0459f-c497-4c48-a55a-f848e61dcbda\") " pod="ai-tenant-e2e-shared-b-w0-b5998f/sub-test-model-b5998f-kserve-6bc79cd46c-qbgzz"
Oct 30 07:58:11.617880 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:11.617847    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58a0459f-c497-4c48-a55a-f848e61dcbda-model-cache\") pod \"sub-test-model-b5998f-kserve-6bc79cd46c-qbgzz\" (UID: \"58a0459f-c497-4c48-a55a-f848e61dcbda\") " pod="ai-tenant-e2e-shared-b-w0-b5998f/sub-test-model-b5998f-kserve-6bc79cd46c-qbgzz"
Oct 30 07:58:11.618149 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:11.618131    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58a0459f-c497-4c48-a55a-f848e61dcbda-tmp-dir\") pod \"sub-test-model-b5998f-kserve-6bc79cd46c-qbgzz\" (UID: \"58a0459f-c497-4c48-a55a-f848e61dcbda\") " pod="ai-tenant-e2e-shared-b-w0-b5998f/sub-test-model-b5998f-kserve-6bc79cd46c-qbgzz"
Oct 30 07:58:11.620767 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:11.618851    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58a0459f-c497-4c48-a55a-f848e61dcbda-home\") pod \"sub-test-model-b5998f-kserve-6bc79cd46c-qbgzz\" (UID: \"58a0459f-c497-4c48-a55a-f848e61dcbda\") " pod="ai-tenant-e2e-shared-b-w0-b5998f/sub-test-model-b5998f-kserve-6bc79cd46c-qbgzz"
Oct 30 07:58:11.620767 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:11.619305    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58a0459f-c497-4c48-a55a-f848e61dcbda-model-cache\") pod \"sub-test-model-b5998f-kserve-6bc79cd46c-qbgzz\" (UID: \"58a0459f-c497-4c48-a55a-f848e61dcbda\") " pod="ai-tenant-e2e-shared-b-w0-b5998f/sub-test-model-b5998f-kserve-6bc79cd46c-qbgzz"
Oct 30 07:58:11.620767 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:11.619714    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58a0459f-c497-4c48-a55a-f848e61dcbda-tmp-dir\") pod \"sub-test-model-b5998f-kserve-6bc79cd46c-qbgzz\" (UID: \"58a0459f-c497-4c48-a55a-f848e61dcbda\") " pod="ai-tenant-e2e-shared-b-w0-b5998f/sub-test-model-b5998f-kserve-6bc79cd46c-qbgzz"
Oct 30 07:58:11.637593 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:11.637560    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58a0459f-c497-4c48-a55a-f848e61dcbda-dshm\") pod \"sub-test-model-b5998f-kserve-6bc79cd46c-qbgzz\" (UID: \"58a0459f-c497-4c48-a55a-f848e61dcbda\") " pod="ai-tenant-e2e-shared-b-w0-b5998f/sub-test-model-b5998f-kserve-6bc79cd46c-qbgzz"
Oct 30 07:58:11.641620 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:11.641587    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58a0459f-c497-4c48-a55a-f848e61dcbda-tls-certs\") pod \"sub-test-model-b5998f-kserve-6bc79cd46c-qbgzz\" (UID: \"58a0459f-c497-4c48-a55a-f848e61dcbda\") " pod="ai-tenant-e2e-shared-b-w0-b5998f/sub-test-model-b5998f-kserve-6bc79cd46c-qbgzz"
Oct 30 07:58:11.643910 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:11.643882    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxsft\" (UniqueName: \"kubernetes.io/projected/58a0459f-c497-4c48-a55a-f848e61dcbda-kube-api-access-xxsft\") pod \"sub-test-model-b5998f-kserve-6bc79cd46c-qbgzz\" (UID: \"58a0459f-c497-4c48-a55a-f848e61dcbda\") " pod="ai-tenant-e2e-shared-b-w0-b5998f/sub-test-model-b5998f-kserve-6bc79cd46c-qbgzz"
Oct 30 07:58:11.677655 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:11.677617    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-b5998f/sub-test-model-b5998f-kserve-6bc79cd46c-qbgzz"
Oct 30 07:58:11.990381 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:58:11.990342    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod58a0459f_c497_4c48_a55a_f848e61dcbda.slice/crio-9e3eeddbc1f27238209a24eb117c9abcf2e76bc7cfdc7412c388c4006390b04b WatchSource:0}: Error finding container 9e3eeddbc1f27238209a24eb117c9abcf2e76bc7cfdc7412c388c4006390b04b: Status 404 returned error can't find the container with id 9e3eeddbc1f27238209a24eb117c9abcf2e76bc7cfdc7412c388c4006390b04b
Oct 30 07:58:11.991607 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:11.991574    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-b5998f/sub-test-model-b5998f-kserve-6bc79cd46c-qbgzz"]
Oct 30 07:58:12.315069 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:12.315026    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-896ca4-54fd8b66c5-52wzf" podUID="3f7c9bd0-cb40-4bae-b3b7-9301bc73534d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 07:58:12.315260 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:12.315028    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-896ca4-7c64c58848-895qc" podUID="78d46aa4-8d9e-4094-9e97-5ef71b817928" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 07:58:12.723475 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:12.723391    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-b5998f/sub-test-model-b5998f-kserve-6bc79cd46c-qbgzz" event={"ID":"58a0459f-c497-4c48-a55a-f848e61dcbda","Type":"ContainerStarted","Data":"2f02c807ccddd4437b3ada3a3e48a516e520ef58b9ed2f8529896b647efc0d57"}
Oct 30 07:58:12.723475 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:12.723432    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-b5998f/sub-test-model-b5998f-kserve-6bc79cd46c-qbgzz" event={"ID":"58a0459f-c497-4c48-a55a-f848e61dcbda","Type":"ContainerStarted","Data":"9e3eeddbc1f27238209a24eb117c9abcf2e76bc7cfdc7412c388c4006390b04b"}
Oct 30 07:58:12.769413 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:12.769365    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-b5998f/sub-test-model-b5998f-kserve-6bc79cd46c-qbgzz" podStartSLOduration=1.769352051 podStartE2EDuration="1.769352051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:58:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:58:12.767895412 +0000 UTC m=+3651.377302383" watchObservedRunningTime="2026-10-30 07:58:12.769352051 +0000 UTC m=+3651.378759021"
Oct 30 07:58:13.678436 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:13.678403    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-b5998f/sub-test-model-b5998f-kserve-6bc79cd46c-qbgzz"
Oct 30 07:58:16.322494 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:16.322466    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-896ca4-7c64c58848-895qc"
Oct 30 07:58:16.338217 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:16.338191    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-896ca4-54fd8b66c5-52wzf"
Oct 30 07:58:16.483442 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:16.483404    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78d46aa4-8d9e-4094-9e97-5ef71b817928-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78d46aa4-8d9e-4094-9e97-5ef71b817928-plugins-config-volume\") pod \"78d46aa4-8d9e-4094-9e97-5ef71b817928\" (UID: \"78d46aa4-8d9e-4094-9e97-5ef71b817928\") "
Oct 30 07:58:16.483619 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:16.483463    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78d46aa4-8d9e-4094-9e97-5ef71b817928-kube-api-access-5clhk\" (UniqueName: \"kubernetes.io/projected/78d46aa4-8d9e-4094-9e97-5ef71b817928-kube-api-access-5clhk\") pod \"78d46aa4-8d9e-4094-9e97-5ef71b817928\" (UID: \"78d46aa4-8d9e-4094-9e97-5ef71b817928\") "
Oct 30 07:58:16.483619 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:16.483562    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f7c9bd0-cb40-4bae-b3b7-9301bc73534d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f7c9bd0-cb40-4bae-b3b7-9301bc73534d-plugins-config-volume\") pod \"3f7c9bd0-cb40-4bae-b3b7-9301bc73534d\" (UID: \"3f7c9bd0-cb40-4bae-b3b7-9301bc73534d\") "
Oct 30 07:58:16.483696 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:16.483623    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f7c9bd0-cb40-4bae-b3b7-9301bc73534d-kube-api-access-dsfjb\" (UniqueName: \"kubernetes.io/projected/3f7c9bd0-cb40-4bae-b3b7-9301bc73534d-kube-api-access-dsfjb\") pod \"3f7c9bd0-cb40-4bae-b3b7-9301bc73534d\" (UID: \"3f7c9bd0-cb40-4bae-b3b7-9301bc73534d\") "
Oct 30 07:58:16.485284 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:16.485253    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78d46aa4-8d9e-4094-9e97-5ef71b817928-plugins-config-volume" pod "78d46aa4-8d9e-4094-9e97-5ef71b817928" (UID: "78d46aa4-8d9e-4094-9e97-5ef71b817928"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:58:16.485490 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:16.485281    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f7c9bd0-cb40-4bae-b3b7-9301bc73534d-plugins-config-volume" pod "3f7c9bd0-cb40-4bae-b3b7-9301bc73534d" (UID: "3f7c9bd0-cb40-4bae-b3b7-9301bc73534d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:58:16.486427 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:16.486397    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f7c9bd0-cb40-4bae-b3b7-9301bc73534d-kube-api-access-dsfjb" pod "3f7c9bd0-cb40-4bae-b3b7-9301bc73534d" (UID: "3f7c9bd0-cb40-4bae-b3b7-9301bc73534d"). InnerVolumeSpecName "kube-api-access-dsfjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:58:16.486626 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:16.486604    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78d46aa4-8d9e-4094-9e97-5ef71b817928-kube-api-access-5clhk" pod "78d46aa4-8d9e-4094-9e97-5ef71b817928" (UID: "78d46aa4-8d9e-4094-9e97-5ef71b817928"). InnerVolumeSpecName "kube-api-access-5clhk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:58:16.585334 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:16.585292    6224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f7c9bd0-cb40-4bae-b3b7-9301bc73534d-plugins-config-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:58:16.585334 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:16.585335    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsfjb\" (UniqueName: \"kubernetes.io/projected/3f7c9bd0-cb40-4bae-b3b7-9301bc73534d-kube-api-access-dsfjb\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:58:16.585564 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:16.585350    6224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78d46aa4-8d9e-4094-9e97-5ef71b817928-plugins-config-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:58:16.585564 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:16.585364    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5clhk\" (UniqueName: \"kubernetes.io/projected/78d46aa4-8d9e-4094-9e97-5ef71b817928-kube-api-access-5clhk\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:58:16.751419 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:16.751383    6224 generic.go:356] "Generic (PLEG): container finished" podID="3f7c9bd0-cb40-4bae-b3b7-9301bc73534d" containerID="7ff7beb3018f0e2dbe12a6c2b6b323a61d2a14b3783a0a7fa0c620bb3b5cea8f" exitCode=0
Oct 30 07:58:16.751587 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:16.751431    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-896ca4-54fd8b66c5-52wzf"
Oct 30 07:58:16.751587 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:16.751468    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-896ca4-54fd8b66c5-52wzf" event={"ID":"3f7c9bd0-cb40-4bae-b3b7-9301bc73534d","Type":"ContainerDied","Data":"7ff7beb3018f0e2dbe12a6c2b6b323a61d2a14b3783a0a7fa0c620bb3b5cea8f"}
Oct 30 07:58:16.751587 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:16.751533    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-896ca4-54fd8b66c5-52wzf" event={"ID":"3f7c9bd0-cb40-4bae-b3b7-9301bc73534d","Type":"ContainerDied","Data":"478d0ea6362029dee7bac09a8cfcb135b84e98982ca70a932bf5226cc2589590"}
Oct 30 07:58:16.751587 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:16.751557    6224 scope.go:122] "RemoveContainer" containerID="7ff7beb3018f0e2dbe12a6c2b6b323a61d2a14b3783a0a7fa0c620bb3b5cea8f"
Oct 30 07:58:16.753523 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:16.753496    6224 generic.go:356] "Generic (PLEG): container finished" podID="78d46aa4-8d9e-4094-9e97-5ef71b817928" containerID="9e8c5b845d5a00b692cd9c84a62668c3f0333422dee9f1603a3c6e98abd248f0" exitCode=0
Oct 30 07:58:16.753659 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:16.753547    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-896ca4-7c64c58848-895qc" event={"ID":"78d46aa4-8d9e-4094-9e97-5ef71b817928","Type":"ContainerDied","Data":"9e8c5b845d5a00b692cd9c84a62668c3f0333422dee9f1603a3c6e98abd248f0"}
Oct 30 07:58:16.753659 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:16.753578    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-896ca4-7c64c58848-895qc" event={"ID":"78d46aa4-8d9e-4094-9e97-5ef71b817928","Type":"ContainerDied","Data":"ae3a359df5380fce63a8f944f95e4185c535b2b2d996edfc85ca5a2f6f8948e4"}
Oct 30 07:58:16.753659 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:16.753651    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-896ca4-7c64c58848-895qc"
Oct 30 07:58:16.768099 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:16.768082    6224 scope.go:122] "RemoveContainer" containerID="7ff7beb3018f0e2dbe12a6c2b6b323a61d2a14b3783a0a7fa0c620bb3b5cea8f"
Oct 30 07:58:16.769817 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:58:16.769778    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ff7beb3018f0e2dbe12a6c2b6b323a61d2a14b3783a0a7fa0c620bb3b5cea8f\": container with ID starting with 7ff7beb3018f0e2dbe12a6c2b6b323a61d2a14b3783a0a7fa0c620bb3b5cea8f not found: ID does not exist" containerID="7ff7beb3018f0e2dbe12a6c2b6b323a61d2a14b3783a0a7fa0c620bb3b5cea8f"
Oct 30 07:58:16.769976 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:16.769950    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ff7beb3018f0e2dbe12a6c2b6b323a61d2a14b3783a0a7fa0c620bb3b5cea8f"} err="failed to get container status \"7ff7beb3018f0e2dbe12a6c2b6b323a61d2a14b3783a0a7fa0c620bb3b5cea8f\": rpc error: code = NotFound desc = could not find container \"7ff7beb3018f0e2dbe12a6c2b6b323a61d2a14b3783a0a7fa0c620bb3b5cea8f\": container with ID starting with 7ff7beb3018f0e2dbe12a6c2b6b323a61d2a14b3783a0a7fa0c620bb3b5cea8f not found: ID does not exist"
Oct 30 07:58:16.770089 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:16.770071    6224 scope.go:122] "RemoveContainer" containerID="9e8c5b845d5a00b692cd9c84a62668c3f0333422dee9f1603a3c6e98abd248f0"
Oct 30 07:58:16.781788 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:16.781762    6224 scope.go:122] "RemoveContainer" containerID="9e8c5b845d5a00b692cd9c84a62668c3f0333422dee9f1603a3c6e98abd248f0"
Oct 30 07:58:16.782098 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:58:16.782079    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e8c5b845d5a00b692cd9c84a62668c3f0333422dee9f1603a3c6e98abd248f0\": container with ID starting with 9e8c5b845d5a00b692cd9c84a62668c3f0333422dee9f1603a3c6e98abd248f0 not found: ID does not exist" containerID="9e8c5b845d5a00b692cd9c84a62668c3f0333422dee9f1603a3c6e98abd248f0"
Oct 30 07:58:16.782179 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:16.782109    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e8c5b845d5a00b692cd9c84a62668c3f0333422dee9f1603a3c6e98abd248f0"} err="failed to get container status \"9e8c5b845d5a00b692cd9c84a62668c3f0333422dee9f1603a3c6e98abd248f0\": rpc error: code = NotFound desc = could not find container \"9e8c5b845d5a00b692cd9c84a62668c3f0333422dee9f1603a3c6e98abd248f0\": container with ID starting with 9e8c5b845d5a00b692cd9c84a62668c3f0333422dee9f1603a3c6e98abd248f0 not found: ID does not exist"
Oct 30 07:58:16.810680 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:16.810644    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-896ca4-7c64c58848-895qc"]
Oct 30 07:58:16.815998 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:16.815966    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-896ca4-7c64c58848-895qc"]
Oct 30 07:58:16.853388 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:16.853287    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-896ca4-54fd8b66c5-52wzf"]
Oct 30 07:58:16.857548 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:16.857513    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-896ca4-54fd8b66c5-52wzf"]
Oct 30 07:58:17.504818 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:17.504782    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f7c9bd0-cb40-4bae-b3b7-9301bc73534d" path="/var/lib/kubelet/pods/3f7c9bd0-cb40-4bae-b3b7-9301bc73534d/volumes"
Oct 30 07:58:17.505272 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:17.505256    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78d46aa4-8d9e-4094-9e97-5ef71b817928" path="/var/lib/kubelet/pods/78d46aa4-8d9e-4094-9e97-5ef71b817928/volumes"
Oct 30 07:58:18.121366 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:18.121325    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:58:21.678872 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:21.678839    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-b5998f/sub-test-model-b5998f-kserve-6bc79cd46c-qbgzz"
Oct 30 07:58:21.692342 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:21.692312    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-b5998f/sub-test-model-b5998f-kserve-6bc79cd46c-qbgzz"
Oct 30 07:58:21.813531 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:21.813497    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-b5998f/sub-test-model-b5998f-kserve-6bc79cd46c-qbgzz"
Oct 30 07:58:31.858315 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:58:31.858277    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:58:35.743470 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:35.743426    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-932c00/rate-test-model-932c00-kserve-59bbdb5c89-vgxtp"]
Oct 30 07:58:35.744384 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:35.744348    6224 cpu_manager.go:414] "containerMap: removing container" podUID="3f7c9bd0-cb40-4bae-b3b7-9301bc73534d" containerName="payload-pre-processing"
Oct 30 07:58:35.744384 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:35.744377    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f7c9bd0-cb40-4bae-b3b7-9301bc73534d" containerName="payload-pre-processing"
Oct 30 07:58:35.744550 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:35.744415    6224 cpu_manager.go:414] "containerMap: removing container" podUID="78d46aa4-8d9e-4094-9e97-5ef71b817928" containerName="payload-processing"
Oct 30 07:58:35.744550 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:35.744425    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78d46aa4-8d9e-4094-9e97-5ef71b817928" containerName="payload-processing"
Oct 30 07:58:35.744622 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:35.744592    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f7c9bd0-cb40-4bae-b3b7-9301bc73534d" containerName="payload-pre-processing"
Oct 30 07:58:35.744622 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:35.744611    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="78d46aa4-8d9e-4094-9e97-5ef71b817928" containerName="payload-processing"
Oct 30 07:58:35.749048 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:35.749028    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-932c00/rate-test-model-932c00-kserve-59bbdb5c89-vgxtp"
Oct 30 07:58:35.777709 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:35.777676    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-932c00/rate-test-model-932c00-kserve-59bbdb5c89-vgxtp"]
Oct 30 07:58:35.786415 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:35.786386    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-932c00\"/\"rate-test-model-932c00-kserve-self-signed-certs\""
Oct 30 07:58:35.786994 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:35.786970    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-932c00\"/\"openshift-service-ca.crt\""
Oct 30 07:58:35.791888 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:35.791856    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-932c00\"/\"default-dockercfg-9fsbl\""
Oct 30 07:58:35.791888 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:35.791866    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-932c00\"/\"kube-root-ca.crt\""
Oct 30 07:58:35.927360 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:35.927327    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ad02f3d-27a3-4934-b4eb-60b39ab74aa8-tmp-dir\") pod \"rate-test-model-932c00-kserve-59bbdb5c89-vgxtp\" (UID: \"6ad02f3d-27a3-4934-b4eb-60b39ab74aa8\") " pod="ai-tenant-e2e-shared-a-w2-932c00/rate-test-model-932c00-kserve-59bbdb5c89-vgxtp"
Oct 30 07:58:35.927554 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:35.927500    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ad02f3d-27a3-4934-b4eb-60b39ab74aa8-home\") pod \"rate-test-model-932c00-kserve-59bbdb5c89-vgxtp\" (UID: \"6ad02f3d-27a3-4934-b4eb-60b39ab74aa8\") " pod="ai-tenant-e2e-shared-a-w2-932c00/rate-test-model-932c00-kserve-59bbdb5c89-vgxtp"
Oct 30 07:58:35.927638 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:35.927562    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ad02f3d-27a3-4934-b4eb-60b39ab74aa8-dshm\") pod \"rate-test-model-932c00-kserve-59bbdb5c89-vgxtp\" (UID: \"6ad02f3d-27a3-4934-b4eb-60b39ab74aa8\") " pod="ai-tenant-e2e-shared-a-w2-932c00/rate-test-model-932c00-kserve-59bbdb5c89-vgxtp"
Oct 30 07:58:35.927701 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:35.927660    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ad02f3d-27a3-4934-b4eb-60b39ab74aa8-tls-certs\") pod \"rate-test-model-932c00-kserve-59bbdb5c89-vgxtp\" (UID: \"6ad02f3d-27a3-4934-b4eb-60b39ab74aa8\") " pod="ai-tenant-e2e-shared-a-w2-932c00/rate-test-model-932c00-kserve-59bbdb5c89-vgxtp"
Oct 30 07:58:35.927768 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:35.927726    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8btw4\" (UniqueName: \"kubernetes.io/projected/6ad02f3d-27a3-4934-b4eb-60b39ab74aa8-kube-api-access-8btw4\") pod \"rate-test-model-932c00-kserve-59bbdb5c89-vgxtp\" (UID: \"6ad02f3d-27a3-4934-b4eb-60b39ab74aa8\") " pod="ai-tenant-e2e-shared-a-w2-932c00/rate-test-model-932c00-kserve-59bbdb5c89-vgxtp"
Oct 30 07:58:35.927768 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:35.927758    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ad02f3d-27a3-4934-b4eb-60b39ab74aa8-model-cache\") pod \"rate-test-model-932c00-kserve-59bbdb5c89-vgxtp\" (UID: \"6ad02f3d-27a3-4934-b4eb-60b39ab74aa8\") " pod="ai-tenant-e2e-shared-a-w2-932c00/rate-test-model-932c00-kserve-59bbdb5c89-vgxtp"
Oct 30 07:58:36.028955 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:36.028866    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ad02f3d-27a3-4934-b4eb-60b39ab74aa8-tmp-dir\") pod \"rate-test-model-932c00-kserve-59bbdb5c89-vgxtp\" (UID: \"6ad02f3d-27a3-4934-b4eb-60b39ab74aa8\") " pod="ai-tenant-e2e-shared-a-w2-932c00/rate-test-model-932c00-kserve-59bbdb5c89-vgxtp"
Oct 30 07:58:36.029828 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:36.029786    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ad02f3d-27a3-4934-b4eb-60b39ab74aa8-home\") pod \"rate-test-model-932c00-kserve-59bbdb5c89-vgxtp\" (UID: \"6ad02f3d-27a3-4934-b4eb-60b39ab74aa8\") " pod="ai-tenant-e2e-shared-a-w2-932c00/rate-test-model-932c00-kserve-59bbdb5c89-vgxtp"
Oct 30 07:58:36.030268 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:36.030250    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ad02f3d-27a3-4934-b4eb-60b39ab74aa8-dshm\") pod \"rate-test-model-932c00-kserve-59bbdb5c89-vgxtp\" (UID: \"6ad02f3d-27a3-4934-b4eb-60b39ab74aa8\") " pod="ai-tenant-e2e-shared-a-w2-932c00/rate-test-model-932c00-kserve-59bbdb5c89-vgxtp"
Oct 30 07:58:36.030418 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:36.030391    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ad02f3d-27a3-4934-b4eb-60b39ab74aa8-tls-certs\") pod \"rate-test-model-932c00-kserve-59bbdb5c89-vgxtp\" (UID: \"6ad02f3d-27a3-4934-b4eb-60b39ab74aa8\") " pod="ai-tenant-e2e-shared-a-w2-932c00/rate-test-model-932c00-kserve-59bbdb5c89-vgxtp"
Oct 30 07:58:36.030502 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:36.030446    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8btw4\" (UniqueName: \"kubernetes.io/projected/6ad02f3d-27a3-4934-b4eb-60b39ab74aa8-kube-api-access-8btw4\") pod \"rate-test-model-932c00-kserve-59bbdb5c89-vgxtp\" (UID: \"6ad02f3d-27a3-4934-b4eb-60b39ab74aa8\") " pod="ai-tenant-e2e-shared-a-w2-932c00/rate-test-model-932c00-kserve-59bbdb5c89-vgxtp"
Oct 30 07:58:36.030502 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:36.030478    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ad02f3d-27a3-4934-b4eb-60b39ab74aa8-model-cache\") pod \"rate-test-model-932c00-kserve-59bbdb5c89-vgxtp\" (UID: \"6ad02f3d-27a3-4934-b4eb-60b39ab74aa8\") " pod="ai-tenant-e2e-shared-a-w2-932c00/rate-test-model-932c00-kserve-59bbdb5c89-vgxtp"
Oct 30 07:58:36.030635 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:36.029629    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ad02f3d-27a3-4934-b4eb-60b39ab74aa8-tmp-dir\") pod \"rate-test-model-932c00-kserve-59bbdb5c89-vgxtp\" (UID: \"6ad02f3d-27a3-4934-b4eb-60b39ab74aa8\") " pod="ai-tenant-e2e-shared-a-w2-932c00/rate-test-model-932c00-kserve-59bbdb5c89-vgxtp"
Oct 30 07:58:36.030734 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:36.030176    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ad02f3d-27a3-4934-b4eb-60b39ab74aa8-home\") pod \"rate-test-model-932c00-kserve-59bbdb5c89-vgxtp\" (UID: \"6ad02f3d-27a3-4934-b4eb-60b39ab74aa8\") " pod="ai-tenant-e2e-shared-a-w2-932c00/rate-test-model-932c00-kserve-59bbdb5c89-vgxtp"
Oct 30 07:58:36.030951 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:36.030928    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ad02f3d-27a3-4934-b4eb-60b39ab74aa8-model-cache\") pod \"rate-test-model-932c00-kserve-59bbdb5c89-vgxtp\" (UID: \"6ad02f3d-27a3-4934-b4eb-60b39ab74aa8\") " pod="ai-tenant-e2e-shared-a-w2-932c00/rate-test-model-932c00-kserve-59bbdb5c89-vgxtp"
Oct 30 07:58:36.034286 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:36.034258    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ad02f3d-27a3-4934-b4eb-60b39ab74aa8-tls-certs\") pod \"rate-test-model-932c00-kserve-59bbdb5c89-vgxtp\" (UID: \"6ad02f3d-27a3-4934-b4eb-60b39ab74aa8\") " pod="ai-tenant-e2e-shared-a-w2-932c00/rate-test-model-932c00-kserve-59bbdb5c89-vgxtp"
Oct 30 07:58:36.049427 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:36.049086    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ad02f3d-27a3-4934-b4eb-60b39ab74aa8-dshm\") pod \"rate-test-model-932c00-kserve-59bbdb5c89-vgxtp\" (UID: \"6ad02f3d-27a3-4934-b4eb-60b39ab74aa8\") " pod="ai-tenant-e2e-shared-a-w2-932c00/rate-test-model-932c00-kserve-59bbdb5c89-vgxtp"
Oct 30 07:58:36.064560 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:36.064500    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8btw4\" (UniqueName: \"kubernetes.io/projected/6ad02f3d-27a3-4934-b4eb-60b39ab74aa8-kube-api-access-8btw4\") pod \"rate-test-model-932c00-kserve-59bbdb5c89-vgxtp\" (UID: \"6ad02f3d-27a3-4934-b4eb-60b39ab74aa8\") " pod="ai-tenant-e2e-shared-a-w2-932c00/rate-test-model-932c00-kserve-59bbdb5c89-vgxtp"
Oct 30 07:58:36.362211 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:36.362177    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-932c00/rate-test-model-932c00-kserve-59bbdb5c89-vgxtp"
Oct 30 07:58:36.666028 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:36.666002    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-932c00/rate-test-model-932c00-kserve-59bbdb5c89-vgxtp"]
Oct 30 07:58:36.666028 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:58:36.665999    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ad02f3d_27a3_4934_b4eb_60b39ab74aa8.slice/crio-d35dd2083e3d9eb848fd75fa8bc120678a12f058cf3e11b8169a0fd4cdc68cbf WatchSource:0}: Error finding container d35dd2083e3d9eb848fd75fa8bc120678a12f058cf3e11b8169a0fd4cdc68cbf: Status 404 returned error can't find the container with id d35dd2083e3d9eb848fd75fa8bc120678a12f058cf3e11b8169a0fd4cdc68cbf
Oct 30 07:58:36.923114 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:36.923025    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-932c00/rate-test-model-932c00-kserve-59bbdb5c89-vgxtp" event={"ID":"6ad02f3d-27a3-4934-b4eb-60b39ab74aa8","Type":"ContainerStarted","Data":"0a7c406a5b15f9173bb5674e56ea47849b8e054828945d956120d7f9c7497d6c"}
Oct 30 07:58:36.923114 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:36.923069    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-932c00/rate-test-model-932c00-kserve-59bbdb5c89-vgxtp" event={"ID":"6ad02f3d-27a3-4934-b4eb-60b39ab74aa8","Type":"ContainerStarted","Data":"d35dd2083e3d9eb848fd75fa8bc120678a12f058cf3e11b8169a0fd4cdc68cbf"}
Oct 30 07:58:36.974672 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:36.974615    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-932c00/rate-test-model-932c00-kserve-59bbdb5c89-vgxtp" podStartSLOduration=1.974601685 podStartE2EDuration="1.974601685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:58:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:58:36.97268403 +0000 UTC m=+3675.582091002" watchObservedRunningTime="2026-10-30 07:58:36.974601685 +0000 UTC m=+3675.584008674"
Oct 30 07:58:37.362981 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:37.362946    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-932c00/rate-test-model-932c00-kserve-59bbdb5c89-vgxtp"
Oct 30 07:58:43.095859 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:43.092366    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688"]
Oct 30 07:58:43.095859 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:43.092697    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688" podUID="ecd9f1bd-d4f6-468f-9994-203a3620a1a3" containerName="istio-proxy" containerID="cri-o://361ac3190018129223fe4d8c7a12945a7d8ca60cbacfea0cf70ee00679ac1482" gracePeriod=30
Oct 30 07:58:44.781125 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:44.779895    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2bab4c/simulator-w4-2bab4c-kserve-6db9444549-7nrk8" podUID="c7b09178-9ac7-492f-890b-5ff3da130981" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:58:44.816950 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:44.816901    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-b5998f/sub-test-model-b5998f-kserve-6bc79cd46c-qbgzz" podUID="58a0459f-c497-4c48-a55a-f848e61dcbda" containerName="main" probeResult="failure" output="Get \"https://10.134.0.216:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:58:44.861977 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:44.861900    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-06b26e/distinct-w0-06b26e-kserve-545cf57898-2fhhd" podUID="71294bfa-ac2c-40b7-b889-0527daa07f63" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:58:44.862644 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:44.862613    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2eda2c/simulator-w3-2eda2c-kserve-56776b9849-9bqzm" podUID="482e6b7c-2231-423c-b54f-a54e56f16ab7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded"
Oct 30 07:58:44.864232 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:44.864203    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2bab4c/distinct-w4-2bab4c-kserve-6868d767c9-j7wlm" podUID="11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:58:45.415398 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:45.415343    6224 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-2bab4c/distinct-w4-2bab4c-kserve-6868d767c9-j7wlm" podUID="11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 07:58:46.363825 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:46.363314    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-932c00/rate-test-model-932c00-kserve-59bbdb5c89-vgxtp"
Oct 30 07:58:46.380467 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:46.380436    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-932c00/rate-test-model-932c00-kserve-59bbdb5c89-vgxtp"
Oct 30 07:58:47.099497 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:47.099467    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-932c00/rate-test-model-932c00-kserve-59bbdb5c89-vgxtp"
Oct 30 07:58:48.511402 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:48.511362    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688"
Oct 30 07:58:48.554681 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:48.554655    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecd9f1bd-d4f6-468f-9994-203a3620a1a3-istio-token\" (UniqueName: \"kubernetes.io/projected/ecd9f1bd-d4f6-468f-9994-203a3620a1a3-istio-token\") pod \"ecd9f1bd-d4f6-468f-9994-203a3620a1a3\" (UID: \"ecd9f1bd-d4f6-468f-9994-203a3620a1a3\") "
Oct 30 07:58:48.554851 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:48.554698    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecd9f1bd-d4f6-468f-9994-203a3620a1a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecd9f1bd-d4f6-468f-9994-203a3620a1a3-istiod-ca-cert\") pod \"ecd9f1bd-d4f6-468f-9994-203a3620a1a3\" (UID: \"ecd9f1bd-d4f6-468f-9994-203a3620a1a3\") "
Oct 30 07:58:48.554851 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:48.554756    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecd9f1bd-d4f6-468f-9994-203a3620a1a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecd9f1bd-d4f6-468f-9994-203a3620a1a3-credential-socket\") pod \"ecd9f1bd-d4f6-468f-9994-203a3620a1a3\" (UID: \"ecd9f1bd-d4f6-468f-9994-203a3620a1a3\") "
Oct 30 07:58:48.554959 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:48.554917    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecd9f1bd-d4f6-468f-9994-203a3620a1a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecd9f1bd-d4f6-468f-9994-203a3620a1a3-workload-socket\") pod \"ecd9f1bd-d4f6-468f-9994-203a3620a1a3\" (UID: \"ecd9f1bd-d4f6-468f-9994-203a3620a1a3\") "
Oct 30 07:58:48.555016 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:48.554975    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecd9f1bd-d4f6-468f-9994-203a3620a1a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecd9f1bd-d4f6-468f-9994-203a3620a1a3-istio-data\") pod \"ecd9f1bd-d4f6-468f-9994-203a3620a1a3\" (UID: \"ecd9f1bd-d4f6-468f-9994-203a3620a1a3\") "
Oct 30 07:58:48.555078 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:48.555027    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecd9f1bd-d4f6-468f-9994-203a3620a1a3-kube-api-access-mbptl\" (UniqueName: \"kubernetes.io/projected/ecd9f1bd-d4f6-468f-9994-203a3620a1a3-kube-api-access-mbptl\") pod \"ecd9f1bd-d4f6-468f-9994-203a3620a1a3\" (UID: \"ecd9f1bd-d4f6-468f-9994-203a3620a1a3\") "
Oct 30 07:58:48.555078 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:48.555059    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecd9f1bd-d4f6-468f-9994-203a3620a1a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecd9f1bd-d4f6-468f-9994-203a3620a1a3-workload-certs\") pod \"ecd9f1bd-d4f6-468f-9994-203a3620a1a3\" (UID: \"ecd9f1bd-d4f6-468f-9994-203a3620a1a3\") "
Oct 30 07:58:48.555201 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:48.555136    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecd9f1bd-d4f6-468f-9994-203a3620a1a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecd9f1bd-d4f6-468f-9994-203a3620a1a3-istio-envoy\") pod \"ecd9f1bd-d4f6-468f-9994-203a3620a1a3\" (UID: \"ecd9f1bd-d4f6-468f-9994-203a3620a1a3\") "
Oct 30 07:58:48.555201 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:48.555133    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecd9f1bd-d4f6-468f-9994-203a3620a1a3-credential-socket" pod "ecd9f1bd-d4f6-468f-9994-203a3620a1a3" (UID: "ecd9f1bd-d4f6-468f-9994-203a3620a1a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:58:48.555201 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:48.555183    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ecd9f1bd-d4f6-468f-9994-203a3620a1a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecd9f1bd-d4f6-468f-9994-203a3620a1a3-istio-podinfo\") pod \"ecd9f1bd-d4f6-468f-9994-203a3620a1a3\" (UID: \"ecd9f1bd-d4f6-468f-9994-203a3620a1a3\") "
Oct 30 07:58:48.555375 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:48.555211    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecd9f1bd-d4f6-468f-9994-203a3620a1a3-workload-socket" pod "ecd9f1bd-d4f6-468f-9994-203a3620a1a3" (UID: "ecd9f1bd-d4f6-468f-9994-203a3620a1a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:58:48.555375 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:48.555225    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecd9f1bd-d4f6-468f-9994-203a3620a1a3-istiod-ca-cert" pod "ecd9f1bd-d4f6-468f-9994-203a3620a1a3" (UID: "ecd9f1bd-d4f6-468f-9994-203a3620a1a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:58:48.555375 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:48.555240    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecd9f1bd-d4f6-468f-9994-203a3620a1a3-istio-data" pod "ecd9f1bd-d4f6-468f-9994-203a3620a1a3" (UID: "ecd9f1bd-d4f6-468f-9994-203a3620a1a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:58:48.555550 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:48.555430    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecd9f1bd-d4f6-468f-9994-203a3620a1a3-workload-certs" pod "ecd9f1bd-d4f6-468f-9994-203a3620a1a3" (UID: "ecd9f1bd-d4f6-468f-9994-203a3620a1a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:58:48.555875 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:48.555848    6224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecd9f1bd-d4f6-468f-9994-203a3620a1a3-credential-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:58:48.556008 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:48.555877    6224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecd9f1bd-d4f6-468f-9994-203a3620a1a3-workload-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:58:48.556008 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:48.555894    6224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecd9f1bd-d4f6-468f-9994-203a3620a1a3-istio-data\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:58:48.556008 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:48.555907    6224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecd9f1bd-d4f6-468f-9994-203a3620a1a3-workload-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:58:48.556008 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:48.555920    6224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecd9f1bd-d4f6-468f-9994-203a3620a1a3-istiod-ca-cert\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:58:48.557371 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:48.557345    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ecd9f1bd-d4f6-468f-9994-203a3620a1a3-istio-podinfo" pod "ecd9f1bd-d4f6-468f-9994-203a3620a1a3" (UID: "ecd9f1bd-d4f6-468f-9994-203a3620a1a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 07:58:48.557474 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:48.557432    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecd9f1bd-d4f6-468f-9994-203a3620a1a3-istio-envoy" pod "ecd9f1bd-d4f6-468f-9994-203a3620a1a3" (UID: "ecd9f1bd-d4f6-468f-9994-203a3620a1a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 07:58:48.557577 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:48.557549    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecd9f1bd-d4f6-468f-9994-203a3620a1a3-kube-api-access-mbptl" pod "ecd9f1bd-d4f6-468f-9994-203a3620a1a3" (UID: "ecd9f1bd-d4f6-468f-9994-203a3620a1a3"). InnerVolumeSpecName "kube-api-access-mbptl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:58:48.557636 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:48.557592    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecd9f1bd-d4f6-468f-9994-203a3620a1a3-istio-token" pod "ecd9f1bd-d4f6-468f-9994-203a3620a1a3" (UID: "ecd9f1bd-d4f6-468f-9994-203a3620a1a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:58:48.657328 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:48.657290    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbptl\" (UniqueName: \"kubernetes.io/projected/ecd9f1bd-d4f6-468f-9994-203a3620a1a3-kube-api-access-mbptl\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:58:48.657328 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:48.657326    6224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecd9f1bd-d4f6-468f-9994-203a3620a1a3-istio-envoy\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:58:48.657328 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:48.657335    6224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecd9f1bd-d4f6-468f-9994-203a3620a1a3-istio-podinfo\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:58:48.657623 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:48.657343    6224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecd9f1bd-d4f6-468f-9994-203a3620a1a3-istio-token\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:58:49.100219 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:49.100186    6224 generic.go:356] "Generic (PLEG): container finished" podID="ecd9f1bd-d4f6-468f-9994-203a3620a1a3" containerID="361ac3190018129223fe4d8c7a12945a7d8ca60cbacfea0cf70ee00679ac1482" exitCode=0
Oct 30 07:58:49.100403 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:49.100240    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688"
Oct 30 07:58:49.100403 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:49.100249    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688" event={"ID":"ecd9f1bd-d4f6-468f-9994-203a3620a1a3","Type":"ContainerDied","Data":"361ac3190018129223fe4d8c7a12945a7d8ca60cbacfea0cf70ee00679ac1482"}
Oct 30 07:58:49.100403 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:49.100293    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688" event={"ID":"ecd9f1bd-d4f6-468f-9994-203a3620a1a3","Type":"ContainerDied","Data":"637803a9aea52b4852e32e3609d9a805362e92c58b52cce498bf62a2c2e52cd1"}
Oct 30 07:58:49.100403 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:49.100314    6224 scope.go:122] "RemoveContainer" containerID="361ac3190018129223fe4d8c7a12945a7d8ca60cbacfea0cf70ee00679ac1482"
Oct 30 07:58:49.115349 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:49.112911    6224 scope.go:122] "RemoveContainer" containerID="55da16416b464b23e612a907bafbbed04a2c7eb62d44560a1fc60d393233730e"
Oct 30 07:58:49.124950 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:49.124932    6224 scope.go:122] "RemoveContainer" containerID="361ac3190018129223fe4d8c7a12945a7d8ca60cbacfea0cf70ee00679ac1482"
Oct 30 07:58:49.125235 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:58:49.125211    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"361ac3190018129223fe4d8c7a12945a7d8ca60cbacfea0cf70ee00679ac1482\": container with ID starting with 361ac3190018129223fe4d8c7a12945a7d8ca60cbacfea0cf70ee00679ac1482 not found: ID does not exist" containerID="361ac3190018129223fe4d8c7a12945a7d8ca60cbacfea0cf70ee00679ac1482"
Oct 30 07:58:49.125331 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:49.125242    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"361ac3190018129223fe4d8c7a12945a7d8ca60cbacfea0cf70ee00679ac1482"} err="failed to get container status \"361ac3190018129223fe4d8c7a12945a7d8ca60cbacfea0cf70ee00679ac1482\": rpc error: code = NotFound desc = could not find container \"361ac3190018129223fe4d8c7a12945a7d8ca60cbacfea0cf70ee00679ac1482\": container with ID starting with 361ac3190018129223fe4d8c7a12945a7d8ca60cbacfea0cf70ee00679ac1482 not found: ID does not exist"
Oct 30 07:58:49.125331 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:49.125260    6224 scope.go:122] "RemoveContainer" containerID="55da16416b464b23e612a907bafbbed04a2c7eb62d44560a1fc60d393233730e"
Oct 30 07:58:49.125478 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:58:49.125464    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55da16416b464b23e612a907bafbbed04a2c7eb62d44560a1fc60d393233730e\": container with ID starting with 55da16416b464b23e612a907bafbbed04a2c7eb62d44560a1fc60d393233730e not found: ID does not exist" containerID="55da16416b464b23e612a907bafbbed04a2c7eb62d44560a1fc60d393233730e"
Oct 30 07:58:49.125520 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:49.125481    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55da16416b464b23e612a907bafbbed04a2c7eb62d44560a1fc60d393233730e"} err="failed to get container status \"55da16416b464b23e612a907bafbbed04a2c7eb62d44560a1fc60d393233730e\": rpc error: code = NotFound desc = could not find container \"55da16416b464b23e612a907bafbbed04a2c7eb62d44560a1fc60d393233730e\": container with ID starting with 55da16416b464b23e612a907bafbbed04a2c7eb62d44560a1fc60d393233730e not found: ID does not exist"
Oct 30 07:58:49.161548 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:49.161524    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688"]
Oct 30 07:58:49.191451 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:49.191416    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-896ca4-openshift-default-7684b6b7-sq688"]
Oct 30 07:58:49.504195 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:49.504118    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecd9f1bd-d4f6-468f-9994-203a3620a1a3" path="/var/lib/kubelet/pods/ecd9f1bd-d4f6-468f-9994-203a3620a1a3/volumes"
Oct 30 07:58:54.158985 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:54.158939    6224 generic.go:356] "Generic (PLEG): container finished" podID="ca0e2ee4-723d-4fc1-b13e-72b13c83d280" containerID="5b53486850f289a4bf760dee28e46d37112e535569df409558a611e76a20b4c9" exitCode=0
Oct 30 07:58:54.159523 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:54.159012    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj" event={"ID":"ca0e2ee4-723d-4fc1-b13e-72b13c83d280","Type":"ContainerDied","Data":"5b53486850f289a4bf760dee28e46d37112e535569df409558a611e76a20b4c9"}
Oct 30 07:58:54.159523 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:54.159458    6224 scope.go:122] "RemoveContainer" containerID="5b53486850f289a4bf760dee28e46d37112e535569df409558a611e76a20b4c9"
Oct 30 07:58:54.161208 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:54.161175    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:58:54.161341 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:54.161288    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:58:54.161412 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:54.161338    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 07:58:54.800856 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:54.800744    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-137122/rate-test-model-137122-kserve-6bb88d4545-29s55"]
Oct 30 07:58:54.801847 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:54.801741    6224 cpu_manager.go:414] "containerMap: removing container" podUID="ecd9f1bd-d4f6-468f-9994-203a3620a1a3" containerName="istio-proxy"
Oct 30 07:58:54.801847 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:54.801767    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecd9f1bd-d4f6-468f-9994-203a3620a1a3" containerName="istio-proxy"
Oct 30 07:58:54.802056 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:54.801968    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecd9f1bd-d4f6-468f-9994-203a3620a1a3" containerName="istio-proxy"
Oct 30 07:58:54.802056 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:54.801990    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecd9f1bd-d4f6-468f-9994-203a3620a1a3" containerName="istio-proxy"
Oct 30 07:58:54.823475 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:54.821158    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-137122/rate-test-model-137122-kserve-6bb88d4545-29s55"
Oct 30 07:58:54.826791 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:54.826704    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-137122/rate-test-model-137122-kserve-6bb88d4545-29s55"]
Oct 30 07:58:54.861096 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:54.861063    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-137122\"/\"kube-root-ca.crt\""
Oct 30 07:58:54.861293 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:54.861157    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-137122\"/\"default-dockercfg-hfdg8\""
Oct 30 07:58:54.861293 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:54.861160    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-137122\"/\"openshift-service-ca.crt\""
Oct 30 07:58:54.861293 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:54.861063    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-137122\"/\"rate-test-model-137122-kserve-self-signed-certs\""
Oct 30 07:58:54.990117 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:54.989978    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477-tls-certs\") pod \"rate-test-model-137122-kserve-6bb88d4545-29s55\" (UID: \"85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477\") " pod="ai-tenant-e2e-shared-b-w2-137122/rate-test-model-137122-kserve-6bb88d4545-29s55"
Oct 30 07:58:54.990117 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:54.990072    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chmzx\" (UniqueName: \"kubernetes.io/projected/85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477-kube-api-access-chmzx\") pod \"rate-test-model-137122-kserve-6bb88d4545-29s55\" (UID: \"85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477\") " pod="ai-tenant-e2e-shared-b-w2-137122/rate-test-model-137122-kserve-6bb88d4545-29s55"
Oct 30 07:58:54.990410 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:54.990126    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477-dshm\") pod \"rate-test-model-137122-kserve-6bb88d4545-29s55\" (UID: \"85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477\") " pod="ai-tenant-e2e-shared-b-w2-137122/rate-test-model-137122-kserve-6bb88d4545-29s55"
Oct 30 07:58:54.990410 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:54.990304    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477-tmp-dir\") pod \"rate-test-model-137122-kserve-6bb88d4545-29s55\" (UID: \"85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477\") " pod="ai-tenant-e2e-shared-b-w2-137122/rate-test-model-137122-kserve-6bb88d4545-29s55"
Oct 30 07:58:54.990536 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:54.990411    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477-home\") pod \"rate-test-model-137122-kserve-6bb88d4545-29s55\" (UID: \"85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477\") " pod="ai-tenant-e2e-shared-b-w2-137122/rate-test-model-137122-kserve-6bb88d4545-29s55"
Oct 30 07:58:54.990536 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:54.990444    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477-model-cache\") pod \"rate-test-model-137122-kserve-6bb88d4545-29s55\" (UID: \"85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477\") " pod="ai-tenant-e2e-shared-b-w2-137122/rate-test-model-137122-kserve-6bb88d4545-29s55"
Oct 30 07:58:55.099882 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:55.099152    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477-home\") pod \"rate-test-model-137122-kserve-6bb88d4545-29s55\" (UID: \"85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477\") " pod="ai-tenant-e2e-shared-b-w2-137122/rate-test-model-137122-kserve-6bb88d4545-29s55"
Oct 30 07:58:55.099882 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:55.099208    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477-model-cache\") pod \"rate-test-model-137122-kserve-6bb88d4545-29s55\" (UID: \"85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477\") " pod="ai-tenant-e2e-shared-b-w2-137122/rate-test-model-137122-kserve-6bb88d4545-29s55"
Oct 30 07:58:55.099882 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:55.099310    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477-tls-certs\") pod \"rate-test-model-137122-kserve-6bb88d4545-29s55\" (UID: \"85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477\") " pod="ai-tenant-e2e-shared-b-w2-137122/rate-test-model-137122-kserve-6bb88d4545-29s55"
Oct 30 07:58:55.099882 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:55.099372    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chmzx\" (UniqueName: \"kubernetes.io/projected/85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477-kube-api-access-chmzx\") pod \"rate-test-model-137122-kserve-6bb88d4545-29s55\" (UID: \"85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477\") " pod="ai-tenant-e2e-shared-b-w2-137122/rate-test-model-137122-kserve-6bb88d4545-29s55"
Oct 30 07:58:55.099882 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:55.099431    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477-dshm\") pod \"rate-test-model-137122-kserve-6bb88d4545-29s55\" (UID: \"85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477\") " pod="ai-tenant-e2e-shared-b-w2-137122/rate-test-model-137122-kserve-6bb88d4545-29s55"
Oct 30 07:58:55.099882 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:55.099460    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477-tmp-dir\") pod \"rate-test-model-137122-kserve-6bb88d4545-29s55\" (UID: \"85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477\") " pod="ai-tenant-e2e-shared-b-w2-137122/rate-test-model-137122-kserve-6bb88d4545-29s55"
Oct 30 07:58:55.100435 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:55.099982    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477-tmp-dir\") pod \"rate-test-model-137122-kserve-6bb88d4545-29s55\" (UID: \"85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477\") " pod="ai-tenant-e2e-shared-b-w2-137122/rate-test-model-137122-kserve-6bb88d4545-29s55"
Oct 30 07:58:55.100435 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:55.100241    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477-home\") pod \"rate-test-model-137122-kserve-6bb88d4545-29s55\" (UID: \"85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477\") " pod="ai-tenant-e2e-shared-b-w2-137122/rate-test-model-137122-kserve-6bb88d4545-29s55"
Oct 30 07:58:55.100939 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:55.100493    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477-model-cache\") pod \"rate-test-model-137122-kserve-6bb88d4545-29s55\" (UID: \"85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477\") " pod="ai-tenant-e2e-shared-b-w2-137122/rate-test-model-137122-kserve-6bb88d4545-29s55"
Oct 30 07:58:55.124851 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:55.105599    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477-tls-certs\") pod \"rate-test-model-137122-kserve-6bb88d4545-29s55\" (UID: \"85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477\") " pod="ai-tenant-e2e-shared-b-w2-137122/rate-test-model-137122-kserve-6bb88d4545-29s55"
Oct 30 07:58:55.124851 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:55.105746    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477-dshm\") pod \"rate-test-model-137122-kserve-6bb88d4545-29s55\" (UID: \"85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477\") " pod="ai-tenant-e2e-shared-b-w2-137122/rate-test-model-137122-kserve-6bb88d4545-29s55"
Oct 30 07:58:55.137273 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:55.133934    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chmzx\" (UniqueName: \"kubernetes.io/projected/85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477-kube-api-access-chmzx\") pod \"rate-test-model-137122-kserve-6bb88d4545-29s55\" (UID: \"85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477\") " pod="ai-tenant-e2e-shared-b-w2-137122/rate-test-model-137122-kserve-6bb88d4545-29s55"
Oct 30 07:58:55.149848 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:55.141435    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-137122/rate-test-model-137122-kserve-6bb88d4545-29s55"
Oct 30 07:58:55.174439 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:55.173879    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj" event={"ID":"ca0e2ee4-723d-4fc1-b13e-72b13c83d280","Type":"ContainerStarted","Data":"34bc958339bd74659bdfd09f020156ffdb2ed2a83c22f7d7651934a04be40cd2"}
Oct 30 07:58:55.493971 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:55.493051    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-137122/rate-test-model-137122-kserve-6bb88d4545-29s55"]
Oct 30 07:58:55.493971 ip-10-0-128-125 kubenswrapper[6224]: W1030 07:58:55.493080    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod85dfc83f_9c0c_4bc6_a4ab_998d6c2d1477.slice/crio-e1c52bf236d7ad0d64e33d08061d9151e289b76c2a099fefc151386da351f886 WatchSource:0}: Error finding container e1c52bf236d7ad0d64e33d08061d9151e289b76c2a099fefc151386da351f886: Status 404 returned error can't find the container with id e1c52bf236d7ad0d64e33d08061d9151e289b76c2a099fefc151386da351f886
Oct 30 07:58:55.840737 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:55.840705    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj"
Oct 30 07:58:55.841861 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:55.841836    6224 patch_prober.go:29] interesting pod/e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body=
Oct 30 07:58:55.842012 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:55.841885    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj" podUID="ca0e2ee4-723d-4fc1-b13e-72b13c83d280" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused"
Oct 30 07:58:56.194053 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:56.193961    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-137122/rate-test-model-137122-kserve-6bb88d4545-29s55" event={"ID":"85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477","Type":"ContainerStarted","Data":"dac9882b70ea007883f5f3bfa68c81654ccddd49ab0f459c024d08bdf13eacb7"}
Oct 30 07:58:56.194053 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:56.194013    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-137122/rate-test-model-137122-kserve-6bb88d4545-29s55" event={"ID":"85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477","Type":"ContainerStarted","Data":"e1c52bf236d7ad0d64e33d08061d9151e289b76c2a099fefc151386da351f886"}
Oct 30 07:58:56.241319 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:56.241272    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-137122/rate-test-model-137122-kserve-6bb88d4545-29s55" podStartSLOduration=2.241259224 podStartE2EDuration="2.241259224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:58:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:58:56.241109188 +0000 UTC m=+3694.850516174" watchObservedRunningTime="2026-10-30 07:58:56.241259224 +0000 UTC m=+3694.850666238"
Oct 30 07:58:56.840923 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:56.840887    6224 patch_prober.go:29] interesting pod/e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body=
Oct 30 07:58:56.841092 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:56.840941    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj" podUID="ca0e2ee4-723d-4fc1-b13e-72b13c83d280" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused"
Oct 30 07:58:57.142372 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:57.142293    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-137122/rate-test-model-137122-kserve-6bb88d4545-29s55"
Oct 30 07:58:57.840769 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:57.840729    6224 patch_prober.go:29] interesting pod/e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body=
Oct 30 07:58:57.841294 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:57.840792    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj" podUID="ca0e2ee4-723d-4fc1-b13e-72b13c83d280" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused"
Oct 30 07:58:58.840377 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:58.840340    6224 patch_prober.go:29] interesting pod/e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body=
Oct 30 07:58:58.840544 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:58.840390    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj" podUID="ca0e2ee4-723d-4fc1-b13e-72b13c83d280" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused"
Oct 30 07:58:59.106292 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:59.106217    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj"
Oct 30 07:58:59.840495 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:59.840461    6224 patch_prober.go:29] interesting pod/e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body=
Oct 30 07:58:59.840682 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:58:59.840525    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj" podUID="ca0e2ee4-723d-4fc1-b13e-72b13c83d280" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused"
Oct 30 07:59:00.845270 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:59:00.845239    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj"
Oct 30 07:59:00.846500 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:59:00.846478    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj"
Oct 30 07:59:01.864275 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:59:01.864236    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:59:05.142286 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:59:05.142208    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-137122/rate-test-model-137122-kserve-6bb88d4545-29s55"
Oct 30 07:59:05.155152 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:59:05.155127    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-137122/rate-test-model-137122-kserve-6bb88d4545-29s55"
Oct 30 07:59:05.274174 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:59:05.274144    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-137122/rate-test-model-137122-kserve-6bb88d4545-29s55"
Oct 30 07:59:15.700792 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:59:15.700756    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:59:31.857616 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:59:31.857575    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 07:59:38.816371 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:59:38.816335    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8f1df-f98c8b658-g2h4t"]
Oct 30 07:59:38.816854 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:59:38.816561    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8f1df-f98c8b658-g2h4t" podUID="319ad66a-6d8b-40bf-8a32-3576e26f34ed" containerName="maas-api" containerID="cri-o://ddf2d0fa99cd2f1814e7e202d2ce7d8a861c4fb7895918564a58fbc1e98382c8" gracePeriod=30
Oct 30 07:59:39.122604 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:59:39.120507    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8f1df-f98c8b658-g2h4t"
Oct 30 07:59:39.279372 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:59:39.279339    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/319ad66a-6d8b-40bf-8a32-3576e26f34ed-metrics-certs\" (UniqueName: \"kubernetes.io/secret/319ad66a-6d8b-40bf-8a32-3576e26f34ed-metrics-certs\") pod \"319ad66a-6d8b-40bf-8a32-3576e26f34ed\" (UID: \"319ad66a-6d8b-40bf-8a32-3576e26f34ed\") "
Oct 30 07:59:39.279567 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:59:39.279519    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/319ad66a-6d8b-40bf-8a32-3576e26f34ed-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/319ad66a-6d8b-40bf-8a32-3576e26f34ed-openshift-service-ca\") pod \"319ad66a-6d8b-40bf-8a32-3576e26f34ed\" (UID: \"319ad66a-6d8b-40bf-8a32-3576e26f34ed\") "
Oct 30 07:59:39.279567 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:59:39.279540    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/319ad66a-6d8b-40bf-8a32-3576e26f34ed-kube-api-access-7p5xd\" (UniqueName: \"kubernetes.io/projected/319ad66a-6d8b-40bf-8a32-3576e26f34ed-kube-api-access-7p5xd\") pod \"319ad66a-6d8b-40bf-8a32-3576e26f34ed\" (UID: \"319ad66a-6d8b-40bf-8a32-3576e26f34ed\") "
Oct 30 07:59:39.279712 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:59:39.279570    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/319ad66a-6d8b-40bf-8a32-3576e26f34ed-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/319ad66a-6d8b-40bf-8a32-3576e26f34ed-maas-api-tls\") pod \"319ad66a-6d8b-40bf-8a32-3576e26f34ed\" (UID: \"319ad66a-6d8b-40bf-8a32-3576e26f34ed\") "
Oct 30 07:59:39.279992 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:59:39.279961    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/319ad66a-6d8b-40bf-8a32-3576e26f34ed-openshift-service-ca" pod "319ad66a-6d8b-40bf-8a32-3576e26f34ed" (UID: "319ad66a-6d8b-40bf-8a32-3576e26f34ed"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 07:59:39.281677 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:59:39.281654    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/319ad66a-6d8b-40bf-8a32-3576e26f34ed-kube-api-access-7p5xd" pod "319ad66a-6d8b-40bf-8a32-3576e26f34ed" (UID: "319ad66a-6d8b-40bf-8a32-3576e26f34ed"). InnerVolumeSpecName "kube-api-access-7p5xd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 07:59:39.282073 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:59:39.282054    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/319ad66a-6d8b-40bf-8a32-3576e26f34ed-metrics-certs" pod "319ad66a-6d8b-40bf-8a32-3576e26f34ed" (UID: "319ad66a-6d8b-40bf-8a32-3576e26f34ed"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:59:39.282175 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:59:39.282078    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/319ad66a-6d8b-40bf-8a32-3576e26f34ed-maas-api-tls" pod "319ad66a-6d8b-40bf-8a32-3576e26f34ed" (UID: "319ad66a-6d8b-40bf-8a32-3576e26f34ed"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 07:59:39.381350 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:59:39.381264    6224 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/319ad66a-6d8b-40bf-8a32-3576e26f34ed-metrics-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:59:39.381350 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:59:39.381296    6224 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/319ad66a-6d8b-40bf-8a32-3576e26f34ed-openshift-service-ca\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:59:39.381350 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:59:39.381306    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p5xd\" (UniqueName: \"kubernetes.io/projected/319ad66a-6d8b-40bf-8a32-3576e26f34ed-kube-api-access-7p5xd\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:59:39.381350 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:59:39.381315    6224 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/319ad66a-6d8b-40bf-8a32-3576e26f34ed-maas-api-tls\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 07:59:39.505791 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:59:39.505753    6224 generic.go:356] "Generic (PLEG): container finished" podID="319ad66a-6d8b-40bf-8a32-3576e26f34ed" containerID="ddf2d0fa99cd2f1814e7e202d2ce7d8a861c4fb7895918564a58fbc1e98382c8" exitCode=0
Oct 30 07:59:39.505791 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:59:39.505793    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8f1df-f98c8b658-g2h4t" event={"ID":"319ad66a-6d8b-40bf-8a32-3576e26f34ed","Type":"ContainerDied","Data":"ddf2d0fa99cd2f1814e7e202d2ce7d8a861c4fb7895918564a58fbc1e98382c8"}
Oct 30 07:59:39.506084 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:59:39.505840    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8f1df-f98c8b658-g2h4t"
Oct 30 07:59:39.506084 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:59:39.505864    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8f1df-f98c8b658-g2h4t" event={"ID":"319ad66a-6d8b-40bf-8a32-3576e26f34ed","Type":"ContainerDied","Data":"b4baec71214b99314a2e3f1adc94dfbc982e941932ef6bbfdea8a5cffe119d53"}
Oct 30 07:59:39.506084 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:59:39.505890    6224 scope.go:122] "RemoveContainer" containerID="ddf2d0fa99cd2f1814e7e202d2ce7d8a861c4fb7895918564a58fbc1e98382c8"
Oct 30 07:59:39.521306 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:59:39.521285    6224 scope.go:122] "RemoveContainer" containerID="ddf2d0fa99cd2f1814e7e202d2ce7d8a861c4fb7895918564a58fbc1e98382c8"
Oct 30 07:59:39.521602 ip-10-0-128-125 kubenswrapper[6224]: E1030 07:59:39.521582    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddf2d0fa99cd2f1814e7e202d2ce7d8a861c4fb7895918564a58fbc1e98382c8\": container with ID starting with ddf2d0fa99cd2f1814e7e202d2ce7d8a861c4fb7895918564a58fbc1e98382c8 not found: ID does not exist" containerID="ddf2d0fa99cd2f1814e7e202d2ce7d8a861c4fb7895918564a58fbc1e98382c8"
Oct 30 07:59:39.521684 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:59:39.521609    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddf2d0fa99cd2f1814e7e202d2ce7d8a861c4fb7895918564a58fbc1e98382c8"} err="failed to get container status \"ddf2d0fa99cd2f1814e7e202d2ce7d8a861c4fb7895918564a58fbc1e98382c8\": rpc error: code = NotFound desc = could not find container \"ddf2d0fa99cd2f1814e7e202d2ce7d8a861c4fb7895918564a58fbc1e98382c8\": container with ID starting with ddf2d0fa99cd2f1814e7e202d2ce7d8a861c4fb7895918564a58fbc1e98382c8 not found: ID does not exist"
Oct 30 07:59:39.569094 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:59:39.569069    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8f1df-f98c8b658-g2h4t"]
Oct 30 07:59:39.582915 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:59:39.582892    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8f1df-f98c8b658-g2h4t"]
Oct 30 07:59:41.505586 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:59:41.505553    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="319ad66a-6d8b-40bf-8a32-3576e26f34ed" path="/var/lib/kubelet/pods/319ad66a-6d8b-40bf-8a32-3576e26f34ed/volumes"
Oct 30 07:59:57.061435 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:59:57.061369    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 07:59:57.962152 ip-10-0-128-125 kubenswrapper[6224]: I1030 07:59:57.962120    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-1f34aa-6c5cc4f795-pqzbx"]
Oct 30 08:00:00.172312 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.172274    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8e7948-29889120-jt7qq"]
Oct 30 08:00:00.172938 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.172923    6224 cpu_manager.go:414] "containerMap: removing container" podUID="ecd9f1bd-d4f6-468f-9994-203a3620a1a3" containerName="istio-proxy"
Oct 30 08:00:00.173006 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.172938    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecd9f1bd-d4f6-468f-9994-203a3620a1a3" containerName="istio-proxy"
Oct 30 08:00:00.173006 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.172974    6224 cpu_manager.go:414] "containerMap: removing container" podUID="319ad66a-6d8b-40bf-8a32-3576e26f34ed" containerName="maas-api"
Oct 30 08:00:00.173006 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.172979    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="319ad66a-6d8b-40bf-8a32-3576e26f34ed" containerName="maas-api"
Oct 30 08:00:00.173111 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.173074    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="319ad66a-6d8b-40bf-8a32-3576e26f34ed" containerName="maas-api"
Oct 30 08:00:00.176598 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.176578    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8e7948-29889120-jt7qq"
Oct 30 08:00:00.197000 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.196975    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889120-rzdrx"]
Oct 30 08:00:00.201125 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.201100    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889120-rzdrx"
Oct 30 08:00:00.215680 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.215650    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8e7948-29889120-jt7qq"]
Oct 30 08:00:00.238796 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.238773    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889120-sd6xv"]
Oct 30 08:00:00.243067 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.243044    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889120-tww9v"]
Oct 30 08:00:00.243202 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.243179    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889120-sd6xv"
Oct 30 08:00:00.248556 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.248534    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889120-tww9v"
Oct 30 08:00:00.282340 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.282315    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889120-rzdrx"]
Oct 30 08:00:00.301603 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.301574    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 08:00:00.303020 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.302995    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889120-sd6xv"]
Oct 30 08:00:00.308139 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.308117    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889120-tww9v"]
Oct 30 08:00:00.312642 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.312618    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889120-pvxg6"]
Oct 30 08:00:00.317063 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.317041    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889120-rbbl8"]
Oct 30 08:00:00.317183 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.317154    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889120-pvxg6"
Oct 30 08:00:00.322299 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.322276    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889120-rbbl8"
Oct 30 08:00:00.334509 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.334484    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da7c6736-ac92-44ec-a1cf-5f835eb860b1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889120-tww9v\" (UID: \"da7c6736-ac92-44ec-a1cf-5f835eb860b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889120-tww9v"
Oct 30 08:00:00.334617 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.334564    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv4rq\" (UniqueName: \"kubernetes.io/projected/246d2cbc-c719-46fd-a84d-0e47a21358fe-kube-api-access-rv4rq\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-265384-29889120-rzdrx\" (UID: \"246d2cbc-c719-46fd-a84d-0e47a21358fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889120-rzdrx"
Oct 30 08:00:00.334691 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.334614    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cd532fb-2369-434f-9622-d6792f8ee7a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889120-sd6xv\" (UID: \"1cd532fb-2369-434f-9622-d6792f8ee7a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889120-sd6xv"
Oct 30 08:00:00.334799 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.334784    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r5hb\" (UniqueName: \"kubernetes.io/projected/e8936cc2-40e5-4d2e-81b0-1ad1ca9e5a2a-kube-api-access-4r5hb\") pod \"maas-api-key-cleanup-e2e-extmod-a-8e7948-29889120-jt7qq\" (UID: \"e8936cc2-40e5-4d2e-81b0-1ad1ca9e5a2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8e7948-29889120-jt7qq"
Oct 30 08:00:00.334881 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.334844    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg92c\" (UniqueName: \"kubernetes.io/projected/1cd532fb-2369-434f-9622-d6792f8ee7a0-kube-api-access-lg92c\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889120-sd6xv\" (UID: \"1cd532fb-2369-434f-9622-d6792f8ee7a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889120-sd6xv"
Oct 30 08:00:00.334947 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.334929    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8936cc2-40e5-4d2e-81b0-1ad1ca9e5a2a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-8e7948-29889120-jt7qq\" (UID: \"e8936cc2-40e5-4d2e-81b0-1ad1ca9e5a2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8e7948-29889120-jt7qq"
Oct 30 08:00:00.334997 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.334963    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t24rl\" (UniqueName: \"kubernetes.io/projected/da7c6736-ac92-44ec-a1cf-5f835eb860b1-kube-api-access-t24rl\") pod \"maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889120-tww9v\" (UID: \"da7c6736-ac92-44ec-a1cf-5f835eb860b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889120-tww9v"
Oct 30 08:00:00.335080 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.335065    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/246d2cbc-c719-46fd-a84d-0e47a21358fe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-265384-29889120-rzdrx\" (UID: \"246d2cbc-c719-46fd-a84d-0e47a21358fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889120-rzdrx"
Oct 30 08:00:00.345130 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.344819    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889120-qw752"]
Oct 30 08:00:00.351167 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.351147    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889120-qw752"
Oct 30 08:00:00.353317 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.352824    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889120-rbbl8"]
Oct 30 08:00:00.378902 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.378865    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889120-pvxg6"]
Oct 30 08:00:00.392850 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.392825    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889120-9vz2g"]
Oct 30 08:00:00.397344 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.397320    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889120-9vz2g"
Oct 30 08:00:00.405081 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.405038    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889120-qw752"]
Oct 30 08:00:00.436667 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.436600    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfvf2\" (UniqueName: \"kubernetes.io/projected/a0e05a96-75ea-4636-b9d9-f08b5c1ba1ad-kube-api-access-rfvf2\") pod \"maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889120-pvxg6\" (UID: \"a0e05a96-75ea-4636-b9d9-f08b5c1ba1ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889120-pvxg6"
Oct 30 08:00:00.436967 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.436945    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb6ead4a-d324-450f-adb6-dbbb75afc174-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889120-rbbl8\" (UID: \"fb6ead4a-d324-450f-adb6-dbbb75afc174\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889120-rbbl8"
Oct 30 08:00:00.437204 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.437182    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r5hb\" (UniqueName: \"kubernetes.io/projected/e8936cc2-40e5-4d2e-81b0-1ad1ca9e5a2a-kube-api-access-4r5hb\") pod \"maas-api-key-cleanup-e2e-extmod-a-8e7948-29889120-jt7qq\" (UID: \"e8936cc2-40e5-4d2e-81b0-1ad1ca9e5a2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8e7948-29889120-jt7qq"
Oct 30 08:00:00.437395 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.437367    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkhfj\" (UniqueName: \"kubernetes.io/projected/f68a7a87-6354-4456-ab2c-17cf8995865f-kube-api-access-fkhfj\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-137122-29889120-qw752\" (UID: \"f68a7a87-6354-4456-ab2c-17cf8995865f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889120-qw752"
Oct 30 08:00:00.437578 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.437540    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg92c\" (UniqueName: \"kubernetes.io/projected/1cd532fb-2369-434f-9622-d6792f8ee7a0-kube-api-access-lg92c\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889120-sd6xv\" (UID: \"1cd532fb-2369-434f-9622-d6792f8ee7a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889120-sd6xv"
Oct 30 08:00:00.437674 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.437604    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0e05a96-75ea-4636-b9d9-f08b5c1ba1ad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889120-pvxg6\" (UID: \"a0e05a96-75ea-4636-b9d9-f08b5c1ba1ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889120-pvxg6"
Oct 30 08:00:00.437749 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.437674    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8936cc2-40e5-4d2e-81b0-1ad1ca9e5a2a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-8e7948-29889120-jt7qq\" (UID: \"e8936cc2-40e5-4d2e-81b0-1ad1ca9e5a2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8e7948-29889120-jt7qq"
Oct 30 08:00:00.437749 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.437714    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t24rl\" (UniqueName: \"kubernetes.io/projected/da7c6736-ac92-44ec-a1cf-5f835eb860b1-kube-api-access-t24rl\") pod \"maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889120-tww9v\" (UID: \"da7c6736-ac92-44ec-a1cf-5f835eb860b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889120-tww9v"
Oct 30 08:00:00.437914 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.437755    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvmp8\" (UniqueName: \"kubernetes.io/projected/fb6ead4a-d324-450f-adb6-dbbb75afc174-kube-api-access-jvmp8\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889120-rbbl8\" (UID: \"fb6ead4a-d324-450f-adb6-dbbb75afc174\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889120-rbbl8"
Oct 30 08:00:00.437914 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.437796    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f68a7a87-6354-4456-ab2c-17cf8995865f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-137122-29889120-qw752\" (UID: \"f68a7a87-6354-4456-ab2c-17cf8995865f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889120-qw752"
Oct 30 08:00:00.437914 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.437859    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/246d2cbc-c719-46fd-a84d-0e47a21358fe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-265384-29889120-rzdrx\" (UID: \"246d2cbc-c719-46fd-a84d-0e47a21358fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889120-rzdrx"
Oct 30 08:00:00.437914 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.437899    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da7c6736-ac92-44ec-a1cf-5f835eb860b1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889120-tww9v\" (UID: \"da7c6736-ac92-44ec-a1cf-5f835eb860b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889120-tww9v"
Oct 30 08:00:00.438146 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.437979    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv4rq\" (UniqueName: \"kubernetes.io/projected/246d2cbc-c719-46fd-a84d-0e47a21358fe-kube-api-access-rv4rq\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-265384-29889120-rzdrx\" (UID: \"246d2cbc-c719-46fd-a84d-0e47a21358fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889120-rzdrx"
Oct 30 08:00:00.438146 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.438027    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cd532fb-2369-434f-9622-d6792f8ee7a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889120-sd6xv\" (UID: \"1cd532fb-2369-434f-9622-d6792f8ee7a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889120-sd6xv"
Oct 30 08:00:00.439290 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.439269    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8936cc2-40e5-4d2e-81b0-1ad1ca9e5a2a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-8e7948-29889120-jt7qq\" (UID: \"e8936cc2-40e5-4d2e-81b0-1ad1ca9e5a2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8e7948-29889120-jt7qq"
Oct 30 08:00:00.439794 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.439772    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/246d2cbc-c719-46fd-a84d-0e47a21358fe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-265384-29889120-rzdrx\" (UID: \"246d2cbc-c719-46fd-a84d-0e47a21358fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889120-rzdrx"
Oct 30 08:00:00.440127 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.440104    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da7c6736-ac92-44ec-a1cf-5f835eb860b1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889120-tww9v\" (UID: \"da7c6736-ac92-44ec-a1cf-5f835eb860b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889120-tww9v"
Oct 30 08:00:00.440265 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.440242    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cd532fb-2369-434f-9622-d6792f8ee7a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889120-sd6xv\" (UID: \"1cd532fb-2369-434f-9622-d6792f8ee7a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889120-sd6xv"
Oct 30 08:00:00.450204 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.450150    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889120-wc4p7"]
Oct 30 08:00:00.456598 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.456575    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889120-9vz2g"]
Oct 30 08:00:00.456721 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.456675    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889120-wc4p7"
Oct 30 08:00:00.481641 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.481570    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r5hb\" (UniqueName: \"kubernetes.io/projected/e8936cc2-40e5-4d2e-81b0-1ad1ca9e5a2a-kube-api-access-4r5hb\") pod \"maas-api-key-cleanup-e2e-extmod-a-8e7948-29889120-jt7qq\" (UID: \"e8936cc2-40e5-4d2e-81b0-1ad1ca9e5a2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8e7948-29889120-jt7qq"
Oct 30 08:00:00.495443 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.490564    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8e7948-29889120-jt7qq"
Oct 30 08:00:00.510074 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.510046    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv4rq\" (UniqueName: \"kubernetes.io/projected/246d2cbc-c719-46fd-a84d-0e47a21358fe-kube-api-access-rv4rq\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-265384-29889120-rzdrx\" (UID: \"246d2cbc-c719-46fd-a84d-0e47a21358fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889120-rzdrx"
Oct 30 08:00:00.510232 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.510151    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t24rl\" (UniqueName: \"kubernetes.io/projected/da7c6736-ac92-44ec-a1cf-5f835eb860b1-kube-api-access-t24rl\") pod \"maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889120-tww9v\" (UID: \"da7c6736-ac92-44ec-a1cf-5f835eb860b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889120-tww9v"
Oct 30 08:00:00.511548 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.511503    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg92c\" (UniqueName: \"kubernetes.io/projected/1cd532fb-2369-434f-9622-d6792f8ee7a0-kube-api-access-lg92c\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889120-sd6xv\" (UID: \"1cd532fb-2369-434f-9622-d6792f8ee7a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889120-sd6xv"
Oct 30 08:00:00.521154 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.517776    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889120-wc4p7"]
Oct 30 08:00:00.522522 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.522497    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889120-rzdrx"
Oct 30 08:00:00.526257 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.526230    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f5081-29889120-pchn8"]
Oct 30 08:00:00.536519 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.536495    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f5081-29889120-pchn8"]
Oct 30 08:00:00.536866 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.536843    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f5081-29889120-pchn8"
Oct 30 08:00:00.539955 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.539928    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkhfj\" (UniqueName: \"kubernetes.io/projected/f68a7a87-6354-4456-ab2c-17cf8995865f-kube-api-access-fkhfj\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-137122-29889120-qw752\" (UID: \"f68a7a87-6354-4456-ab2c-17cf8995865f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889120-qw752"
Oct 30 08:00:00.540109 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.539988    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0e05a96-75ea-4636-b9d9-f08b5c1ba1ad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889120-pvxg6\" (UID: \"a0e05a96-75ea-4636-b9d9-f08b5c1ba1ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889120-pvxg6"
Oct 30 08:00:00.540109 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.540022    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38de25ac-6fa2-4650-9cfa-ec23daf2ec76-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889120-wc4p7\" (UID: \"38de25ac-6fa2-4650-9cfa-ec23daf2ec76\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889120-wc4p7"
Oct 30 08:00:00.540109 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.540097    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvmp8\" (UniqueName: \"kubernetes.io/projected/fb6ead4a-d324-450f-adb6-dbbb75afc174-kube-api-access-jvmp8\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889120-rbbl8\" (UID: \"fb6ead4a-d324-450f-adb6-dbbb75afc174\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889120-rbbl8"
Oct 30 08:00:00.540358 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.540121    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p754p\" (UniqueName: \"kubernetes.io/projected/38de25ac-6fa2-4650-9cfa-ec23daf2ec76-kube-api-access-p754p\") pod \"maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889120-wc4p7\" (UID: \"38de25ac-6fa2-4650-9cfa-ec23daf2ec76\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889120-wc4p7"
Oct 30 08:00:00.540358 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.540161    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f68a7a87-6354-4456-ab2c-17cf8995865f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-137122-29889120-qw752\" (UID: \"f68a7a87-6354-4456-ab2c-17cf8995865f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889120-qw752"
Oct 30 08:00:00.540358 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.540187    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svn5t\" (UniqueName: \"kubernetes.io/projected/3a639cc2-221b-4e62-86c7-64ae4f7c43f7-kube-api-access-svn5t\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889120-9vz2g\" (UID: \"3a639cc2-221b-4e62-86c7-64ae4f7c43f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889120-9vz2g"
Oct 30 08:00:00.540358 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.540282    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a639cc2-221b-4e62-86c7-64ae4f7c43f7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889120-9vz2g\" (UID: \"3a639cc2-221b-4e62-86c7-64ae4f7c43f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889120-9vz2g"
Oct 30 08:00:00.540686 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.540469    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfvf2\" (UniqueName: \"kubernetes.io/projected/a0e05a96-75ea-4636-b9d9-f08b5c1ba1ad-kube-api-access-rfvf2\") pod \"maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889120-pvxg6\" (UID: \"a0e05a96-75ea-4636-b9d9-f08b5c1ba1ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889120-pvxg6"
Oct 30 08:00:00.540686 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.540500    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb6ead4a-d324-450f-adb6-dbbb75afc174-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889120-rbbl8\" (UID: \"fb6ead4a-d324-450f-adb6-dbbb75afc174\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889120-rbbl8"
Oct 30 08:00:00.540686 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.540530    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0e05a96-75ea-4636-b9d9-f08b5c1ba1ad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889120-pvxg6\" (UID: \"a0e05a96-75ea-4636-b9d9-f08b5c1ba1ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889120-pvxg6"
Oct 30 08:00:00.541056 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.541037    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f68a7a87-6354-4456-ab2c-17cf8995865f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-137122-29889120-qw752\" (UID: \"f68a7a87-6354-4456-ab2c-17cf8995865f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889120-qw752"
Oct 30 08:00:00.541630 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.541603    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb6ead4a-d324-450f-adb6-dbbb75afc174-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889120-rbbl8\" (UID: \"fb6ead4a-d324-450f-adb6-dbbb75afc174\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889120-rbbl8"
Oct 30 08:00:00.561693 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.561529    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889120-sd6xv"
Oct 30 08:00:00.569164 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.567994    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfvf2\" (UniqueName: \"kubernetes.io/projected/a0e05a96-75ea-4636-b9d9-f08b5c1ba1ad-kube-api-access-rfvf2\") pod \"maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889120-pvxg6\" (UID: \"a0e05a96-75ea-4636-b9d9-f08b5c1ba1ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889120-pvxg6"
Oct 30 08:00:00.573764 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.573738    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkhfj\" (UniqueName: \"kubernetes.io/projected/f68a7a87-6354-4456-ab2c-17cf8995865f-kube-api-access-fkhfj\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-137122-29889120-qw752\" (UID: \"f68a7a87-6354-4456-ab2c-17cf8995865f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889120-qw752"
Oct 30 08:00:00.573979 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.573959    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889120-tww9v"
Oct 30 08:00:00.575488 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.575440    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvmp8\" (UniqueName: \"kubernetes.io/projected/fb6ead4a-d324-450f-adb6-dbbb75afc174-kube-api-access-jvmp8\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889120-rbbl8\" (UID: \"fb6ead4a-d324-450f-adb6-dbbb75afc174\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889120-rbbl8"
Oct 30 08:00:00.633604 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.633414    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889120-pvxg6"
Oct 30 08:00:00.638708 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.638631    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889120-rbbl8"
Oct 30 08:00:00.644115 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.643175    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p754p\" (UniqueName: \"kubernetes.io/projected/38de25ac-6fa2-4650-9cfa-ec23daf2ec76-kube-api-access-p754p\") pod \"maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889120-wc4p7\" (UID: \"38de25ac-6fa2-4650-9cfa-ec23daf2ec76\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889120-wc4p7"
Oct 30 08:00:00.644115 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.643223    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svn5t\" (UniqueName: \"kubernetes.io/projected/3a639cc2-221b-4e62-86c7-64ae4f7c43f7-kube-api-access-svn5t\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889120-9vz2g\" (UID: \"3a639cc2-221b-4e62-86c7-64ae4f7c43f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889120-9vz2g"
Oct 30 08:00:00.644115 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.643257    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da938ecc-f439-4873-b964-8102a4d0dfb5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-0f5081-29889120-pchn8\" (UID: \"da938ecc-f439-4873-b964-8102a4d0dfb5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f5081-29889120-pchn8"
Oct 30 08:00:00.644115 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.643326    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blkgt\" (UniqueName: \"kubernetes.io/projected/da938ecc-f439-4873-b964-8102a4d0dfb5-kube-api-access-blkgt\") pod \"maas-api-key-cleanup-e2e-extmod-a-0f5081-29889120-pchn8\" (UID: \"da938ecc-f439-4873-b964-8102a4d0dfb5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f5081-29889120-pchn8"
Oct 30 08:00:00.644115 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.643353    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a639cc2-221b-4e62-86c7-64ae4f7c43f7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889120-9vz2g\" (UID: \"3a639cc2-221b-4e62-86c7-64ae4f7c43f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889120-9vz2g"
Oct 30 08:00:00.644115 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.643554    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38de25ac-6fa2-4650-9cfa-ec23daf2ec76-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889120-wc4p7\" (UID: \"38de25ac-6fa2-4650-9cfa-ec23daf2ec76\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889120-wc4p7"
Oct 30 08:00:00.646963 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.644574    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a639cc2-221b-4e62-86c7-64ae4f7c43f7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889120-9vz2g\" (UID: \"3a639cc2-221b-4e62-86c7-64ae4f7c43f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889120-9vz2g"
Oct 30 08:00:00.646963 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.646161    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38de25ac-6fa2-4650-9cfa-ec23daf2ec76-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889120-wc4p7\" (UID: \"38de25ac-6fa2-4650-9cfa-ec23daf2ec76\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889120-wc4p7"
Oct 30 08:00:00.668305 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.668081    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889120-qw752"
Oct 30 08:00:00.686129 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.686096    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p754p\" (UniqueName: \"kubernetes.io/projected/38de25ac-6fa2-4650-9cfa-ec23daf2ec76-kube-api-access-p754p\") pod \"maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889120-wc4p7\" (UID: \"38de25ac-6fa2-4650-9cfa-ec23daf2ec76\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889120-wc4p7"
Oct 30 08:00:00.705066 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.705004    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svn5t\" (UniqueName: \"kubernetes.io/projected/3a639cc2-221b-4e62-86c7-64ae4f7c43f7-kube-api-access-svn5t\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889120-9vz2g\" (UID: \"3a639cc2-221b-4e62-86c7-64ae4f7c43f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889120-9vz2g"
Oct 30 08:00:00.719194 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.718233    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889120-9vz2g"
Oct 30 08:00:00.746717 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.745400    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da938ecc-f439-4873-b964-8102a4d0dfb5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-0f5081-29889120-pchn8\" (UID: \"da938ecc-f439-4873-b964-8102a4d0dfb5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f5081-29889120-pchn8"
Oct 30 08:00:00.746717 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.745489    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blkgt\" (UniqueName: \"kubernetes.io/projected/da938ecc-f439-4873-b964-8102a4d0dfb5-kube-api-access-blkgt\") pod \"maas-api-key-cleanup-e2e-extmod-a-0f5081-29889120-pchn8\" (UID: \"da938ecc-f439-4873-b964-8102a4d0dfb5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f5081-29889120-pchn8"
Oct 30 08:00:00.746717 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.746619    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da938ecc-f439-4873-b964-8102a4d0dfb5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-0f5081-29889120-pchn8\" (UID: \"da938ecc-f439-4873-b964-8102a4d0dfb5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f5081-29889120-pchn8"
Oct 30 08:00:00.774950 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.774672    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889120-wc4p7"
Oct 30 08:00:00.786736 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.785403    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blkgt\" (UniqueName: \"kubernetes.io/projected/da938ecc-f439-4873-b964-8102a4d0dfb5-kube-api-access-blkgt\") pod \"maas-api-key-cleanup-e2e-extmod-a-0f5081-29889120-pchn8\" (UID: \"da938ecc-f439-4873-b964-8102a4d0dfb5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f5081-29889120-pchn8"
Oct 30 08:00:00.858930 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.857863    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8e7948-29889120-jt7qq"]
Oct 30 08:00:00.885212 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.885127    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f5081-29889120-pchn8"
Oct 30 08:00:00.933621 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:00.933586    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889120-tww9v"]
Oct 30 08:00:01.064668 ip-10-0-128-125 kubenswrapper[6224]: W1030 08:00:01.064616    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf68a7a87_6354_4456_ab2c_17cf8995865f.slice/crio-3bbf6cf87a49b81b7459fed491369ccc1ac827faf7c827f1ddd4a22c8cfb59a3 WatchSource:0}: Error finding container 3bbf6cf87a49b81b7459fed491369ccc1ac827faf7c827f1ddd4a22c8cfb59a3: Status 404 returned error can't find the container with id 3bbf6cf87a49b81b7459fed491369ccc1ac827faf7c827f1ddd4a22c8cfb59a3
Oct 30 08:00:01.067777 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:01.067611    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889120-rzdrx"]
Oct 30 08:00:01.069741 ip-10-0-128-125 kubenswrapper[6224]: W1030 08:00:01.069712    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod246d2cbc_c719_46fd_a84d_0e47a21358fe.slice/crio-f43c05d3a816bd1a4cdf93bedf3ea67feb16bb81f228e5d90553a74ec43aeaf8 WatchSource:0}: Error finding container f43c05d3a816bd1a4cdf93bedf3ea67feb16bb81f228e5d90553a74ec43aeaf8: Status 404 returned error can't find the container with id f43c05d3a816bd1a4cdf93bedf3ea67feb16bb81f228e5d90553a74ec43aeaf8
Oct 30 08:00:01.072720 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:01.072609    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889120-qw752"]
Oct 30 08:00:01.128683 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:01.128658    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889120-wc4p7"]
Oct 30 08:00:01.158373 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:01.158346    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889120-sd6xv"]
Oct 30 08:00:01.177658 ip-10-0-128-125 kubenswrapper[6224]: W1030 08:00:01.177396    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38de25ac_6fa2_4650_9cfa_ec23daf2ec76.slice/crio-20751035f567d597d29791cddfc31d19204e9b121d670719eec349ddb9fe93ea WatchSource:0}: Error finding container 20751035f567d597d29791cddfc31d19204e9b121d670719eec349ddb9fe93ea: Status 404 returned error can't find the container with id 20751035f567d597d29791cddfc31d19204e9b121d670719eec349ddb9fe93ea
Oct 30 08:00:01.179926 ip-10-0-128-125 kubenswrapper[6224]: W1030 08:00:01.179163    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cd532fb_2369_434f_9622_d6792f8ee7a0.slice/crio-3db2cb58d0ddcd13cb127b85f29aea95a0e82d79e71c3d482a60320ff5b777ae WatchSource:0}: Error finding container 3db2cb58d0ddcd13cb127b85f29aea95a0e82d79e71c3d482a60320ff5b777ae: Status 404 returned error can't find the container with id 3db2cb58d0ddcd13cb127b85f29aea95a0e82d79e71c3d482a60320ff5b777ae
Oct 30 08:00:01.212852 ip-10-0-128-125 kubenswrapper[6224]: W1030 08:00:01.212781    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0e05a96_75ea_4636_b9d9_f08b5c1ba1ad.slice/crio-1ff9e658f9fe6fcf0d23be229a56f51652d73c8644d7ca53a9504bba04bbf261 WatchSource:0}: Error finding container 1ff9e658f9fe6fcf0d23be229a56f51652d73c8644d7ca53a9504bba04bbf261: Status 404 returned error can't find the container with id 1ff9e658f9fe6fcf0d23be229a56f51652d73c8644d7ca53a9504bba04bbf261
Oct 30 08:00:01.215004 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:01.214979    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889120-pvxg6"]
Oct 30 08:00:01.217052 ip-10-0-128-125 kubenswrapper[6224]: W1030 08:00:01.217025    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda938ecc_f439_4873_b964_8102a4d0dfb5.slice/crio-c7ce046417203213c3463e8bcec5f047fb731501ac5b92d6180ebdac23b49602 WatchSource:0}: Error finding container c7ce046417203213c3463e8bcec5f047fb731501ac5b92d6180ebdac23b49602: Status 404 returned error can't find the container with id c7ce046417203213c3463e8bcec5f047fb731501ac5b92d6180ebdac23b49602
Oct 30 08:00:01.219493 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:01.219469    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f5081-29889120-pchn8"]
Oct 30 08:00:01.224273 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:01.224252    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889120-rbbl8"]
Oct 30 08:00:01.225816 ip-10-0-128-125 kubenswrapper[6224]: W1030 08:00:01.225781    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb6ead4a_d324_450f_adb6_dbbb75afc174.slice/crio-8ad2969317068ebe767c26ead43855ffad9c27ad8d4bd4d3f3d916958205188f WatchSource:0}: Error finding container 8ad2969317068ebe767c26ead43855ffad9c27ad8d4bd4d3f3d916958205188f: Status 404 returned error can't find the container with id 8ad2969317068ebe767c26ead43855ffad9c27ad8d4bd4d3f3d916958205188f
Oct 30 08:00:01.271678 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:01.270745    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889120-9vz2g"]
Oct 30 08:00:01.698932 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:01.698905    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889120-wc4p7" event={"ID":"38de25ac-6fa2-4650-9cfa-ec23daf2ec76","Type":"ContainerStarted","Data":"2651bd9514dae72a315dff77666eda49877eb2c961efb2a3d3c1ff14044c7fb4"}
Oct 30 08:00:01.699062 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:01.698964    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889120-wc4p7" event={"ID":"38de25ac-6fa2-4650-9cfa-ec23daf2ec76","Type":"ContainerStarted","Data":"20751035f567d597d29791cddfc31d19204e9b121d670719eec349ddb9fe93ea"}
Oct 30 08:00:01.700305 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:01.700282    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889120-9vz2g" event={"ID":"3a639cc2-221b-4e62-86c7-64ae4f7c43f7","Type":"ContainerStarted","Data":"31c74e9cde79557257d7aa131785d563f78ef91991c8e2d757f4eb7d47cacb10"}
Oct 30 08:00:01.701623 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:01.701599    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889120-sd6xv" event={"ID":"1cd532fb-2369-434f-9622-d6792f8ee7a0","Type":"ContainerStarted","Data":"3db2cb58d0ddcd13cb127b85f29aea95a0e82d79e71c3d482a60320ff5b777ae"}
Oct 30 08:00:01.703010 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:01.702985    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889120-qw752" event={"ID":"f68a7a87-6354-4456-ab2c-17cf8995865f","Type":"ContainerStarted","Data":"3bbf6cf87a49b81b7459fed491369ccc1ac827faf7c827f1ddd4a22c8cfb59a3"}
Oct 30 08:00:01.704757 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:01.704732    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889120-rzdrx" event={"ID":"246d2cbc-c719-46fd-a84d-0e47a21358fe","Type":"ContainerStarted","Data":"44e962bb3eb4f902f1852c1d1207d16f4c9254a2704d62c3602290bf50ade759"}
Oct 30 08:00:01.704865 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:01.704776    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889120-rzdrx" event={"ID":"246d2cbc-c719-46fd-a84d-0e47a21358fe","Type":"ContainerStarted","Data":"f43c05d3a816bd1a4cdf93bedf3ea67feb16bb81f228e5d90553a74ec43aeaf8"}
Oct 30 08:00:01.706659 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:01.706397    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f5081-29889120-pchn8" event={"ID":"da938ecc-f439-4873-b964-8102a4d0dfb5","Type":"ContainerStarted","Data":"eedd8b925eec168ffed64000388d0f994bdea53bc4b1cabc2880ef1c9f1e9776"}
Oct 30 08:00:01.706659 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:01.706436    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f5081-29889120-pchn8" event={"ID":"da938ecc-f439-4873-b964-8102a4d0dfb5","Type":"ContainerStarted","Data":"c7ce046417203213c3463e8bcec5f047fb731501ac5b92d6180ebdac23b49602"}
Oct 30 08:00:01.707853 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:01.707829    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889120-pvxg6" event={"ID":"a0e05a96-75ea-4636-b9d9-f08b5c1ba1ad","Type":"ContainerStarted","Data":"1ff9e658f9fe6fcf0d23be229a56f51652d73c8644d7ca53a9504bba04bbf261"}
Oct 30 08:00:01.709471 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:01.709444    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889120-rbbl8" event={"ID":"fb6ead4a-d324-450f-adb6-dbbb75afc174","Type":"ContainerStarted","Data":"8ad2969317068ebe767c26ead43855ffad9c27ad8d4bd4d3f3d916958205188f"}
Oct 30 08:00:01.711332 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:01.711308    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889120-tww9v" event={"ID":"da7c6736-ac92-44ec-a1cf-5f835eb860b1","Type":"ContainerStarted","Data":"1297493ac2169360654e084332a2fa52a3f0c0d768b6c9494ddb62e25fb5737b"}
Oct 30 08:00:01.711452 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:01.711359    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889120-tww9v" event={"ID":"da7c6736-ac92-44ec-a1cf-5f835eb860b1","Type":"ContainerStarted","Data":"a5ee34b62b003d16f330b1eec6f6b006883571883cdb570b78ed30b3292679bd"}
Oct 30 08:00:01.713523 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:01.713498    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8e7948-29889120-jt7qq" event={"ID":"e8936cc2-40e5-4d2e-81b0-1ad1ca9e5a2a","Type":"ContainerStarted","Data":"328ff910fe28649f23dc0f2eecfb83c84969a1ec8cc225e9660077a598e5b14f"}
Oct 30 08:00:01.713626 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:01.713536    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8e7948-29889120-jt7qq" event={"ID":"e8936cc2-40e5-4d2e-81b0-1ad1ca9e5a2a","Type":"ContainerStarted","Data":"7add94ae0a2e918d383330a8dc366d59e121f75f1397af389376e692a0910385"}
Oct 30 08:00:01.749448 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:01.749404    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889120-wc4p7" podStartSLOduration=1.7493901269999999 podStartE2EDuration="1.749390127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:00:01.747836759 +0000 UTC m=+3760.357243720" watchObservedRunningTime="2026-10-30 08:00:01.749390127 +0000 UTC m=+3760.358797093"
Oct 30 08:00:01.801521 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:01.801475    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889120-rzdrx" podStartSLOduration=1.801463461 podStartE2EDuration="1.801463461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:00:01.798993173 +0000 UTC m=+3760.408400142" watchObservedRunningTime="2026-10-30 08:00:01.801463461 +0000 UTC m=+3760.410870430"
Oct 30 08:00:01.853317 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:01.853254    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8e7948-29889120-jt7qq" podStartSLOduration=1.8532351120000001 podStartE2EDuration="1.853235112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:00:01.852899527 +0000 UTC m=+3760.462306498" watchObservedRunningTime="2026-10-30 08:00:01.853235112 +0000 UTC m=+3760.462642083"
Oct 30 08:00:01.876945 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:00:01.876908    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 08:00:01.919907 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:01.919837    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f5081-29889120-pchn8" podStartSLOduration=1.919818486 podStartE2EDuration="1.919818486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:00:01.914997002 +0000 UTC m=+3760.524403972" watchObservedRunningTime="2026-10-30 08:00:01.919818486 +0000 UTC m=+3760.529225448"
Oct 30 08:00:01.971899 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:01.971849    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889120-tww9v" podStartSLOduration=1.971831841 podStartE2EDuration="1.971831841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:00:01.970408543 +0000 UTC m=+3760.579815515" watchObservedRunningTime="2026-10-30 08:00:01.971831841 +0000 UTC m=+3760.581238812"
Oct 30 08:00:02.723218 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:02.723184    6224 generic.go:356] "Generic (PLEG): container finished" podID="1cd532fb-2369-434f-9622-d6792f8ee7a0" containerID="e445a49e322efdd2919d140419c274c0fc47aa7c20f726b029cf7e97391a4018" exitCode=60
Oct 30 08:00:02.723668 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:02.723267    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889120-sd6xv" event={"ID":"1cd532fb-2369-434f-9622-d6792f8ee7a0","Type":"ContainerDied","Data":"e445a49e322efdd2919d140419c274c0fc47aa7c20f726b029cf7e97391a4018"}
Oct 30 08:00:02.723668 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:02.723528    6224 scope.go:122] "RemoveContainer" containerID="e445a49e322efdd2919d140419c274c0fc47aa7c20f726b029cf7e97391a4018"
Oct 30 08:00:02.724949 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:02.724927    6224 generic.go:356] "Generic (PLEG): container finished" podID="f68a7a87-6354-4456-ab2c-17cf8995865f" containerID="44cf4c39bdb045093ad1fab33643080088f67cc4c3a54ce62617f9916c6bed0a" exitCode=60
Oct 30 08:00:02.725079 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:02.725001    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889120-qw752" event={"ID":"f68a7a87-6354-4456-ab2c-17cf8995865f","Type":"ContainerDied","Data":"44cf4c39bdb045093ad1fab33643080088f67cc4c3a54ce62617f9916c6bed0a"}
Oct 30 08:00:02.725237 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:02.725218    6224 scope.go:122] "RemoveContainer" containerID="44cf4c39bdb045093ad1fab33643080088f67cc4c3a54ce62617f9916c6bed0a"
Oct 30 08:00:02.726731 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:02.726708    6224 generic.go:356] "Generic (PLEG): container finished" podID="246d2cbc-c719-46fd-a84d-0e47a21358fe" containerID="44e962bb3eb4f902f1852c1d1207d16f4c9254a2704d62c3602290bf50ade759" exitCode=60
Oct 30 08:00:02.726876 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:02.726841    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889120-rzdrx" event={"ID":"246d2cbc-c719-46fd-a84d-0e47a21358fe","Type":"ContainerDied","Data":"44e962bb3eb4f902f1852c1d1207d16f4c9254a2704d62c3602290bf50ade759"}
Oct 30 08:00:02.727041 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:02.727025    6224 scope.go:122] "RemoveContainer" containerID="44e962bb3eb4f902f1852c1d1207d16f4c9254a2704d62c3602290bf50ade759"
Oct 30 08:00:02.728968 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:02.728948    6224 generic.go:356] "Generic (PLEG): container finished" podID="da938ecc-f439-4873-b964-8102a4d0dfb5" containerID="eedd8b925eec168ffed64000388d0f994bdea53bc4b1cabc2880ef1c9f1e9776" exitCode=60
Oct 30 08:00:02.729029 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:02.729009    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f5081-29889120-pchn8" event={"ID":"da938ecc-f439-4873-b964-8102a4d0dfb5","Type":"ContainerDied","Data":"eedd8b925eec168ffed64000388d0f994bdea53bc4b1cabc2880ef1c9f1e9776"}
Oct 30 08:00:02.729204 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:02.729189    6224 scope.go:122] "RemoveContainer" containerID="eedd8b925eec168ffed64000388d0f994bdea53bc4b1cabc2880ef1c9f1e9776"
Oct 30 08:00:02.730706 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:02.730682    6224 generic.go:356] "Generic (PLEG): container finished" podID="a0e05a96-75ea-4636-b9d9-f08b5c1ba1ad" containerID="73f949300186bcdaa7541883772aa8662269fc785a66c4499698dce1021f2af3" exitCode=60
Oct 30 08:00:02.730839 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:02.730760    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889120-pvxg6" event={"ID":"a0e05a96-75ea-4636-b9d9-f08b5c1ba1ad","Type":"ContainerDied","Data":"73f949300186bcdaa7541883772aa8662269fc785a66c4499698dce1021f2af3"}
Oct 30 08:00:02.730948 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:02.730932    6224 scope.go:122] "RemoveContainer" containerID="73f949300186bcdaa7541883772aa8662269fc785a66c4499698dce1021f2af3"
Oct 30 08:00:02.732484 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:02.732462    6224 generic.go:356] "Generic (PLEG): container finished" podID="fb6ead4a-d324-450f-adb6-dbbb75afc174" containerID="a03e8cddfd6353f2204b5055b92944045ce88fb2731513b0af332a7504cc2aeb" exitCode=60
Oct 30 08:00:02.732746 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:02.732535    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889120-rbbl8" event={"ID":"fb6ead4a-d324-450f-adb6-dbbb75afc174","Type":"ContainerDied","Data":"a03e8cddfd6353f2204b5055b92944045ce88fb2731513b0af332a7504cc2aeb"}
Oct 30 08:00:02.732746 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:02.732714    6224 scope.go:122] "RemoveContainer" containerID="a03e8cddfd6353f2204b5055b92944045ce88fb2731513b0af332a7504cc2aeb"
Oct 30 08:00:02.734455 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:02.734433    6224 generic.go:356] "Generic (PLEG): container finished" podID="da7c6736-ac92-44ec-a1cf-5f835eb860b1" containerID="1297493ac2169360654e084332a2fa52a3f0c0d768b6c9494ddb62e25fb5737b" exitCode=60
Oct 30 08:00:02.734579 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:02.734499    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889120-tww9v" event={"ID":"da7c6736-ac92-44ec-a1cf-5f835eb860b1","Type":"ContainerDied","Data":"1297493ac2169360654e084332a2fa52a3f0c0d768b6c9494ddb62e25fb5737b"}
Oct 30 08:00:02.734832 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:02.734786    6224 scope.go:122] "RemoveContainer" containerID="1297493ac2169360654e084332a2fa52a3f0c0d768b6c9494ddb62e25fb5737b"
Oct 30 08:00:02.737167 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:02.737142    6224 generic.go:356] "Generic (PLEG): container finished" podID="e8936cc2-40e5-4d2e-81b0-1ad1ca9e5a2a" containerID="328ff910fe28649f23dc0f2eecfb83c84969a1ec8cc225e9660077a598e5b14f" exitCode=60
Oct 30 08:00:02.737298 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:02.737275    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8e7948-29889120-jt7qq" event={"ID":"e8936cc2-40e5-4d2e-81b0-1ad1ca9e5a2a","Type":"ContainerDied","Data":"328ff910fe28649f23dc0f2eecfb83c84969a1ec8cc225e9660077a598e5b14f"}
Oct 30 08:00:02.737511 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:02.737493    6224 scope.go:122] "RemoveContainer" containerID="328ff910fe28649f23dc0f2eecfb83c84969a1ec8cc225e9660077a598e5b14f"
Oct 30 08:00:02.738995 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:02.738977    6224 generic.go:356] "Generic (PLEG): container finished" podID="38de25ac-6fa2-4650-9cfa-ec23daf2ec76" containerID="2651bd9514dae72a315dff77666eda49877eb2c961efb2a3d3c1ff14044c7fb4" exitCode=60
Oct 30 08:00:02.739098 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:02.739040    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889120-wc4p7" event={"ID":"38de25ac-6fa2-4650-9cfa-ec23daf2ec76","Type":"ContainerDied","Data":"2651bd9514dae72a315dff77666eda49877eb2c961efb2a3d3c1ff14044c7fb4"}
Oct 30 08:00:02.739280 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:02.739262    6224 scope.go:122] "RemoveContainer" containerID="2651bd9514dae72a315dff77666eda49877eb2c961efb2a3d3c1ff14044c7fb4"
Oct 30 08:00:02.740823 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:02.740790    6224 generic.go:356] "Generic (PLEG): container finished" podID="3a639cc2-221b-4e62-86c7-64ae4f7c43f7" containerID="ea2b3ae4ae3dc55d96b824e011bbeb5eaef683db9dff33153cc4473e821c95e8" exitCode=60
Oct 30 08:00:02.740919 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:02.740844    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889120-9vz2g" event={"ID":"3a639cc2-221b-4e62-86c7-64ae4f7c43f7","Type":"ContainerDied","Data":"ea2b3ae4ae3dc55d96b824e011bbeb5eaef683db9dff33153cc4473e821c95e8"}
Oct 30 08:00:02.741115 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:02.741097    6224 scope.go:122] "RemoveContainer" containerID="ea2b3ae4ae3dc55d96b824e011bbeb5eaef683db9dff33153cc4473e821c95e8"
Oct 30 08:00:02.963364 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:02.963324    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1f34aa-6c5cc4f795-pqzbx" podUID="1ea09206-913c-471c-b942-4a26878ee5e6" containerName="payload-pre-processing" containerID="cri-o://3a39589607e717e403a08cb455e2d6b8de2e16c8b3edf6287415780351c62b4c" gracePeriod=25
Oct 30 08:00:03.753392 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:03.753348    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889120-pvxg6" event={"ID":"a0e05a96-75ea-4636-b9d9-f08b5c1ba1ad","Type":"ContainerStarted","Data":"4f20ed1b2fe0c37b7b75900c3c511b96dbd516a0dc9592a4ef70dc3c823fb908"}
Oct 30 08:00:03.769290 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:03.767883    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889120-rbbl8" event={"ID":"fb6ead4a-d324-450f-adb6-dbbb75afc174","Type":"ContainerStarted","Data":"a75ee3184394b52329c1f71b6a661425b3cfd5ce6257aed4a048dfcb2b0e875e"}
Oct 30 08:00:03.774064 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:03.774033    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889120-tww9v" event={"ID":"da7c6736-ac92-44ec-a1cf-5f835eb860b1","Type":"ContainerStarted","Data":"5502156ca2e53ad57f235b60bab403910e88d278665b80ba4b02dd0022d6d2b3"}
Oct 30 08:00:03.776574 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:03.776217    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8e7948-29889120-jt7qq" event={"ID":"e8936cc2-40e5-4d2e-81b0-1ad1ca9e5a2a","Type":"ContainerStarted","Data":"219beb46467f7e314db6b67a32d6f2f21c3861247ffbad8b707b3d105e242a6a"}
Oct 30 08:00:03.778393 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:03.778366    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889120-wc4p7" event={"ID":"38de25ac-6fa2-4650-9cfa-ec23daf2ec76","Type":"ContainerStarted","Data":"24445de2fee4c31d2ad221a92fccc61e872ed9198ea11a2edfe2949889318ca5"}
Oct 30 08:00:03.780961 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:03.780703    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889120-9vz2g" event={"ID":"3a639cc2-221b-4e62-86c7-64ae4f7c43f7","Type":"ContainerStarted","Data":"69fc98da6856ec84820de295422466e6f164bc43928d8451f23b7b25c2dd8659"}
Oct 30 08:00:03.784410 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:03.784389    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889120-sd6xv" event={"ID":"1cd532fb-2369-434f-9622-d6792f8ee7a0","Type":"ContainerStarted","Data":"5bc69e9ea6640c93729e86081106543b2e2fecf2e501d5949eb0a0dd9b5f456d"}
Oct 30 08:00:03.787061 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:03.787042    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889120-rzdrx" event={"ID":"246d2cbc-c719-46fd-a84d-0e47a21358fe","Type":"ContainerStarted","Data":"0cac4e45be6d6ca9fd9059ba43afb8e2f9b672a07bc076e8a9a54ded1af8edef"}
Oct 30 08:00:03.810478 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:03.810435    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889120-pvxg6" podStartSLOduration=3.810422012 podStartE2EDuration="3.810422012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:00:03.808854991 +0000 UTC m=+3762.418261961" watchObservedRunningTime="2026-10-30 08:00:03.810422012 +0000 UTC m=+3762.419828979"
Oct 30 08:00:03.882709 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:03.882657    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889120-rbbl8" podStartSLOduration=3.8826427089999997 podStartE2EDuration="3.882642709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:00:03.87843293 +0000 UTC m=+3762.487839900" watchObservedRunningTime="2026-10-30 08:00:03.882642709 +0000 UTC m=+3762.492049678"
Oct 30 08:00:03.933503 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:03.933414    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889120-sd6xv" podStartSLOduration=3.933400946 podStartE2EDuration="3.933400946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:00:03.931862302 +0000 UTC m=+3762.541269272" watchObservedRunningTime="2026-10-30 08:00:03.933400946 +0000 UTC m=+3762.542807916"
Oct 30 08:00:04.028636 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:04.028582    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889120-9vz2g" podStartSLOduration=4.02856834 podStartE2EDuration="4.02856834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:00:04.027317189 +0000 UTC m=+3762.636724158" watchObservedRunningTime="2026-10-30 08:00:04.02856834 +0000 UTC m=+3762.637975326"
Oct 30 08:00:04.799434 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:04.799395    6224 generic.go:356] "Generic (PLEG): container finished" podID="da7c6736-ac92-44ec-a1cf-5f835eb860b1" containerID="5502156ca2e53ad57f235b60bab403910e88d278665b80ba4b02dd0022d6d2b3" exitCode=60
Oct 30 08:00:04.799969 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:04.799475    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889120-tww9v" event={"ID":"da7c6736-ac92-44ec-a1cf-5f835eb860b1","Type":"ContainerDied","Data":"5502156ca2e53ad57f235b60bab403910e88d278665b80ba4b02dd0022d6d2b3"}
Oct 30 08:00:04.799969 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:04.799539    6224 scope.go:122] "RemoveContainer" containerID="1297493ac2169360654e084332a2fa52a3f0c0d768b6c9494ddb62e25fb5737b"
Oct 30 08:00:04.799969 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:04.799856    6224 scope.go:122] "RemoveContainer" containerID="5502156ca2e53ad57f235b60bab403910e88d278665b80ba4b02dd0022d6d2b3"
Oct 30 08:00:04.800232 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:00:04.800207    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889120-tww9v_odh-ai-gateway-infra(da7c6736-ac92-44ec-a1cf-5f835eb860b1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889120-tww9v" podUID="da7c6736-ac92-44ec-a1cf-5f835eb860b1"
Oct 30 08:00:04.801725 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:04.801704    6224 generic.go:356] "Generic (PLEG): container finished" podID="e8936cc2-40e5-4d2e-81b0-1ad1ca9e5a2a" containerID="219beb46467f7e314db6b67a32d6f2f21c3861247ffbad8b707b3d105e242a6a" exitCode=60
Oct 30 08:00:04.801852 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:04.801793    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8e7948-29889120-jt7qq" event={"ID":"e8936cc2-40e5-4d2e-81b0-1ad1ca9e5a2a","Type":"ContainerDied","Data":"219beb46467f7e314db6b67a32d6f2f21c3861247ffbad8b707b3d105e242a6a"}
Oct 30 08:00:04.802073 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:04.802055    6224 scope.go:122] "RemoveContainer" containerID="219beb46467f7e314db6b67a32d6f2f21c3861247ffbad8b707b3d105e242a6a"
Oct 30 08:00:04.802473 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:00:04.802373    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-8e7948-29889120-jt7qq_odh-ai-gateway-infra(e8936cc2-40e5-4d2e-81b0-1ad1ca9e5a2a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8e7948-29889120-jt7qq" podUID="e8936cc2-40e5-4d2e-81b0-1ad1ca9e5a2a"
Oct 30 08:00:04.804215 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:04.804079    6224 generic.go:356] "Generic (PLEG): container finished" podID="38de25ac-6fa2-4650-9cfa-ec23daf2ec76" containerID="24445de2fee4c31d2ad221a92fccc61e872ed9198ea11a2edfe2949889318ca5" exitCode=60
Oct 30 08:00:04.804215 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:04.804109    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889120-wc4p7" event={"ID":"38de25ac-6fa2-4650-9cfa-ec23daf2ec76","Type":"ContainerDied","Data":"24445de2fee4c31d2ad221a92fccc61e872ed9198ea11a2edfe2949889318ca5"}
Oct 30 08:00:04.804436 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:04.804413    6224 scope.go:122] "RemoveContainer" containerID="24445de2fee4c31d2ad221a92fccc61e872ed9198ea11a2edfe2949889318ca5"
Oct 30 08:00:04.804860 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:00:04.804794    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889120-wc4p7_odh-ai-gateway-infra(38de25ac-6fa2-4650-9cfa-ec23daf2ec76)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889120-wc4p7" podUID="38de25ac-6fa2-4650-9cfa-ec23daf2ec76"
Oct 30 08:00:04.806424 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:04.806402    6224 generic.go:356] "Generic (PLEG): container finished" podID="3a639cc2-221b-4e62-86c7-64ae4f7c43f7" containerID="69fc98da6856ec84820de295422466e6f164bc43928d8451f23b7b25c2dd8659" exitCode=60
Oct 30 08:00:04.806516 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:04.806446    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889120-9vz2g" event={"ID":"3a639cc2-221b-4e62-86c7-64ae4f7c43f7","Type":"ContainerDied","Data":"69fc98da6856ec84820de295422466e6f164bc43928d8451f23b7b25c2dd8659"}
Oct 30 08:00:04.806747 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:04.806723    6224 scope.go:122] "RemoveContainer" containerID="69fc98da6856ec84820de295422466e6f164bc43928d8451f23b7b25c2dd8659"
Oct 30 08:00:04.807073 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:00:04.807047    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889120-9vz2g_odh-ai-gateway-infra(3a639cc2-221b-4e62-86c7-64ae4f7c43f7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889120-9vz2g" podUID="3a639cc2-221b-4e62-86c7-64ae4f7c43f7"
Oct 30 08:00:04.808618 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:04.808596    6224 generic.go:356] "Generic (PLEG): container finished" podID="1cd532fb-2369-434f-9622-d6792f8ee7a0" containerID="5bc69e9ea6640c93729e86081106543b2e2fecf2e501d5949eb0a0dd9b5f456d" exitCode=60
Oct 30 08:00:04.808754 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:04.808651    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889120-sd6xv" event={"ID":"1cd532fb-2369-434f-9622-d6792f8ee7a0","Type":"ContainerDied","Data":"5bc69e9ea6640c93729e86081106543b2e2fecf2e501d5949eb0a0dd9b5f456d"}
Oct 30 08:00:04.808972 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:04.808948    6224 scope.go:122] "RemoveContainer" containerID="5bc69e9ea6640c93729e86081106543b2e2fecf2e501d5949eb0a0dd9b5f456d"
Oct 30 08:00:04.809259 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:00:04.809237    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889120-sd6xv_odh-ai-gateway-infra(1cd532fb-2369-434f-9622-d6792f8ee7a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889120-sd6xv" podUID="1cd532fb-2369-434f-9622-d6792f8ee7a0"
Oct 30 08:00:04.810750 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:04.810727    6224 generic.go:356] "Generic (PLEG): container finished" podID="f68a7a87-6354-4456-ab2c-17cf8995865f" containerID="a00ed259c8bf86aad1018b8d0ba3d2d6e7f482524c69519171529d326a9817c6" exitCode=60
Oct 30 08:00:04.810881 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:04.810841    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889120-qw752" event={"ID":"f68a7a87-6354-4456-ab2c-17cf8995865f","Type":"ContainerDied","Data":"a00ed259c8bf86aad1018b8d0ba3d2d6e7f482524c69519171529d326a9817c6"}
Oct 30 08:00:04.811147 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:04.811124    6224 scope.go:122] "RemoveContainer" containerID="a00ed259c8bf86aad1018b8d0ba3d2d6e7f482524c69519171529d326a9817c6"
Oct 30 08:00:04.811400 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:00:04.811373    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-137122-29889120-qw752_odh-ai-gateway-infra(f68a7a87-6354-4456-ab2c-17cf8995865f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889120-qw752" podUID="f68a7a87-6354-4456-ab2c-17cf8995865f"
Oct 30 08:00:04.812938 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:04.812916    6224 generic.go:356] "Generic (PLEG): container finished" podID="246d2cbc-c719-46fd-a84d-0e47a21358fe" containerID="0cac4e45be6d6ca9fd9059ba43afb8e2f9b672a07bc076e8a9a54ded1af8edef" exitCode=60
Oct 30 08:00:04.813042 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:04.812995    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889120-rzdrx" event={"ID":"246d2cbc-c719-46fd-a84d-0e47a21358fe","Type":"ContainerDied","Data":"0cac4e45be6d6ca9fd9059ba43afb8e2f9b672a07bc076e8a9a54ded1af8edef"}
Oct 30 08:00:04.813243 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:04.813226    6224 scope.go:122] "RemoveContainer" containerID="0cac4e45be6d6ca9fd9059ba43afb8e2f9b672a07bc076e8a9a54ded1af8edef"
Oct 30 08:00:04.813452 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:00:04.813428    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-265384-29889120-rzdrx_odh-ai-gateway-infra(246d2cbc-c719-46fd-a84d-0e47a21358fe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889120-rzdrx" podUID="246d2cbc-c719-46fd-a84d-0e47a21358fe"
Oct 30 08:00:04.815103 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:04.815080    6224 generic.go:356] "Generic (PLEG): container finished" podID="da938ecc-f439-4873-b964-8102a4d0dfb5" containerID="a1fa79242378d48ff28343fd1aeb15510c48006f1680487f22ce3211b2961a89" exitCode=60
Oct 30 08:00:04.815214 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:04.815152    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f5081-29889120-pchn8" event={"ID":"da938ecc-f439-4873-b964-8102a4d0dfb5","Type":"ContainerDied","Data":"a1fa79242378d48ff28343fd1aeb15510c48006f1680487f22ce3211b2961a89"}
Oct 30 08:00:04.815429 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:04.815408    6224 scope.go:122] "RemoveContainer" containerID="a1fa79242378d48ff28343fd1aeb15510c48006f1680487f22ce3211b2961a89"
Oct 30 08:00:04.815698 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:00:04.815672    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-0f5081-29889120-pchn8_odh-ai-gateway-infra(da938ecc-f439-4873-b964-8102a4d0dfb5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f5081-29889120-pchn8" podUID="da938ecc-f439-4873-b964-8102a4d0dfb5"
Oct 30 08:00:04.816410 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:04.816386    6224 scope.go:122] "RemoveContainer" containerID="328ff910fe28649f23dc0f2eecfb83c84969a1ec8cc225e9660077a598e5b14f"
Oct 30 08:00:04.817211 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:04.817094    6224 generic.go:356] "Generic (PLEG): container finished" podID="a0e05a96-75ea-4636-b9d9-f08b5c1ba1ad" containerID="4f20ed1b2fe0c37b7b75900c3c511b96dbd516a0dc9592a4ef70dc3c823fb908" exitCode=60
Oct 30 08:00:04.817211 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:04.817176    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889120-pvxg6" event={"ID":"a0e05a96-75ea-4636-b9d9-f08b5c1ba1ad","Type":"ContainerDied","Data":"4f20ed1b2fe0c37b7b75900c3c511b96dbd516a0dc9592a4ef70dc3c823fb908"}
Oct 30 08:00:04.817400 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:04.817381    6224 scope.go:122] "RemoveContainer" containerID="4f20ed1b2fe0c37b7b75900c3c511b96dbd516a0dc9592a4ef70dc3c823fb908"
Oct 30 08:00:04.817648 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:00:04.817609    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889120-pvxg6_odh-ai-gateway-infra(a0e05a96-75ea-4636-b9d9-f08b5c1ba1ad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889120-pvxg6" podUID="a0e05a96-75ea-4636-b9d9-f08b5c1ba1ad"
Oct 30 08:00:04.819237 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:04.819212    6224 generic.go:356] "Generic (PLEG): container finished" podID="fb6ead4a-d324-450f-adb6-dbbb75afc174" containerID="a75ee3184394b52329c1f71b6a661425b3cfd5ce6257aed4a048dfcb2b0e875e" exitCode=60
Oct 30 08:00:04.819360 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:04.819258    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889120-rbbl8" event={"ID":"fb6ead4a-d324-450f-adb6-dbbb75afc174","Type":"ContainerDied","Data":"a75ee3184394b52329c1f71b6a661425b3cfd5ce6257aed4a048dfcb2b0e875e"}
Oct 30 08:00:04.819542 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:04.819523    6224 scope.go:122] "RemoveContainer" containerID="a75ee3184394b52329c1f71b6a661425b3cfd5ce6257aed4a048dfcb2b0e875e"
Oct 30 08:00:04.819915 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:00:04.819894    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889120-rbbl8_odh-ai-gateway-infra(fb6ead4a-d324-450f-adb6-dbbb75afc174)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889120-rbbl8" podUID="fb6ead4a-d324-450f-adb6-dbbb75afc174"
Oct 30 08:00:04.830166 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:04.830143    6224 scope.go:122] "RemoveContainer" containerID="2651bd9514dae72a315dff77666eda49877eb2c961efb2a3d3c1ff14044c7fb4"
Oct 30 08:00:04.844002 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:04.843974    6224 scope.go:122] "RemoveContainer" containerID="ea2b3ae4ae3dc55d96b824e011bbeb5eaef683db9dff33153cc4473e821c95e8"
Oct 30 08:00:04.858309 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:04.858288    6224 scope.go:122] "RemoveContainer" containerID="e445a49e322efdd2919d140419c274c0fc47aa7c20f726b029cf7e97391a4018"
Oct 30 08:00:04.873463 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:04.873443    6224 scope.go:122] "RemoveContainer" containerID="44cf4c39bdb045093ad1fab33643080088f67cc4c3a54ce62617f9916c6bed0a"
Oct 30 08:00:04.887158 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:04.887136    6224 scope.go:122] "RemoveContainer" containerID="44e962bb3eb4f902f1852c1d1207d16f4c9254a2704d62c3602290bf50ade759"
Oct 30 08:00:04.901391 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:04.901369    6224 scope.go:122] "RemoveContainer" containerID="eedd8b925eec168ffed64000388d0f994bdea53bc4b1cabc2880ef1c9f1e9776"
Oct 30 08:00:04.920869 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:04.920842    6224 scope.go:122] "RemoveContainer" containerID="73f949300186bcdaa7541883772aa8662269fc785a66c4499698dce1021f2af3"
Oct 30 08:00:04.934697 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:04.934675    6224 scope.go:122] "RemoveContainer" containerID="a03e8cddfd6353f2204b5055b92944045ce88fb2731513b0af332a7504cc2aeb"
Oct 30 08:00:05.829797 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:05.829763    6224 scope.go:122] "RemoveContainer" containerID="24445de2fee4c31d2ad221a92fccc61e872ed9198ea11a2edfe2949889318ca5"
Oct 30 08:00:05.830242 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:00:05.830088    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889120-wc4p7_odh-ai-gateway-infra(38de25ac-6fa2-4650-9cfa-ec23daf2ec76)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889120-wc4p7" podUID="38de25ac-6fa2-4650-9cfa-ec23daf2ec76"
Oct 30 08:00:05.831483 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:05.831461    6224 scope.go:122] "RemoveContainer" containerID="69fc98da6856ec84820de295422466e6f164bc43928d8451f23b7b25c2dd8659"
Oct 30 08:00:05.831735 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:00:05.831713    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889120-9vz2g_odh-ai-gateway-infra(3a639cc2-221b-4e62-86c7-64ae4f7c43f7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889120-9vz2g" podUID="3a639cc2-221b-4e62-86c7-64ae4f7c43f7"
Oct 30 08:00:05.833029 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:05.833014    6224 scope.go:122] "RemoveContainer" containerID="5bc69e9ea6640c93729e86081106543b2e2fecf2e501d5949eb0a0dd9b5f456d"
Oct 30 08:00:05.833306 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:00:05.833284    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889120-sd6xv_odh-ai-gateway-infra(1cd532fb-2369-434f-9622-d6792f8ee7a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889120-sd6xv" podUID="1cd532fb-2369-434f-9622-d6792f8ee7a0"
Oct 30 08:00:05.834559 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:05.834536    6224 scope.go:122] "RemoveContainer" containerID="a00ed259c8bf86aad1018b8d0ba3d2d6e7f482524c69519171529d326a9817c6"
Oct 30 08:00:05.834802 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:00:05.834778    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-137122-29889120-qw752_odh-ai-gateway-infra(f68a7a87-6354-4456-ab2c-17cf8995865f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889120-qw752" podUID="f68a7a87-6354-4456-ab2c-17cf8995865f"
Oct 30 08:00:05.835987 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:05.835969    6224 scope.go:122] "RemoveContainer" containerID="0cac4e45be6d6ca9fd9059ba43afb8e2f9b672a07bc076e8a9a54ded1af8edef"
Oct 30 08:00:05.836170 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:00:05.836149    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-265384-29889120-rzdrx_odh-ai-gateway-infra(246d2cbc-c719-46fd-a84d-0e47a21358fe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889120-rzdrx" podUID="246d2cbc-c719-46fd-a84d-0e47a21358fe"
Oct 30 08:00:05.837545 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:05.837527    6224 scope.go:122] "RemoveContainer" containerID="a1fa79242378d48ff28343fd1aeb15510c48006f1680487f22ce3211b2961a89"
Oct 30 08:00:05.837744 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:00:05.837717    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-0f5081-29889120-pchn8_odh-ai-gateway-infra(da938ecc-f439-4873-b964-8102a4d0dfb5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f5081-29889120-pchn8" podUID="da938ecc-f439-4873-b964-8102a4d0dfb5"
Oct 30 08:00:05.838966 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:05.838950    6224 scope.go:122] "RemoveContainer" containerID="4f20ed1b2fe0c37b7b75900c3c511b96dbd516a0dc9592a4ef70dc3c823fb908"
Oct 30 08:00:05.839169 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:00:05.839150    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889120-pvxg6_odh-ai-gateway-infra(a0e05a96-75ea-4636-b9d9-f08b5c1ba1ad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889120-pvxg6" podUID="a0e05a96-75ea-4636-b9d9-f08b5c1ba1ad"
Oct 30 08:00:05.840390 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:05.840374    6224 scope.go:122] "RemoveContainer" containerID="a75ee3184394b52329c1f71b6a661425b3cfd5ce6257aed4a048dfcb2b0e875e"
Oct 30 08:00:05.840566 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:00:05.840548    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889120-rbbl8_odh-ai-gateway-infra(fb6ead4a-d324-450f-adb6-dbbb75afc174)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889120-rbbl8" podUID="fb6ead4a-d324-450f-adb6-dbbb75afc174"
Oct 30 08:00:05.841769 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:05.841751    6224 scope.go:122] "RemoveContainer" containerID="5502156ca2e53ad57f235b60bab403910e88d278665b80ba4b02dd0022d6d2b3"
Oct 30 08:00:05.841968 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:00:05.841950    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889120-tww9v_odh-ai-gateway-infra(da7c6736-ac92-44ec-a1cf-5f835eb860b1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889120-tww9v" podUID="da7c6736-ac92-44ec-a1cf-5f835eb860b1"
Oct 30 08:00:05.843163 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:05.843148    6224 scope.go:122] "RemoveContainer" containerID="219beb46467f7e314db6b67a32d6f2f21c3861247ffbad8b707b3d105e242a6a"
Oct 30 08:00:05.843331 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:00:05.843315    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-8e7948-29889120-jt7qq_odh-ai-gateway-infra(e8936cc2-40e5-4d2e-81b0-1ad1ca9e5a2a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8e7948-29889120-jt7qq" podUID="e8936cc2-40e5-4d2e-81b0-1ad1ca9e5a2a"
Oct 30 08:00:09.869612 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:09.869570    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1f34aa-6c5cc4f795-pqzbx" podUID="1ea09206-913c-471c-b942-4a26878ee5e6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:00:15.496338 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:15.496300    6224 scope.go:122] "RemoveContainer" containerID="69fc98da6856ec84820de295422466e6f164bc43928d8451f23b7b25c2dd8659"
Oct 30 08:00:15.497376 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:15.496448    6224 scope.go:122] "RemoveContainer" containerID="24445de2fee4c31d2ad221a92fccc61e872ed9198ea11a2edfe2949889318ca5"
Oct 30 08:00:15.497376 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:15.496590    6224 scope.go:122] "RemoveContainer" containerID="4f20ed1b2fe0c37b7b75900c3c511b96dbd516a0dc9592a4ef70dc3c823fb908"
Oct 30 08:00:15.497376 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:15.496707    6224 scope.go:122] "RemoveContainer" containerID="5bc69e9ea6640c93729e86081106543b2e2fecf2e501d5949eb0a0dd9b5f456d"
Oct 30 08:00:15.497376 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:15.497004    6224 scope.go:122] "RemoveContainer" containerID="a00ed259c8bf86aad1018b8d0ba3d2d6e7f482524c69519171529d326a9817c6"
Oct 30 08:00:16.497029 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:16.496887    6224 scope.go:122] "RemoveContainer" containerID="219beb46467f7e314db6b67a32d6f2f21c3861247ffbad8b707b3d105e242a6a"
Oct 30 08:00:16.929529 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:16.929497    6224 generic.go:356] "Generic (PLEG): container finished" podID="a0e05a96-75ea-4636-b9d9-f08b5c1ba1ad" containerID="e0928b014a8f70eb42c69566d6d437e040f02efc48984dc77272ba497a80a642" exitCode=60
Oct 30 08:00:16.929725 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:16.929536    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889120-pvxg6" event={"ID":"a0e05a96-75ea-4636-b9d9-f08b5c1ba1ad","Type":"ContainerDied","Data":"e0928b014a8f70eb42c69566d6d437e040f02efc48984dc77272ba497a80a642"}
Oct 30 08:00:16.929725 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:16.929607    6224 scope.go:122] "RemoveContainer" containerID="4f20ed1b2fe0c37b7b75900c3c511b96dbd516a0dc9592a4ef70dc3c823fb908"
Oct 30 08:00:16.930095 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:16.930074    6224 scope.go:122] "RemoveContainer" containerID="e0928b014a8f70eb42c69566d6d437e040f02efc48984dc77272ba497a80a642"
Oct 30 08:00:16.930456 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:00:16.930434    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889120-pvxg6_odh-ai-gateway-infra(a0e05a96-75ea-4636-b9d9-f08b5c1ba1ad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889120-pvxg6" podUID="a0e05a96-75ea-4636-b9d9-f08b5c1ba1ad"
Oct 30 08:00:16.931771 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:16.931750    6224 generic.go:356] "Generic (PLEG): container finished" podID="38de25ac-6fa2-4650-9cfa-ec23daf2ec76" containerID="3f5ffe14cbb9ed9151076a94c504cb63974c757351edd5d22a337218897b0558" exitCode=60
Oct 30 08:00:16.931902 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:16.931769    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889120-wc4p7" event={"ID":"38de25ac-6fa2-4650-9cfa-ec23daf2ec76","Type":"ContainerDied","Data":"3f5ffe14cbb9ed9151076a94c504cb63974c757351edd5d22a337218897b0558"}
Oct 30 08:00:16.932188 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:16.932171    6224 scope.go:122] "RemoveContainer" containerID="3f5ffe14cbb9ed9151076a94c504cb63974c757351edd5d22a337218897b0558"
Oct 30 08:00:16.932510 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:00:16.932487    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889120-wc4p7_odh-ai-gateway-infra(38de25ac-6fa2-4650-9cfa-ec23daf2ec76)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889120-wc4p7" podUID="38de25ac-6fa2-4650-9cfa-ec23daf2ec76"
Oct 30 08:00:16.933816 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:16.933784    6224 generic.go:356] "Generic (PLEG): container finished" podID="3a639cc2-221b-4e62-86c7-64ae4f7c43f7" containerID="bcf9cac475b729432886e179500e6b81ead12a7534da2c9bed571e2a04d79c3f" exitCode=60
Oct 30 08:00:16.933914 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:16.933836    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889120-9vz2g" event={"ID":"3a639cc2-221b-4e62-86c7-64ae4f7c43f7","Type":"ContainerDied","Data":"bcf9cac475b729432886e179500e6b81ead12a7534da2c9bed571e2a04d79c3f"}
Oct 30 08:00:16.934161 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:16.934142    6224 scope.go:122] "RemoveContainer" containerID="bcf9cac475b729432886e179500e6b81ead12a7534da2c9bed571e2a04d79c3f"
Oct 30 08:00:16.934417 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:00:16.934398    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889120-9vz2g_odh-ai-gateway-infra(3a639cc2-221b-4e62-86c7-64ae4f7c43f7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889120-9vz2g" podUID="3a639cc2-221b-4e62-86c7-64ae4f7c43f7"
Oct 30 08:00:16.935947 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:16.935926    6224 generic.go:356] "Generic (PLEG): container finished" podID="1cd532fb-2369-434f-9622-d6792f8ee7a0" containerID="0ee209820aa27e9a45158a131b79d2876e3eabb2ce0b86df8130df2e6311e478" exitCode=60
Oct 30 08:00:16.936037 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:16.935994    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889120-sd6xv" event={"ID":"1cd532fb-2369-434f-9622-d6792f8ee7a0","Type":"ContainerDied","Data":"0ee209820aa27e9a45158a131b79d2876e3eabb2ce0b86df8130df2e6311e478"}
Oct 30 08:00:16.936281 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:16.936267    6224 scope.go:122] "RemoveContainer" containerID="0ee209820aa27e9a45158a131b79d2876e3eabb2ce0b86df8130df2e6311e478"
Oct 30 08:00:16.936516 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:00:16.936497    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889120-sd6xv_odh-ai-gateway-infra(1cd532fb-2369-434f-9622-d6792f8ee7a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889120-sd6xv" podUID="1cd532fb-2369-434f-9622-d6792f8ee7a0"
Oct 30 08:00:16.937878 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:16.937857    6224 generic.go:356] "Generic (PLEG): container finished" podID="f68a7a87-6354-4456-ab2c-17cf8995865f" containerID="2d6a60fc90f0568485a7170f56366fa0213051638979a024f1f5bc6fa0d303b9" exitCode=60
Oct 30 08:00:16.937975 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:16.937917    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889120-qw752" event={"ID":"f68a7a87-6354-4456-ab2c-17cf8995865f","Type":"ContainerDied","Data":"2d6a60fc90f0568485a7170f56366fa0213051638979a024f1f5bc6fa0d303b9"}
Oct 30 08:00:16.938281 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:16.938265    6224 scope.go:122] "RemoveContainer" containerID="2d6a60fc90f0568485a7170f56366fa0213051638979a024f1f5bc6fa0d303b9"
Oct 30 08:00:16.938505 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:00:16.938488    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-137122-29889120-qw752_odh-ai-gateway-infra(f68a7a87-6354-4456-ab2c-17cf8995865f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889120-qw752" podUID="f68a7a87-6354-4456-ab2c-17cf8995865f"
Oct 30 08:00:16.944150 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:16.944131    6224 scope.go:122] "RemoveContainer" containerID="24445de2fee4c31d2ad221a92fccc61e872ed9198ea11a2edfe2949889318ca5"
Oct 30 08:00:16.958087 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:16.958064    6224 scope.go:122] "RemoveContainer" containerID="69fc98da6856ec84820de295422466e6f164bc43928d8451f23b7b25c2dd8659"
Oct 30 08:00:16.974455 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:16.974428    6224 scope.go:122] "RemoveContainer" containerID="5bc69e9ea6640c93729e86081106543b2e2fecf2e501d5949eb0a0dd9b5f456d"
Oct 30 08:00:16.994211 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:16.994195    6224 scope.go:122] "RemoveContainer" containerID="a00ed259c8bf86aad1018b8d0ba3d2d6e7f482524c69519171529d326a9817c6"
Oct 30 08:00:17.496227 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:17.496198    6224 scope.go:122] "RemoveContainer" containerID="5502156ca2e53ad57f235b60bab403910e88d278665b80ba4b02dd0022d6d2b3"
Oct 30 08:00:17.496437 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:17.496418    6224 scope.go:122] "RemoveContainer" containerID="0cac4e45be6d6ca9fd9059ba43afb8e2f9b672a07bc076e8a9a54ded1af8edef"
Oct 30 08:00:17.953858 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:17.953824    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889120-rzdrx" event={"ID":"246d2cbc-c719-46fd-a84d-0e47a21358fe","Type":"ContainerStarted","Data":"a80bf1883cfa219a36b0fd55de6da5c5f67a763da48fe79ed5835670e27e3d5a"}
Oct 30 08:00:17.957390 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:17.957365    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889120-tww9v" event={"ID":"da7c6736-ac92-44ec-a1cf-5f835eb860b1","Type":"ContainerStarted","Data":"38353dead48e22b2cf87444b949cf08a68df0a75f5dd444eeff3583b557d6f76"}
Oct 30 08:00:17.959239 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:17.959218    6224 generic.go:356] "Generic (PLEG): container finished" podID="e8936cc2-40e5-4d2e-81b0-1ad1ca9e5a2a" containerID="fb5cc03ba2cdd0782a371055d58f2ffe60b9d0c95a3178945259d5a4ee9197bd" exitCode=60
Oct 30 08:00:17.959334 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:17.959290    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8e7948-29889120-jt7qq" event={"ID":"e8936cc2-40e5-4d2e-81b0-1ad1ca9e5a2a","Type":"ContainerDied","Data":"fb5cc03ba2cdd0782a371055d58f2ffe60b9d0c95a3178945259d5a4ee9197bd"}
Oct 30 08:00:17.959387 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:17.959337    6224 scope.go:122] "RemoveContainer" containerID="219beb46467f7e314db6b67a32d6f2f21c3861247ffbad8b707b3d105e242a6a"
Oct 30 08:00:17.959671 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:17.959654    6224 scope.go:122] "RemoveContainer" containerID="fb5cc03ba2cdd0782a371055d58f2ffe60b9d0c95a3178945259d5a4ee9197bd"
Oct 30 08:00:17.959996 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:00:17.959974    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-8e7948-29889120-jt7qq_odh-ai-gateway-infra(e8936cc2-40e5-4d2e-81b0-1ad1ca9e5a2a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8e7948-29889120-jt7qq" podUID="e8936cc2-40e5-4d2e-81b0-1ad1ca9e5a2a"
Oct 30 08:00:17.992172 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:17.992149    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889120-pvxg6"]
Oct 30 08:00:18.041137 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:18.041052    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889120-sd6xv"]
Oct 30 08:00:18.091475 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:18.091445    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889120-9vz2g"]
Oct 30 08:00:18.158315 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:18.158287    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889120-wc4p7"]
Oct 30 08:00:18.181280 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:18.181192    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889120-pvxg6"
Oct 30 08:00:18.189459 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:18.189432    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889120-qw752"]
Oct 30 08:00:18.295408 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:18.295008    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0e05a96-75ea-4636-b9d9-f08b5c1ba1ad-kube-api-access-rfvf2\" (UniqueName: \"kubernetes.io/projected/a0e05a96-75ea-4636-b9d9-f08b5c1ba1ad-kube-api-access-rfvf2\") pod \"a0e05a96-75ea-4636-b9d9-f08b5c1ba1ad\" (UID: \"a0e05a96-75ea-4636-b9d9-f08b5c1ba1ad\") "
Oct 30 08:00:18.300828 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:18.300760    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0e05a96-75ea-4636-b9d9-f08b5c1ba1ad-maas-api-service-ca" pod "a0e05a96-75ea-4636-b9d9-f08b5c1ba1ad" (UID: "a0e05a96-75ea-4636-b9d9-f08b5c1ba1ad"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:00:18.301037 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:18.301013    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0e05a96-75ea-4636-b9d9-f08b5c1ba1ad-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0e05a96-75ea-4636-b9d9-f08b5c1ba1ad-maas-api-service-ca\") pod \"a0e05a96-75ea-4636-b9d9-f08b5c1ba1ad\" (UID: \"a0e05a96-75ea-4636-b9d9-f08b5c1ba1ad\") "
Oct 30 08:00:18.302251 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:18.302214    6224 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0e05a96-75ea-4636-b9d9-f08b5c1ba1ad-maas-api-service-ca\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:00:18.304455 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:18.304424    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0e05a96-75ea-4636-b9d9-f08b5c1ba1ad-kube-api-access-rfvf2" pod "a0e05a96-75ea-4636-b9d9-f08b5c1ba1ad" (UID: "a0e05a96-75ea-4636-b9d9-f08b5c1ba1ad"). InnerVolumeSpecName "kube-api-access-rfvf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:00:18.307533 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:18.307511    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889120-sd6xv"
Oct 30 08:00:18.316245 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:18.316217    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889120-9vz2g"
Oct 30 08:00:18.370293 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:18.370177    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889120-wc4p7"
Oct 30 08:00:18.403402 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:18.403375    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a639cc2-221b-4e62-86c7-64ae4f7c43f7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a639cc2-221b-4e62-86c7-64ae4f7c43f7-maas-api-service-ca\") pod \"3a639cc2-221b-4e62-86c7-64ae4f7c43f7\" (UID: \"3a639cc2-221b-4e62-86c7-64ae4f7c43f7\") "
Oct 30 08:00:18.403560 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:18.403475    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cd532fb-2369-434f-9622-d6792f8ee7a0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cd532fb-2369-434f-9622-d6792f8ee7a0-maas-api-service-ca\") pod \"1cd532fb-2369-434f-9622-d6792f8ee7a0\" (UID: \"1cd532fb-2369-434f-9622-d6792f8ee7a0\") "
Oct 30 08:00:18.403616 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:18.403577    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cd532fb-2369-434f-9622-d6792f8ee7a0-kube-api-access-lg92c\" (UniqueName: \"kubernetes.io/projected/1cd532fb-2369-434f-9622-d6792f8ee7a0-kube-api-access-lg92c\") pod \"1cd532fb-2369-434f-9622-d6792f8ee7a0\" (UID: \"1cd532fb-2369-434f-9622-d6792f8ee7a0\") "
Oct 30 08:00:18.403684 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:18.403663    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a639cc2-221b-4e62-86c7-64ae4f7c43f7-kube-api-access-svn5t\" (UniqueName: \"kubernetes.io/projected/3a639cc2-221b-4e62-86c7-64ae4f7c43f7-kube-api-access-svn5t\") pod \"3a639cc2-221b-4e62-86c7-64ae4f7c43f7\" (UID: \"3a639cc2-221b-4e62-86c7-64ae4f7c43f7\") "
Oct 30 08:00:18.404023 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:18.404000    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a639cc2-221b-4e62-86c7-64ae4f7c43f7-maas-api-service-ca" pod "3a639cc2-221b-4e62-86c7-64ae4f7c43f7" (UID: "3a639cc2-221b-4e62-86c7-64ae4f7c43f7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:00:18.404099 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:18.404034    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cd532fb-2369-434f-9622-d6792f8ee7a0-maas-api-service-ca" pod "1cd532fb-2369-434f-9622-d6792f8ee7a0" (UID: "1cd532fb-2369-434f-9622-d6792f8ee7a0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:00:18.404265 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:18.404242    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfvf2\" (UniqueName: \"kubernetes.io/projected/a0e05a96-75ea-4636-b9d9-f08b5c1ba1ad-kube-api-access-rfvf2\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:00:18.404265 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:18.404263    6224 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a639cc2-221b-4e62-86c7-64ae4f7c43f7-maas-api-service-ca\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:00:18.404436 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:18.404273    6224 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cd532fb-2369-434f-9622-d6792f8ee7a0-maas-api-service-ca\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:00:18.463996 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:18.463411    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a639cc2-221b-4e62-86c7-64ae4f7c43f7-kube-api-access-svn5t" pod "3a639cc2-221b-4e62-86c7-64ae4f7c43f7" (UID: "3a639cc2-221b-4e62-86c7-64ae4f7c43f7"). InnerVolumeSpecName "kube-api-access-svn5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:00:18.463996 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:18.463458    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cd532fb-2369-434f-9622-d6792f8ee7a0-kube-api-access-lg92c" pod "1cd532fb-2369-434f-9622-d6792f8ee7a0" (UID: "1cd532fb-2369-434f-9622-d6792f8ee7a0"). InnerVolumeSpecName "kube-api-access-lg92c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:00:18.496431 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:18.496408    6224 scope.go:122] "RemoveContainer" containerID="a1fa79242378d48ff28343fd1aeb15510c48006f1680487f22ce3211b2961a89"
Oct 30 08:00:18.496574 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:18.496524    6224 scope.go:122] "RemoveContainer" containerID="a75ee3184394b52329c1f71b6a661425b3cfd5ce6257aed4a048dfcb2b0e875e"
Oct 30 08:00:18.505078 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:18.505051    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38de25ac-6fa2-4650-9cfa-ec23daf2ec76-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38de25ac-6fa2-4650-9cfa-ec23daf2ec76-maas-api-service-ca\") pod \"38de25ac-6fa2-4650-9cfa-ec23daf2ec76\" (UID: \"38de25ac-6fa2-4650-9cfa-ec23daf2ec76\") "
Oct 30 08:00:18.505203 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:18.505108    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38de25ac-6fa2-4650-9cfa-ec23daf2ec76-kube-api-access-p754p\" (UniqueName: \"kubernetes.io/projected/38de25ac-6fa2-4650-9cfa-ec23daf2ec76-kube-api-access-p754p\") pod \"38de25ac-6fa2-4650-9cfa-ec23daf2ec76\" (UID: \"38de25ac-6fa2-4650-9cfa-ec23daf2ec76\") "
Oct 30 08:00:18.505474 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:18.505454    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lg92c\" (UniqueName: \"kubernetes.io/projected/1cd532fb-2369-434f-9622-d6792f8ee7a0-kube-api-access-lg92c\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:00:18.505561 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:18.505479    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svn5t\" (UniqueName: \"kubernetes.io/projected/3a639cc2-221b-4e62-86c7-64ae4f7c43f7-kube-api-access-svn5t\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:00:18.505630 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:18.505609    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38de25ac-6fa2-4650-9cfa-ec23daf2ec76-maas-api-service-ca" pod "38de25ac-6fa2-4650-9cfa-ec23daf2ec76" (UID: "38de25ac-6fa2-4650-9cfa-ec23daf2ec76"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:00:18.507505 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:18.507479    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38de25ac-6fa2-4650-9cfa-ec23daf2ec76-kube-api-access-p754p" pod "38de25ac-6fa2-4650-9cfa-ec23daf2ec76" (UID: "38de25ac-6fa2-4650-9cfa-ec23daf2ec76"). InnerVolumeSpecName "kube-api-access-p754p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:00:18.554953 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:18.554889    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889120-qw752"
Oct 30 08:00:18.607095 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:18.607053    6224 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38de25ac-6fa2-4650-9cfa-ec23daf2ec76-maas-api-service-ca\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:00:18.607095 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:18.607087    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p754p\" (UniqueName: \"kubernetes.io/projected/38de25ac-6fa2-4650-9cfa-ec23daf2ec76-kube-api-access-p754p\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:00:18.708231 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:18.708204    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f68a7a87-6354-4456-ab2c-17cf8995865f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f68a7a87-6354-4456-ab2c-17cf8995865f-maas-api-service-ca\") pod \"f68a7a87-6354-4456-ab2c-17cf8995865f\" (UID: \"f68a7a87-6354-4456-ab2c-17cf8995865f\") "
Oct 30 08:00:18.708407 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:18.708313    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f68a7a87-6354-4456-ab2c-17cf8995865f-kube-api-access-fkhfj\" (UniqueName: \"kubernetes.io/projected/f68a7a87-6354-4456-ab2c-17cf8995865f-kube-api-access-fkhfj\") pod \"f68a7a87-6354-4456-ab2c-17cf8995865f\" (UID: \"f68a7a87-6354-4456-ab2c-17cf8995865f\") "
Oct 30 08:00:18.708611 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:18.708588    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f68a7a87-6354-4456-ab2c-17cf8995865f-maas-api-service-ca" pod "f68a7a87-6354-4456-ab2c-17cf8995865f" (UID: "f68a7a87-6354-4456-ab2c-17cf8995865f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:00:18.708785 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:18.708768    6224 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f68a7a87-6354-4456-ab2c-17cf8995865f-maas-api-service-ca\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:00:18.764778 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:18.764740    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f68a7a87-6354-4456-ab2c-17cf8995865f-kube-api-access-fkhfj" pod "f68a7a87-6354-4456-ab2c-17cf8995865f" (UID: "f68a7a87-6354-4456-ab2c-17cf8995865f"). InnerVolumeSpecName "kube-api-access-fkhfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:00:18.809911 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:18.809850    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkhfj\" (UniqueName: \"kubernetes.io/projected/f68a7a87-6354-4456-ab2c-17cf8995865f-kube-api-access-fkhfj\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:00:18.970656 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:18.970629    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889120-wc4p7"
Oct 30 08:00:18.971145 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:18.970643    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889120-wc4p7" event={"ID":"38de25ac-6fa2-4650-9cfa-ec23daf2ec76","Type":"ContainerDied","Data":"20751035f567d597d29791cddfc31d19204e9b121d670719eec349ddb9fe93ea"}
Oct 30 08:00:18.971145 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:18.970725    6224 scope.go:122] "RemoveContainer" containerID="3f5ffe14cbb9ed9151076a94c504cb63974c757351edd5d22a337218897b0558"
Oct 30 08:00:18.972281 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:18.972264    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889120-9vz2g"
Oct 30 08:00:18.972490 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:18.972276    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889120-9vz2g" event={"ID":"3a639cc2-221b-4e62-86c7-64ae4f7c43f7","Type":"ContainerDied","Data":"31c74e9cde79557257d7aa131785d563f78ef91991c8e2d757f4eb7d47cacb10"}
Oct 30 08:00:18.974014 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:18.973973    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889120-sd6xv"
Oct 30 08:00:18.974598 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:18.973977    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889120-sd6xv" event={"ID":"1cd532fb-2369-434f-9622-d6792f8ee7a0","Type":"ContainerDied","Data":"3db2cb58d0ddcd13cb127b85f29aea95a0e82d79e71c3d482a60320ff5b777ae"}
Oct 30 08:00:18.983054 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:18.983030    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889120-qw752"
Oct 30 08:00:18.983054 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:18.983066    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889120-qw752" event={"ID":"f68a7a87-6354-4456-ab2c-17cf8995865f","Type":"ContainerDied","Data":"3bbf6cf87a49b81b7459fed491369ccc1ac827faf7c827f1ddd4a22c8cfb59a3"}
Oct 30 08:00:18.986140 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:18.986116    6224 generic.go:356] "Generic (PLEG): container finished" podID="246d2cbc-c719-46fd-a84d-0e47a21358fe" containerID="a80bf1883cfa219a36b0fd55de6da5c5f67a763da48fe79ed5835670e27e3d5a" exitCode=60
Oct 30 08:00:18.986381 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:18.986154    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889120-rzdrx" event={"ID":"246d2cbc-c719-46fd-a84d-0e47a21358fe","Type":"ContainerDied","Data":"a80bf1883cfa219a36b0fd55de6da5c5f67a763da48fe79ed5835670e27e3d5a"}
Oct 30 08:00:18.986895 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:18.986857    6224 scope.go:122] "RemoveContainer" containerID="a80bf1883cfa219a36b0fd55de6da5c5f67a763da48fe79ed5835670e27e3d5a"
Oct 30 08:00:18.987957 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:00:18.987929    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-265384-29889120-rzdrx_odh-ai-gateway-infra(246d2cbc-c719-46fd-a84d-0e47a21358fe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889120-rzdrx" podUID="246d2cbc-c719-46fd-a84d-0e47a21358fe"
Oct 30 08:00:18.988243 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:18.988222    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889120-pvxg6"
Oct 30 08:00:18.988338 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:18.988251    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889120-pvxg6" event={"ID":"a0e05a96-75ea-4636-b9d9-f08b5c1ba1ad","Type":"ContainerDied","Data":"1ff9e658f9fe6fcf0d23be229a56f51652d73c8644d7ca53a9504bba04bbf261"}
Oct 30 08:00:18.988499 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:18.988424    6224 scope.go:122] "RemoveContainer" containerID="bcf9cac475b729432886e179500e6b81ead12a7534da2c9bed571e2a04d79c3f"
Oct 30 08:00:18.992021 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:18.991202    6224 generic.go:356] "Generic (PLEG): container finished" podID="da7c6736-ac92-44ec-a1cf-5f835eb860b1" containerID="38353dead48e22b2cf87444b949cf08a68df0a75f5dd444eeff3583b557d6f76" exitCode=60
Oct 30 08:00:18.992021 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:18.991271    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889120-tww9v" event={"ID":"da7c6736-ac92-44ec-a1cf-5f835eb860b1","Type":"ContainerDied","Data":"38353dead48e22b2cf87444b949cf08a68df0a75f5dd444eeff3583b557d6f76"}
Oct 30 08:00:18.992021 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:18.991571    6224 scope.go:122] "RemoveContainer" containerID="38353dead48e22b2cf87444b949cf08a68df0a75f5dd444eeff3583b557d6f76"
Oct 30 08:00:18.992021 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:00:18.991777    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889120-tww9v_odh-ai-gateway-infra(da7c6736-ac92-44ec-a1cf-5f835eb860b1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889120-tww9v" podUID="da7c6736-ac92-44ec-a1cf-5f835eb860b1"
Oct 30 08:00:19.008399 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:19.008374    6224 scope.go:122] "RemoveContainer" containerID="0ee209820aa27e9a45158a131b79d2876e3eabb2ce0b86df8130df2e6311e478"
Oct 30 08:00:19.018087 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:19.018016    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889120-rzdrx"]
Oct 30 08:00:19.021489 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:19.021464    6224 scope.go:122] "RemoveContainer" containerID="2d6a60fc90f0568485a7170f56366fa0213051638979a024f1f5bc6fa0d303b9"
Oct 30 08:00:19.033114 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:19.033085    6224 scope.go:122] "RemoveContainer" containerID="0cac4e45be6d6ca9fd9059ba43afb8e2f9b672a07bc076e8a9a54ded1af8edef"
Oct 30 08:00:19.048403 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:19.048381    6224 scope.go:122] "RemoveContainer" containerID="e0928b014a8f70eb42c69566d6d437e040f02efc48984dc77272ba497a80a642"
Oct 30 08:00:19.060017 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:19.059933    6224 scope.go:122] "RemoveContainer" containerID="5502156ca2e53ad57f235b60bab403910e88d278665b80ba4b02dd0022d6d2b3"
Oct 30 08:00:19.085304 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:19.085278    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889120-tww9v"]
Oct 30 08:00:19.150278 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:19.150245    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889120-qw752"]
Oct 30 08:00:19.162196 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:19.162169    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-137122-29889120-qw752"]
Oct 30 08:00:19.206743 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:19.206710    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889120-sd6xv"]
Oct 30 08:00:19.214122 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:19.214099    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8e7948-29889120-jt7qq"]
Oct 30 08:00:19.217390 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:19.217367    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d09eb6-29889120-sd6xv"]
Oct 30 08:00:19.314723 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:19.314657    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889120-pvxg6"]
Oct 30 08:00:19.324479 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:19.324446    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2eda2c-29889120-pvxg6"]
Oct 30 08:00:19.362439 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:19.362410    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889120-9vz2g"]
Oct 30 08:00:19.379879 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:19.379851    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-932c00-29889120-9vz2g"]
Oct 30 08:00:19.389405 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:19.389376    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8e7948-29889120-jt7qq"
Oct 30 08:00:19.421273 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:19.421241    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889120-wc4p7"]
Oct 30 08:00:19.426488 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:19.426462    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bab4c-29889120-wc4p7"]
Oct 30 08:00:19.507743 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:19.507695    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cd532fb-2369-434f-9622-d6792f8ee7a0" path="/var/lib/kubelet/pods/1cd532fb-2369-434f-9622-d6792f8ee7a0/volumes"
Oct 30 08:00:19.508264 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:19.508243    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38de25ac-6fa2-4650-9cfa-ec23daf2ec76" path="/var/lib/kubelet/pods/38de25ac-6fa2-4650-9cfa-ec23daf2ec76/volumes"
Oct 30 08:00:19.508647 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:19.508633    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a639cc2-221b-4e62-86c7-64ae4f7c43f7" path="/var/lib/kubelet/pods/3a639cc2-221b-4e62-86c7-64ae4f7c43f7/volumes"
Oct 30 08:00:19.509421 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:19.509402    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0e05a96-75ea-4636-b9d9-f08b5c1ba1ad" path="/var/lib/kubelet/pods/a0e05a96-75ea-4636-b9d9-f08b5c1ba1ad/volumes"
Oct 30 08:00:19.509800 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:19.509787    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f68a7a87-6354-4456-ab2c-17cf8995865f" path="/var/lib/kubelet/pods/f68a7a87-6354-4456-ab2c-17cf8995865f/volumes"
Oct 30 08:00:19.519148 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:19.519123    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8936cc2-40e5-4d2e-81b0-1ad1ca9e5a2a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8936cc2-40e5-4d2e-81b0-1ad1ca9e5a2a-maas-api-service-ca\") pod \"e8936cc2-40e5-4d2e-81b0-1ad1ca9e5a2a\" (UID: \"e8936cc2-40e5-4d2e-81b0-1ad1ca9e5a2a\") "
Oct 30 08:00:19.519342 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:19.519327    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8936cc2-40e5-4d2e-81b0-1ad1ca9e5a2a-kube-api-access-4r5hb\" (UniqueName: \"kubernetes.io/projected/e8936cc2-40e5-4d2e-81b0-1ad1ca9e5a2a-kube-api-access-4r5hb\") pod \"e8936cc2-40e5-4d2e-81b0-1ad1ca9e5a2a\" (UID: \"e8936cc2-40e5-4d2e-81b0-1ad1ca9e5a2a\") "
Oct 30 08:00:19.519503 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:19.519480    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8936cc2-40e5-4d2e-81b0-1ad1ca9e5a2a-maas-api-service-ca" pod "e8936cc2-40e5-4d2e-81b0-1ad1ca9e5a2a" (UID: "e8936cc2-40e5-4d2e-81b0-1ad1ca9e5a2a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:00:19.519708 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:19.519692    6224 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8936cc2-40e5-4d2e-81b0-1ad1ca9e5a2a-maas-api-service-ca\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:00:19.521446 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:19.521421    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8936cc2-40e5-4d2e-81b0-1ad1ca9e5a2a-kube-api-access-4r5hb" pod "e8936cc2-40e5-4d2e-81b0-1ad1ca9e5a2a" (UID: "e8936cc2-40e5-4d2e-81b0-1ad1ca9e5a2a"). InnerVolumeSpecName "kube-api-access-4r5hb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:00:19.620917 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:19.620886    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4r5hb\" (UniqueName: \"kubernetes.io/projected/e8936cc2-40e5-4d2e-81b0-1ad1ca9e5a2a-kube-api-access-4r5hb\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:00:19.868393 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:19.868311    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1f34aa-6c5cc4f795-pqzbx" podUID="1ea09206-913c-471c-b942-4a26878ee5e6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:00:20.010637 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:20.010607    6224 generic.go:356] "Generic (PLEG): container finished" podID="da938ecc-f439-4873-b964-8102a4d0dfb5" containerID="fffffc119c6fca9895f5560ec5383c63fffbb530711376213adb9f38ffef9a58" exitCode=60
Oct 30 08:00:20.011067 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:20.010682    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f5081-29889120-pchn8" event={"ID":"da938ecc-f439-4873-b964-8102a4d0dfb5","Type":"ContainerDied","Data":"fffffc119c6fca9895f5560ec5383c63fffbb530711376213adb9f38ffef9a58"}
Oct 30 08:00:20.011067 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:20.010752    6224 scope.go:122] "RemoveContainer" containerID="a1fa79242378d48ff28343fd1aeb15510c48006f1680487f22ce3211b2961a89"
Oct 30 08:00:20.011186 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:20.011140    6224 scope.go:122] "RemoveContainer" containerID="fffffc119c6fca9895f5560ec5383c63fffbb530711376213adb9f38ffef9a58"
Oct 30 08:00:20.011503 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:00:20.011474    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-0f5081-29889120-pchn8_odh-ai-gateway-infra(da938ecc-f439-4873-b964-8102a4d0dfb5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f5081-29889120-pchn8" podUID="da938ecc-f439-4873-b964-8102a4d0dfb5"
Oct 30 08:00:20.014120 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:20.014100    6224 generic.go:356] "Generic (PLEG): container finished" podID="fb6ead4a-d324-450f-adb6-dbbb75afc174" containerID="1a70d7a36a143224634a528b60e214203b02c1a4b22a8a1f447c1d2994f9109c" exitCode=60
Oct 30 08:00:20.014207 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:20.014173    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889120-rbbl8" event={"ID":"fb6ead4a-d324-450f-adb6-dbbb75afc174","Type":"ContainerDied","Data":"1a70d7a36a143224634a528b60e214203b02c1a4b22a8a1f447c1d2994f9109c"}
Oct 30 08:00:20.014552 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:20.014536    6224 scope.go:122] "RemoveContainer" containerID="1a70d7a36a143224634a528b60e214203b02c1a4b22a8a1f447c1d2994f9109c"
Oct 30 08:00:20.014884 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:00:20.014858    6224 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889120-rbbl8_odh-ai-gateway-infra(fb6ead4a-d324-450f-adb6-dbbb75afc174)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889120-rbbl8" podUID="fb6ead4a-d324-450f-adb6-dbbb75afc174"
Oct 30 08:00:20.017397 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:20.017380    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8e7948-29889120-jt7qq"
Oct 30 08:00:20.017502 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:20.017477    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8e7948-29889120-jt7qq" event={"ID":"e8936cc2-40e5-4d2e-81b0-1ad1ca9e5a2a","Type":"ContainerDied","Data":"7add94ae0a2e918d383330a8dc366d59e121f75f1397af389376e692a0910385"}
Oct 30 08:00:20.027337 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:20.027217    6224 scope.go:122] "RemoveContainer" containerID="a75ee3184394b52329c1f71b6a661425b3cfd5ce6257aed4a048dfcb2b0e875e"
Oct 30 08:00:20.065248 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:20.065221    6224 scope.go:122] "RemoveContainer" containerID="fb5cc03ba2cdd0782a371055d58f2ffe60b9d0c95a3178945259d5a4ee9197bd"
Oct 30 08:00:20.118769 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:20.118696    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8e7948-29889120-jt7qq"]
Oct 30 08:00:20.127167 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:20.127091    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8e7948-29889120-jt7qq"]
Oct 30 08:00:20.206622 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:20.206595    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889120-rzdrx"
Oct 30 08:00:20.225049 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:20.225024    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889120-tww9v"
Oct 30 08:00:20.329515 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:20.329480    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/246d2cbc-c719-46fd-a84d-0e47a21358fe-kube-api-access-rv4rq\" (UniqueName: \"kubernetes.io/projected/246d2cbc-c719-46fd-a84d-0e47a21358fe-kube-api-access-rv4rq\") pod \"246d2cbc-c719-46fd-a84d-0e47a21358fe\" (UID: \"246d2cbc-c719-46fd-a84d-0e47a21358fe\") "
Oct 30 08:00:20.329711 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:20.329596    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da7c6736-ac92-44ec-a1cf-5f835eb860b1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da7c6736-ac92-44ec-a1cf-5f835eb860b1-maas-api-service-ca\") pod \"da7c6736-ac92-44ec-a1cf-5f835eb860b1\" (UID: \"da7c6736-ac92-44ec-a1cf-5f835eb860b1\") "
Oct 30 08:00:20.329711 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:20.329659    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/246d2cbc-c719-46fd-a84d-0e47a21358fe-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/246d2cbc-c719-46fd-a84d-0e47a21358fe-maas-api-service-ca\") pod \"246d2cbc-c719-46fd-a84d-0e47a21358fe\" (UID: \"246d2cbc-c719-46fd-a84d-0e47a21358fe\") "
Oct 30 08:00:20.329848 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:20.329744    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da7c6736-ac92-44ec-a1cf-5f835eb860b1-kube-api-access-t24rl\" (UniqueName: \"kubernetes.io/projected/da7c6736-ac92-44ec-a1cf-5f835eb860b1-kube-api-access-t24rl\") pod \"da7c6736-ac92-44ec-a1cf-5f835eb860b1\" (UID: \"da7c6736-ac92-44ec-a1cf-5f835eb860b1\") "
Oct 30 08:00:20.330063 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:20.330029    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/246d2cbc-c719-46fd-a84d-0e47a21358fe-maas-api-service-ca" pod "246d2cbc-c719-46fd-a84d-0e47a21358fe" (UID: "246d2cbc-c719-46fd-a84d-0e47a21358fe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:00:20.330063 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:20.330040    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da7c6736-ac92-44ec-a1cf-5f835eb860b1-maas-api-service-ca" pod "da7c6736-ac92-44ec-a1cf-5f835eb860b1" (UID: "da7c6736-ac92-44ec-a1cf-5f835eb860b1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:00:20.330472 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:20.330449    6224 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da7c6736-ac92-44ec-a1cf-5f835eb860b1-maas-api-service-ca\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:00:20.330575 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:20.330479    6224 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/246d2cbc-c719-46fd-a84d-0e47a21358fe-maas-api-service-ca\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:00:20.331834 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:20.331789    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/246d2cbc-c719-46fd-a84d-0e47a21358fe-kube-api-access-rv4rq" pod "246d2cbc-c719-46fd-a84d-0e47a21358fe" (UID: "246d2cbc-c719-46fd-a84d-0e47a21358fe"). InnerVolumeSpecName "kube-api-access-rv4rq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:00:20.331943 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:20.331855    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da7c6736-ac92-44ec-a1cf-5f835eb860b1-kube-api-access-t24rl" pod "da7c6736-ac92-44ec-a1cf-5f835eb860b1" (UID: "da7c6736-ac92-44ec-a1cf-5f835eb860b1"). InnerVolumeSpecName "kube-api-access-t24rl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:00:20.432575 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:20.432455    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rv4rq\" (UniqueName: \"kubernetes.io/projected/246d2cbc-c719-46fd-a84d-0e47a21358fe-kube-api-access-rv4rq\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:00:20.432575 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:20.432502    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t24rl\" (UniqueName: \"kubernetes.io/projected/da7c6736-ac92-44ec-a1cf-5f835eb860b1-kube-api-access-t24rl\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:00:21.032983 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:21.032938    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889120-tww9v" event={"ID":"da7c6736-ac92-44ec-a1cf-5f835eb860b1","Type":"ContainerDied","Data":"a5ee34b62b003d16f330b1eec6f6b006883571883cdb570b78ed30b3292679bd"}
Oct 30 08:00:21.033498 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:21.033019    6224 scope.go:122] "RemoveContainer" containerID="38353dead48e22b2cf87444b949cf08a68df0a75f5dd444eeff3583b557d6f76"
Oct 30 08:00:21.033498 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:21.032952    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889120-tww9v"
Oct 30 08:00:21.035678 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:21.035650    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889120-rzdrx" event={"ID":"246d2cbc-c719-46fd-a84d-0e47a21358fe","Type":"ContainerDied","Data":"f43c05d3a816bd1a4cdf93bedf3ea67feb16bb81f228e5d90553a74ec43aeaf8"}
Oct 30 08:00:21.035871 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:21.035679    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889120-rzdrx"
Oct 30 08:00:21.044972 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:21.044950    6224 scope.go:122] "RemoveContainer" containerID="a80bf1883cfa219a36b0fd55de6da5c5f67a763da48fe79ed5835670e27e3d5a"
Oct 30 08:00:21.155464 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:21.155434    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889120-tww9v"]
Oct 30 08:00:21.171208 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:21.171184    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ace8fb-29889120-tww9v"]
Oct 30 08:00:21.201181 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:21.201155    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889120-rbbl8"]
Oct 30 08:00:21.228637 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:21.228596    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889120-rzdrx"]
Oct 30 08:00:21.241224 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:21.241191    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-265384-29889120-rzdrx"]
Oct 30 08:00:21.258293 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:21.256699    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f5081-29889120-pchn8"]
Oct 30 08:00:21.394441 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:21.394411    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889120-rbbl8"
Oct 30 08:00:21.445961 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:21.445932    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb6ead4a-d324-450f-adb6-dbbb75afc174-kube-api-access-jvmp8\" (UniqueName: \"kubernetes.io/projected/fb6ead4a-d324-450f-adb6-dbbb75afc174-kube-api-access-jvmp8\") pod \"fb6ead4a-d324-450f-adb6-dbbb75afc174\" (UID: \"fb6ead4a-d324-450f-adb6-dbbb75afc174\") "
Oct 30 08:00:21.446147 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:21.446017    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb6ead4a-d324-450f-adb6-dbbb75afc174-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb6ead4a-d324-450f-adb6-dbbb75afc174-maas-api-service-ca\") pod \"fb6ead4a-d324-450f-adb6-dbbb75afc174\" (UID: \"fb6ead4a-d324-450f-adb6-dbbb75afc174\") "
Oct 30 08:00:21.446674 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:21.446648    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb6ead4a-d324-450f-adb6-dbbb75afc174-maas-api-service-ca" pod "fb6ead4a-d324-450f-adb6-dbbb75afc174" (UID: "fb6ead4a-d324-450f-adb6-dbbb75afc174"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:00:21.449927 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:21.449888    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb6ead4a-d324-450f-adb6-dbbb75afc174-kube-api-access-jvmp8" pod "fb6ead4a-d324-450f-adb6-dbbb75afc174" (UID: "fb6ead4a-d324-450f-adb6-dbbb75afc174"). InnerVolumeSpecName "kube-api-access-jvmp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:00:21.503253 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:21.503218    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f5081-29889120-pchn8"
Oct 30 08:00:21.517326 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:21.517294    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="246d2cbc-c719-46fd-a84d-0e47a21358fe" path="/var/lib/kubelet/pods/246d2cbc-c719-46fd-a84d-0e47a21358fe/volumes"
Oct 30 08:00:21.517950 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:21.517932    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da7c6736-ac92-44ec-a1cf-5f835eb860b1" path="/var/lib/kubelet/pods/da7c6736-ac92-44ec-a1cf-5f835eb860b1/volumes"
Oct 30 08:00:21.518505 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:21.518489    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8936cc2-40e5-4d2e-81b0-1ad1ca9e5a2a" path="/var/lib/kubelet/pods/e8936cc2-40e5-4d2e-81b0-1ad1ca9e5a2a/volumes"
Oct 30 08:00:21.547271 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:21.547242    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvmp8\" (UniqueName: \"kubernetes.io/projected/fb6ead4a-d324-450f-adb6-dbbb75afc174-kube-api-access-jvmp8\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:00:21.547271 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:21.547269    6224 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb6ead4a-d324-450f-adb6-dbbb75afc174-maas-api-service-ca\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:00:21.648509 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:21.648435    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da938ecc-f439-4873-b964-8102a4d0dfb5-kube-api-access-blkgt\" (UniqueName: \"kubernetes.io/projected/da938ecc-f439-4873-b964-8102a4d0dfb5-kube-api-access-blkgt\") pod \"da938ecc-f439-4873-b964-8102a4d0dfb5\" (UID: \"da938ecc-f439-4873-b964-8102a4d0dfb5\") "
Oct 30 08:00:21.648667 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:21.648651    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da938ecc-f439-4873-b964-8102a4d0dfb5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da938ecc-f439-4873-b964-8102a4d0dfb5-maas-api-service-ca\") pod \"da938ecc-f439-4873-b964-8102a4d0dfb5\" (UID: \"da938ecc-f439-4873-b964-8102a4d0dfb5\") "
Oct 30 08:00:21.649461 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:21.649426    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da938ecc-f439-4873-b964-8102a4d0dfb5-maas-api-service-ca" pod "da938ecc-f439-4873-b964-8102a4d0dfb5" (UID: "da938ecc-f439-4873-b964-8102a4d0dfb5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:00:21.649585 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:21.649554    6224 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da938ecc-f439-4873-b964-8102a4d0dfb5-maas-api-service-ca\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:00:21.651220 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:21.651161    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da938ecc-f439-4873-b964-8102a4d0dfb5-kube-api-access-blkgt" pod "da938ecc-f439-4873-b964-8102a4d0dfb5" (UID: "da938ecc-f439-4873-b964-8102a4d0dfb5"). InnerVolumeSpecName "kube-api-access-blkgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:00:21.750568 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:21.750533    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blkgt\" (UniqueName: \"kubernetes.io/projected/da938ecc-f439-4873-b964-8102a4d0dfb5-kube-api-access-blkgt\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:00:22.045673 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:22.045639    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f5081-29889120-pchn8"
Oct 30 08:00:22.046179 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:22.045678    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f5081-29889120-pchn8" event={"ID":"da938ecc-f439-4873-b964-8102a4d0dfb5","Type":"ContainerDied","Data":"c7ce046417203213c3463e8bcec5f047fb731501ac5b92d6180ebdac23b49602"}
Oct 30 08:00:22.046179 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:22.045746    6224 scope.go:122] "RemoveContainer" containerID="fffffc119c6fca9895f5560ec5383c63fffbb530711376213adb9f38ffef9a58"
Oct 30 08:00:22.047090 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:22.047076    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889120-rbbl8"
Oct 30 08:00:22.047181 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:22.047090    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889120-rbbl8" event={"ID":"fb6ead4a-d324-450f-adb6-dbbb75afc174","Type":"ContainerDied","Data":"8ad2969317068ebe767c26ead43855ffad9c27ad8d4bd4d3f3d916958205188f"}
Oct 30 08:00:22.056866 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:22.056849    6224 scope.go:122] "RemoveContainer" containerID="1a70d7a36a143224634a528b60e214203b02c1a4b22a8a1f447c1d2994f9109c"
Oct 30 08:00:22.107776 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:22.107751    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889120-rbbl8"]
Oct 30 08:00:22.121581 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:22.121560    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bd8589-29889120-rbbl8"]
Oct 30 08:00:22.160140 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:22.160118    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f5081-29889120-pchn8"]
Oct 30 08:00:22.163386 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:22.163363    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f5081-29889120-pchn8"]
Oct 30 08:00:23.249223 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:23.249194    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1f34aa-6c5cc4f795-pqzbx"
Oct 30 08:00:23.368314 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:23.368283    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ea09206-913c-471c-b942-4a26878ee5e6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ea09206-913c-471c-b942-4a26878ee5e6-plugins-config-volume\") pod \"1ea09206-913c-471c-b942-4a26878ee5e6\" (UID: \"1ea09206-913c-471c-b942-4a26878ee5e6\") "
Oct 30 08:00:23.368530 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:23.368466    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ea09206-913c-471c-b942-4a26878ee5e6-kube-api-access-wgx6f\" (UniqueName: \"kubernetes.io/projected/1ea09206-913c-471c-b942-4a26878ee5e6-kube-api-access-wgx6f\") pod \"1ea09206-913c-471c-b942-4a26878ee5e6\" (UID: \"1ea09206-913c-471c-b942-4a26878ee5e6\") "
Oct 30 08:00:23.368725 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:23.368704    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ea09206-913c-471c-b942-4a26878ee5e6-plugins-config-volume" pod "1ea09206-913c-471c-b942-4a26878ee5e6" (UID: "1ea09206-913c-471c-b942-4a26878ee5e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:00:23.369074 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:23.369054    6224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ea09206-913c-471c-b942-4a26878ee5e6-plugins-config-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:00:23.370874 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:23.370841    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ea09206-913c-471c-b942-4a26878ee5e6-kube-api-access-wgx6f" pod "1ea09206-913c-471c-b942-4a26878ee5e6" (UID: "1ea09206-913c-471c-b942-4a26878ee5e6"). InnerVolumeSpecName "kube-api-access-wgx6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:00:23.470484 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:23.470450    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgx6f\" (UniqueName: \"kubernetes.io/projected/1ea09206-913c-471c-b942-4a26878ee5e6-kube-api-access-wgx6f\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:00:23.504586 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:23.504506    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da938ecc-f439-4873-b964-8102a4d0dfb5" path="/var/lib/kubelet/pods/da938ecc-f439-4873-b964-8102a4d0dfb5/volumes"
Oct 30 08:00:23.505031 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:23.505012    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb6ead4a-d324-450f-adb6-dbbb75afc174" path="/var/lib/kubelet/pods/fb6ead4a-d324-450f-adb6-dbbb75afc174/volumes"
Oct 30 08:00:24.069521 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:24.069487    6224 generic.go:356] "Generic (PLEG): container finished" podID="1ea09206-913c-471c-b942-4a26878ee5e6" containerID="3a39589607e717e403a08cb455e2d6b8de2e16c8b3edf6287415780351c62b4c" exitCode=0
Oct 30 08:00:24.069687 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:24.069532    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1f34aa-6c5cc4f795-pqzbx"
Oct 30 08:00:24.069687 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:24.069582    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1f34aa-6c5cc4f795-pqzbx" event={"ID":"1ea09206-913c-471c-b942-4a26878ee5e6","Type":"ContainerDied","Data":"3a39589607e717e403a08cb455e2d6b8de2e16c8b3edf6287415780351c62b4c"}
Oct 30 08:00:24.069687 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:24.069634    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1f34aa-6c5cc4f795-pqzbx" event={"ID":"1ea09206-913c-471c-b942-4a26878ee5e6","Type":"ContainerDied","Data":"ad3ce3c71e4195aba0cee6b1f5bb0ebf339240efe0eb7dcc0b77ddb186427b5f"}
Oct 30 08:00:24.069687 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:24.069658    6224 scope.go:122] "RemoveContainer" containerID="3a39589607e717e403a08cb455e2d6b8de2e16c8b3edf6287415780351c62b4c"
Oct 30 08:00:24.080331 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:24.080310    6224 scope.go:122] "RemoveContainer" containerID="3a39589607e717e403a08cb455e2d6b8de2e16c8b3edf6287415780351c62b4c"
Oct 30 08:00:24.080662 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:00:24.080636    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a39589607e717e403a08cb455e2d6b8de2e16c8b3edf6287415780351c62b4c\": container with ID starting with 3a39589607e717e403a08cb455e2d6b8de2e16c8b3edf6287415780351c62b4c not found: ID does not exist" containerID="3a39589607e717e403a08cb455e2d6b8de2e16c8b3edf6287415780351c62b4c"
Oct 30 08:00:24.080731 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:24.080669    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a39589607e717e403a08cb455e2d6b8de2e16c8b3edf6287415780351c62b4c"} err="failed to get container status \"3a39589607e717e403a08cb455e2d6b8de2e16c8b3edf6287415780351c62b4c\": rpc error: code = NotFound desc = could not find container \"3a39589607e717e403a08cb455e2d6b8de2e16c8b3edf6287415780351c62b4c\": container with ID starting with 3a39589607e717e403a08cb455e2d6b8de2e16c8b3edf6287415780351c62b4c not found: ID does not exist"
Oct 30 08:00:24.131485 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:24.131451    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-1f34aa-6c5cc4f795-pqzbx"]
Oct 30 08:00:24.145512 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:24.145486    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-1f34aa-6c5cc4f795-pqzbx"]
Oct 30 08:00:25.504647 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:25.504614    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ea09206-913c-471c-b942-4a26878ee5e6" path="/var/lib/kubelet/pods/1ea09206-913c-471c-b942-4a26878ee5e6/volumes"
Oct 30 08:00:31.859166 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:00:31.859126    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 08:00:39.599975 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:39.599898    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-1f34aa-5c4b4fdd4f-jwkrk"]
Oct 30 08:00:39.600435 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:39.600121    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1f34aa-5c4b4fdd4f-jwkrk" podUID="bc9e4c3b-9885-4706-817b-4da954d01dfb" containerName="maas-api" containerID="cri-o://96f4170e881ace5f96838eee8522340a6e7f8ecd851c9cea3a201e47e5fe0742" gracePeriod=30
Oct 30 08:00:39.912328 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:39.912293    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1f34aa-5c4b4fdd4f-jwkrk"
Oct 30 08:00:39.970179 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:39.970141    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc9e4c3b-9885-4706-817b-4da954d01dfb-kube-api-access-2gh5q\" (UniqueName: \"kubernetes.io/projected/bc9e4c3b-9885-4706-817b-4da954d01dfb-kube-api-access-2gh5q\") pod \"bc9e4c3b-9885-4706-817b-4da954d01dfb\" (UID: \"bc9e4c3b-9885-4706-817b-4da954d01dfb\") "
Oct 30 08:00:39.970367 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:39.970224    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc9e4c3b-9885-4706-817b-4da954d01dfb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc9e4c3b-9885-4706-817b-4da954d01dfb-metrics-certs\") pod \"bc9e4c3b-9885-4706-817b-4da954d01dfb\" (UID: \"bc9e4c3b-9885-4706-817b-4da954d01dfb\") "
Oct 30 08:00:39.970367 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:39.970254    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc9e4c3b-9885-4706-817b-4da954d01dfb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc9e4c3b-9885-4706-817b-4da954d01dfb-maas-api-tls\") pod \"bc9e4c3b-9885-4706-817b-4da954d01dfb\" (UID: \"bc9e4c3b-9885-4706-817b-4da954d01dfb\") "
Oct 30 08:00:39.970513 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:39.970418    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc9e4c3b-9885-4706-817b-4da954d01dfb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc9e4c3b-9885-4706-817b-4da954d01dfb-openshift-service-ca\") pod \"bc9e4c3b-9885-4706-817b-4da954d01dfb\" (UID: \"bc9e4c3b-9885-4706-817b-4da954d01dfb\") "
Oct 30 08:00:39.970855 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:39.970824    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc9e4c3b-9885-4706-817b-4da954d01dfb-openshift-service-ca" pod "bc9e4c3b-9885-4706-817b-4da954d01dfb" (UID: "bc9e4c3b-9885-4706-817b-4da954d01dfb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:00:39.971008 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:39.970991    6224 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc9e4c3b-9885-4706-817b-4da954d01dfb-openshift-service-ca\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:00:39.973047 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:39.973023    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc9e4c3b-9885-4706-817b-4da954d01dfb-maas-api-tls" pod "bc9e4c3b-9885-4706-817b-4da954d01dfb" (UID: "bc9e4c3b-9885-4706-817b-4da954d01dfb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:00:39.973611 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:39.973583    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc9e4c3b-9885-4706-817b-4da954d01dfb-kube-api-access-2gh5q" pod "bc9e4c3b-9885-4706-817b-4da954d01dfb" (UID: "bc9e4c3b-9885-4706-817b-4da954d01dfb"). InnerVolumeSpecName "kube-api-access-2gh5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:00:39.974567 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:39.974542    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc9e4c3b-9885-4706-817b-4da954d01dfb-metrics-certs" pod "bc9e4c3b-9885-4706-817b-4da954d01dfb" (UID: "bc9e4c3b-9885-4706-817b-4da954d01dfb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:00:40.072381 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:40.072338    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gh5q\" (UniqueName: \"kubernetes.io/projected/bc9e4c3b-9885-4706-817b-4da954d01dfb-kube-api-access-2gh5q\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:00:40.072381 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:40.072380    6224 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc9e4c3b-9885-4706-817b-4da954d01dfb-metrics-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:00:40.072381 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:40.072390    6224 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc9e4c3b-9885-4706-817b-4da954d01dfb-maas-api-tls\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:00:40.185147 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:40.185047    6224 generic.go:356] "Generic (PLEG): container finished" podID="bc9e4c3b-9885-4706-817b-4da954d01dfb" containerID="96f4170e881ace5f96838eee8522340a6e7f8ecd851c9cea3a201e47e5fe0742" exitCode=0
Oct 30 08:00:40.185147 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:40.185095    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1f34aa-5c4b4fdd4f-jwkrk"
Oct 30 08:00:40.185147 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:40.185131    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1f34aa-5c4b4fdd4f-jwkrk" event={"ID":"bc9e4c3b-9885-4706-817b-4da954d01dfb","Type":"ContainerDied","Data":"96f4170e881ace5f96838eee8522340a6e7f8ecd851c9cea3a201e47e5fe0742"}
Oct 30 08:00:40.185425 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:40.185193    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1f34aa-5c4b4fdd4f-jwkrk" event={"ID":"bc9e4c3b-9885-4706-817b-4da954d01dfb","Type":"ContainerDied","Data":"7db87f50411d0136e2d95c008c663cafb7f9273056bc5b8c774c921544a27c6a"}
Oct 30 08:00:40.185425 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:40.185215    6224 scope.go:122] "RemoveContainer" containerID="96f4170e881ace5f96838eee8522340a6e7f8ecd851c9cea3a201e47e5fe0742"
Oct 30 08:00:40.195733 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:40.195713    6224 scope.go:122] "RemoveContainer" containerID="96f4170e881ace5f96838eee8522340a6e7f8ecd851c9cea3a201e47e5fe0742"
Oct 30 08:00:40.196013 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:00:40.195995    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96f4170e881ace5f96838eee8522340a6e7f8ecd851c9cea3a201e47e5fe0742\": container with ID starting with 96f4170e881ace5f96838eee8522340a6e7f8ecd851c9cea3a201e47e5fe0742 not found: ID does not exist" containerID="96f4170e881ace5f96838eee8522340a6e7f8ecd851c9cea3a201e47e5fe0742"
Oct 30 08:00:40.196083 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:40.196020    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96f4170e881ace5f96838eee8522340a6e7f8ecd851c9cea3a201e47e5fe0742"} err="failed to get container status \"96f4170e881ace5f96838eee8522340a6e7f8ecd851c9cea3a201e47e5fe0742\": rpc error: code = NotFound desc = could not find container \"96f4170e881ace5f96838eee8522340a6e7f8ecd851c9cea3a201e47e5fe0742\": container with ID starting with 96f4170e881ace5f96838eee8522340a6e7f8ecd851c9cea3a201e47e5fe0742 not found: ID does not exist"
Oct 30 08:00:40.244833 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:40.244789    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-1f34aa-5c4b4fdd4f-jwkrk"]
Oct 30 08:00:40.259181 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:40.259155    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-1f34aa-5c4b4fdd4f-jwkrk"]
Oct 30 08:00:41.503848 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:41.503802    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc9e4c3b-9885-4706-817b-4da954d01dfb" path="/var/lib/kubelet/pods/bc9e4c3b-9885-4706-817b-4da954d01dfb/volumes"
Oct 30 08:00:51.206575 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:51.206539    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-083c13ed-64db67c4c4-rbvkp"]
Oct 30 08:00:51.241214 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:51.241182    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-083c13ed-6df68c987f-644fr"]
Oct 30 08:00:56.207123 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:56.207073    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-083c13ed-64db67c4c4-rbvkp" podUID="2eec27f2-9ce9-4269-9c99-0ab3839cf2ee" containerName="payload-processing" containerID="cri-o://199b35fd7270e1550b9ef7751b243a48345d3a35ed4667ac72639e9758ef3292" gracePeriod=25
Oct 30 08:00:56.241633 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:56.241590    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-083c13ed-6df68c987f-644fr" podUID="271eca48-7f0b-4619-914c-bbf966c47e3d" containerName="payload-pre-processing" containerID="cri-o://6ab8903e86645043e11ae482d0bb6be570801cfc459e48268b6c129351c24931" gracePeriod=25
Oct 30 08:00:56.518907 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:56.518882    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-083c13ed-64db67c4c4-rbvkp"
Oct 30 08:00:56.553019 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:56.552998    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-083c13ed-6df68c987f-644fr"
Oct 30 08:00:56.680824 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:56.680783    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2eec27f2-9ce9-4269-9c99-0ab3839cf2ee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2eec27f2-9ce9-4269-9c99-0ab3839cf2ee-plugins-config-volume\") pod \"2eec27f2-9ce9-4269-9c99-0ab3839cf2ee\" (UID: \"2eec27f2-9ce9-4269-9c99-0ab3839cf2ee\") "
Oct 30 08:00:56.681026 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:56.680937    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/271eca48-7f0b-4619-914c-bbf966c47e3d-kube-api-access-b5bpv\" (UniqueName: \"kubernetes.io/projected/271eca48-7f0b-4619-914c-bbf966c47e3d-kube-api-access-b5bpv\") pod \"271eca48-7f0b-4619-914c-bbf966c47e3d\" (UID: \"271eca48-7f0b-4619-914c-bbf966c47e3d\") "
Oct 30 08:00:56.681026 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:56.681013    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2eec27f2-9ce9-4269-9c99-0ab3839cf2ee-kube-api-access-n8px9\" (UniqueName: \"kubernetes.io/projected/2eec27f2-9ce9-4269-9c99-0ab3839cf2ee-kube-api-access-n8px9\") pod \"2eec27f2-9ce9-4269-9c99-0ab3839cf2ee\" (UID: \"2eec27f2-9ce9-4269-9c99-0ab3839cf2ee\") "
Oct 30 08:00:56.681156 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:56.681083    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/271eca48-7f0b-4619-914c-bbf966c47e3d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/271eca48-7f0b-4619-914c-bbf966c47e3d-plugins-config-volume\") pod \"271eca48-7f0b-4619-914c-bbf966c47e3d\" (UID: \"271eca48-7f0b-4619-914c-bbf966c47e3d\") "
Oct 30 08:00:56.681215 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:56.681187    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2eec27f2-9ce9-4269-9c99-0ab3839cf2ee-plugins-config-volume" pod "2eec27f2-9ce9-4269-9c99-0ab3839cf2ee" (UID: "2eec27f2-9ce9-4269-9c99-0ab3839cf2ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:00:56.681508 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:56.681478    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/271eca48-7f0b-4619-914c-bbf966c47e3d-plugins-config-volume" pod "271eca48-7f0b-4619-914c-bbf966c47e3d" (UID: "271eca48-7f0b-4619-914c-bbf966c47e3d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:00:56.681779 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:56.681751    6224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2eec27f2-9ce9-4269-9c99-0ab3839cf2ee-plugins-config-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:00:56.681916 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:56.681780    6224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/271eca48-7f0b-4619-914c-bbf966c47e3d-plugins-config-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:00:56.683291 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:56.683266    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/271eca48-7f0b-4619-914c-bbf966c47e3d-kube-api-access-b5bpv" pod "271eca48-7f0b-4619-914c-bbf966c47e3d" (UID: "271eca48-7f0b-4619-914c-bbf966c47e3d"). InnerVolumeSpecName "kube-api-access-b5bpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:00:56.683386 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:56.683295    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eec27f2-9ce9-4269-9c99-0ab3839cf2ee-kube-api-access-n8px9" pod "2eec27f2-9ce9-4269-9c99-0ab3839cf2ee" (UID: "2eec27f2-9ce9-4269-9c99-0ab3839cf2ee"). InnerVolumeSpecName "kube-api-access-n8px9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:00:56.783079 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:56.783055    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5bpv\" (UniqueName: \"kubernetes.io/projected/271eca48-7f0b-4619-914c-bbf966c47e3d-kube-api-access-b5bpv\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:00:56.783198 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:56.783080    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8px9\" (UniqueName: \"kubernetes.io/projected/2eec27f2-9ce9-4269-9c99-0ab3839cf2ee-kube-api-access-n8px9\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:00:57.303723 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:57.303688    6224 generic.go:356] "Generic (PLEG): container finished" podID="2eec27f2-9ce9-4269-9c99-0ab3839cf2ee" containerID="199b35fd7270e1550b9ef7751b243a48345d3a35ed4667ac72639e9758ef3292" exitCode=0
Oct 30 08:00:57.304199 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:57.303738    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-083c13ed-64db67c4c4-rbvkp"
Oct 30 08:00:57.304199 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:57.303773    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-083c13ed-64db67c4c4-rbvkp" event={"ID":"2eec27f2-9ce9-4269-9c99-0ab3839cf2ee","Type":"ContainerDied","Data":"199b35fd7270e1550b9ef7751b243a48345d3a35ed4667ac72639e9758ef3292"}
Oct 30 08:00:57.304199 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:57.303857    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-083c13ed-64db67c4c4-rbvkp" event={"ID":"2eec27f2-9ce9-4269-9c99-0ab3839cf2ee","Type":"ContainerDied","Data":"1337029e4510b4c78f8853e59fa1bee7ad97ad00af24e87db74915d4cd6132ab"}
Oct 30 08:00:57.304199 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:57.303888    6224 scope.go:122] "RemoveContainer" containerID="199b35fd7270e1550b9ef7751b243a48345d3a35ed4667ac72639e9758ef3292"
Oct 30 08:00:57.305689 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:57.305665    6224 generic.go:356] "Generic (PLEG): container finished" podID="271eca48-7f0b-4619-914c-bbf966c47e3d" containerID="6ab8903e86645043e11ae482d0bb6be570801cfc459e48268b6c129351c24931" exitCode=0
Oct 30 08:00:57.305784 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:57.305711    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-083c13ed-6df68c987f-644fr"
Oct 30 08:00:57.305784 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:57.305721    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-083c13ed-6df68c987f-644fr" event={"ID":"271eca48-7f0b-4619-914c-bbf966c47e3d","Type":"ContainerDied","Data":"6ab8903e86645043e11ae482d0bb6be570801cfc459e48268b6c129351c24931"}
Oct 30 08:00:57.305784 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:57.305763    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-083c13ed-6df68c987f-644fr" event={"ID":"271eca48-7f0b-4619-914c-bbf966c47e3d","Type":"ContainerDied","Data":"922d3fd99a976fec2af991f9a932c73c15ca048692ae527b503b36daf51dab4d"}
Oct 30 08:00:57.316035 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:57.316014    6224 scope.go:122] "RemoveContainer" containerID="199b35fd7270e1550b9ef7751b243a48345d3a35ed4667ac72639e9758ef3292"
Oct 30 08:00:57.316330 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:00:57.316306    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"199b35fd7270e1550b9ef7751b243a48345d3a35ed4667ac72639e9758ef3292\": container with ID starting with 199b35fd7270e1550b9ef7751b243a48345d3a35ed4667ac72639e9758ef3292 not found: ID does not exist" containerID="199b35fd7270e1550b9ef7751b243a48345d3a35ed4667ac72639e9758ef3292"
Oct 30 08:00:57.316408 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:57.316339    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"199b35fd7270e1550b9ef7751b243a48345d3a35ed4667ac72639e9758ef3292"} err="failed to get container status \"199b35fd7270e1550b9ef7751b243a48345d3a35ed4667ac72639e9758ef3292\": rpc error: code = NotFound desc = could not find container \"199b35fd7270e1550b9ef7751b243a48345d3a35ed4667ac72639e9758ef3292\": container with ID starting with 199b35fd7270e1550b9ef7751b243a48345d3a35ed4667ac72639e9758ef3292 not found: ID does not exist"
Oct 30 08:00:57.316408 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:57.316362    6224 scope.go:122] "RemoveContainer" containerID="6ab8903e86645043e11ae482d0bb6be570801cfc459e48268b6c129351c24931"
Oct 30 08:00:57.325827 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:57.325793    6224 scope.go:122] "RemoveContainer" containerID="6ab8903e86645043e11ae482d0bb6be570801cfc459e48268b6c129351c24931"
Oct 30 08:00:57.326096 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:00:57.326077    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ab8903e86645043e11ae482d0bb6be570801cfc459e48268b6c129351c24931\": container with ID starting with 6ab8903e86645043e11ae482d0bb6be570801cfc459e48268b6c129351c24931 not found: ID does not exist" containerID="6ab8903e86645043e11ae482d0bb6be570801cfc459e48268b6c129351c24931"
Oct 30 08:00:57.326168 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:57.326104    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ab8903e86645043e11ae482d0bb6be570801cfc459e48268b6c129351c24931"} err="failed to get container status \"6ab8903e86645043e11ae482d0bb6be570801cfc459e48268b6c129351c24931\": rpc error: code = NotFound desc = could not find container \"6ab8903e86645043e11ae482d0bb6be570801cfc459e48268b6c129351c24931\": container with ID starting with 6ab8903e86645043e11ae482d0bb6be570801cfc459e48268b6c129351c24931 not found: ID does not exist"
Oct 30 08:00:57.363258 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:57.363228    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-083c13ed-6df68c987f-644fr"]
Oct 30 08:00:57.369750 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:57.369729    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-083c13ed-6df68c987f-644fr"]
Oct 30 08:00:57.418450 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:57.418427    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-083c13ed-64db67c4c4-rbvkp"]
Oct 30 08:00:57.422017 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:57.421991    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-083c13ed-64db67c4c4-rbvkp"]
Oct 30 08:00:57.503782 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:57.503711    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="271eca48-7f0b-4619-914c-bbf966c47e3d" path="/var/lib/kubelet/pods/271eca48-7f0b-4619-914c-bbf966c47e3d/volumes"
Oct 30 08:00:57.504210 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:57.504194    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2eec27f2-9ce9-4269-9c99-0ab3839cf2ee" path="/var/lib/kubelet/pods/2eec27f2-9ce9-4269-9c99-0ab3839cf2ee/volumes"
Oct 30 08:00:58.254847 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:00:58.254799    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-8e7948-78667cb64b-2lp7r"]
Oct 30 08:01:01.861019 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:01:01.860972    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 08:01:02.336502 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:02.336467    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 08:01:03.259317 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:03.259135    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-8e7948-78667cb64b-2lp7r" podUID="64034fab-2b81-4f54-9d0c-5f2360ab6274" containerName="payload-processing" containerID="cri-o://64a199f4d13078738b4faed451e11cb7b2296b8b5b2a3d8aaa2a63134ccefc38" gracePeriod=25
Oct 30 08:01:04.314393 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:04.314353    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-8e7948-78667cb64b-2lp7r" podUID="64034fab-2b81-4f54-9d0c-5f2360ab6274" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:01:14.314849 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:14.314787    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-8e7948-78667cb64b-2lp7r" podUID="64034fab-2b81-4f54-9d0c-5f2360ab6274" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:01:18.664381 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:18.664344    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-f1f700-6db48b9ff4-chndk"]
Oct 30 08:01:18.699430 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:18.699388    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-f1f700-cccdc9f5-wn4nt"]
Oct 30 08:01:23.511025 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:23.510980    6224 generic.go:356] "Generic (PLEG): container finished" podID="64034fab-2b81-4f54-9d0c-5f2360ab6274" containerID="64a199f4d13078738b4faed451e11cb7b2296b8b5b2a3d8aaa2a63134ccefc38" exitCode=0
Oct 30 08:01:23.511025 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:23.511012    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-8e7948-78667cb64b-2lp7r" event={"ID":"64034fab-2b81-4f54-9d0c-5f2360ab6274","Type":"ContainerDied","Data":"64a199f4d13078738b4faed451e11cb7b2296b8b5b2a3d8aaa2a63134ccefc38"}
Oct 30 08:01:23.552925 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:23.552899    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-8e7948-78667cb64b-2lp7r"
Oct 30 08:01:23.622566 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:23.622520    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64034fab-2b81-4f54-9d0c-5f2360ab6274-kube-api-access-4zckh\" (UniqueName: \"kubernetes.io/projected/64034fab-2b81-4f54-9d0c-5f2360ab6274-kube-api-access-4zckh\") pod \"64034fab-2b81-4f54-9d0c-5f2360ab6274\" (UID: \"64034fab-2b81-4f54-9d0c-5f2360ab6274\") "
Oct 30 08:01:23.622766 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:23.622710    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64034fab-2b81-4f54-9d0c-5f2360ab6274-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64034fab-2b81-4f54-9d0c-5f2360ab6274-plugins-config-volume\") pod \"64034fab-2b81-4f54-9d0c-5f2360ab6274\" (UID: \"64034fab-2b81-4f54-9d0c-5f2360ab6274\") "
Oct 30 08:01:23.623357 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:23.623327    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64034fab-2b81-4f54-9d0c-5f2360ab6274-plugins-config-volume" pod "64034fab-2b81-4f54-9d0c-5f2360ab6274" (UID: "64034fab-2b81-4f54-9d0c-5f2360ab6274"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:01:23.625786 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:23.625760    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64034fab-2b81-4f54-9d0c-5f2360ab6274-kube-api-access-4zckh" pod "64034fab-2b81-4f54-9d0c-5f2360ab6274" (UID: "64034fab-2b81-4f54-9d0c-5f2360ab6274"). InnerVolumeSpecName "kube-api-access-4zckh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:01:23.664676 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:23.664632    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-f1f700-6db48b9ff4-chndk" podUID="1825dd2c-f162-4d90-8ebd-a6923e990abd" containerName="payload-processing" containerID="cri-o://d3ebd0183d13705987a8aea6c939610fd9a359c1a85393f2dd0e50b5044a5f2e" gracePeriod=25
Oct 30 08:01:23.700519 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:23.700461    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f1f700-cccdc9f5-wn4nt" podUID="0d55b266-9ae5-4b44-85e3-ea14553e4e27" containerName="payload-pre-processing" containerID="cri-o://40c1243ac13cc5bd03a5727bed1733e95368ba674d9c8b98b1c4041449f77b3f" gracePeriod=25
Oct 30 08:01:23.724766 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:23.724737    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zckh\" (UniqueName: \"kubernetes.io/projected/64034fab-2b81-4f54-9d0c-5f2360ab6274-kube-api-access-4zckh\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:01:23.724766 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:23.724763    6224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64034fab-2b81-4f54-9d0c-5f2360ab6274-plugins-config-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:01:24.245129 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:24.245083    6224 scope.go:122] "RemoveContainer" containerID="64a199f4d13078738b4faed451e11cb7b2296b8b5b2a3d8aaa2a63134ccefc38"
Oct 30 08:01:24.517361 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:24.517272    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-8e7948-78667cb64b-2lp7r" event={"ID":"64034fab-2b81-4f54-9d0c-5f2360ab6274","Type":"ContainerDied","Data":"e59d84c1e4e47a834f84d55a06c1a44c9bda13dfc9386ad54a3a6aa3548b3933"}
Oct 30 08:01:24.517361 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:24.517358    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-8e7948-78667cb64b-2lp7r"
Oct 30 08:01:24.609325 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:24.609291    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-8e7948-78667cb64b-2lp7r"]
Oct 30 08:01:24.618264 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:24.618232    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-8e7948-78667cb64b-2lp7r"]
Oct 30 08:01:25.504150 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:25.504112    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64034fab-2b81-4f54-9d0c-5f2360ab6274" path="/var/lib/kubelet/pods/64034fab-2b81-4f54-9d0c-5f2360ab6274/volumes"
Oct 30 08:01:31.858605 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:01:31.858569    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 08:01:31.924682 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:31.924645    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-932c00/rate-test-model-932c00-kserve-59bbdb5c89-vgxtp"]
Oct 30 08:01:33.084487 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:33.084447    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f1f700-cccdc9f5-wn4nt" podUID="0d55b266-9ae5-4b44-85e3-ea14553e4e27" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:01:33.084487 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:33.084474    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-f1f700-6db48b9ff4-chndk" podUID="1825dd2c-f162-4d90-8ebd-a6923e990abd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:01:36.758717 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:36.758681    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj"]
Oct 30 08:01:36.759171 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:36.758929    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj" podUID="ca0e2ee4-723d-4fc1-b13e-72b13c83d280" containerName="istio-proxy" containerID="cri-o://34bc958339bd74659bdfd09f020156ffdb2ed2a83c22f7d7651934a04be40cd2" gracePeriod=30
Oct 30 08:01:42.042061 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:42.042027    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj"
Oct 30 08:01:42.150531 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:42.150499    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca0e2ee4-723d-4fc1-b13e-72b13c83d280-kube-api-access-vhmmt\" (UniqueName: \"kubernetes.io/projected/ca0e2ee4-723d-4fc1-b13e-72b13c83d280-kube-api-access-vhmmt\") pod \"ca0e2ee4-723d-4fc1-b13e-72b13c83d280\" (UID: \"ca0e2ee4-723d-4fc1-b13e-72b13c83d280\") "
Oct 30 08:01:42.150854 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:42.150564    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca0e2ee4-723d-4fc1-b13e-72b13c83d280-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca0e2ee4-723d-4fc1-b13e-72b13c83d280-istio-envoy\") pod \"ca0e2ee4-723d-4fc1-b13e-72b13c83d280\" (UID: \"ca0e2ee4-723d-4fc1-b13e-72b13c83d280\") "
Oct 30 08:01:42.150854 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:42.150591    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca0e2ee4-723d-4fc1-b13e-72b13c83d280-istio-token\" (UniqueName: \"kubernetes.io/projected/ca0e2ee4-723d-4fc1-b13e-72b13c83d280-istio-token\") pod \"ca0e2ee4-723d-4fc1-b13e-72b13c83d280\" (UID: \"ca0e2ee4-723d-4fc1-b13e-72b13c83d280\") "
Oct 30 08:01:42.150854 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:42.150638    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca0e2ee4-723d-4fc1-b13e-72b13c83d280-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca0e2ee4-723d-4fc1-b13e-72b13c83d280-workload-certs\") pod \"ca0e2ee4-723d-4fc1-b13e-72b13c83d280\" (UID: \"ca0e2ee4-723d-4fc1-b13e-72b13c83d280\") "
Oct 30 08:01:42.150854 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:42.150665    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca0e2ee4-723d-4fc1-b13e-72b13c83d280-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca0e2ee4-723d-4fc1-b13e-72b13c83d280-workload-socket\") pod \"ca0e2ee4-723d-4fc1-b13e-72b13c83d280\" (UID: \"ca0e2ee4-723d-4fc1-b13e-72b13c83d280\") "
Oct 30 08:01:42.150854 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:42.150774    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca0e2ee4-723d-4fc1-b13e-72b13c83d280-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca0e2ee4-723d-4fc1-b13e-72b13c83d280-credential-socket\") pod \"ca0e2ee4-723d-4fc1-b13e-72b13c83d280\" (UID: \"ca0e2ee4-723d-4fc1-b13e-72b13c83d280\") "
Oct 30 08:01:42.150854 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:42.150802    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca0e2ee4-723d-4fc1-b13e-72b13c83d280-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca0e2ee4-723d-4fc1-b13e-72b13c83d280-istiod-ca-cert\") pod \"ca0e2ee4-723d-4fc1-b13e-72b13c83d280\" (UID: \"ca0e2ee4-723d-4fc1-b13e-72b13c83d280\") "
Oct 30 08:01:42.151204 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:42.150989    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ca0e2ee4-723d-4fc1-b13e-72b13c83d280-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca0e2ee4-723d-4fc1-b13e-72b13c83d280-istio-podinfo\") pod \"ca0e2ee4-723d-4fc1-b13e-72b13c83d280\" (UID: \"ca0e2ee4-723d-4fc1-b13e-72b13c83d280\") "
Oct 30 08:01:42.151204 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:42.151030    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca0e2ee4-723d-4fc1-b13e-72b13c83d280-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca0e2ee4-723d-4fc1-b13e-72b13c83d280-istio-data\") pod \"ca0e2ee4-723d-4fc1-b13e-72b13c83d280\" (UID: \"ca0e2ee4-723d-4fc1-b13e-72b13c83d280\") "
Oct 30 08:01:42.151204 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:42.151035    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca0e2ee4-723d-4fc1-b13e-72b13c83d280-workload-certs" pod "ca0e2ee4-723d-4fc1-b13e-72b13c83d280" (UID: "ca0e2ee4-723d-4fc1-b13e-72b13c83d280"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:01:42.151204 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:42.151040    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca0e2ee4-723d-4fc1-b13e-72b13c83d280-workload-socket" pod "ca0e2ee4-723d-4fc1-b13e-72b13c83d280" (UID: "ca0e2ee4-723d-4fc1-b13e-72b13c83d280"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:01:42.151204 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:42.151075    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca0e2ee4-723d-4fc1-b13e-72b13c83d280-credential-socket" pod "ca0e2ee4-723d-4fc1-b13e-72b13c83d280" (UID: "ca0e2ee4-723d-4fc1-b13e-72b13c83d280"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:01:42.151447 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:42.151334    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca0e2ee4-723d-4fc1-b13e-72b13c83d280-istio-data" pod "ca0e2ee4-723d-4fc1-b13e-72b13c83d280" (UID: "ca0e2ee4-723d-4fc1-b13e-72b13c83d280"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:01:42.151447 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:42.151359    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca0e2ee4-723d-4fc1-b13e-72b13c83d280-istiod-ca-cert" pod "ca0e2ee4-723d-4fc1-b13e-72b13c83d280" (UID: "ca0e2ee4-723d-4fc1-b13e-72b13c83d280"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:01:42.151574 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:42.151557    6224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca0e2ee4-723d-4fc1-b13e-72b13c83d280-workload-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:01:42.151617 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:42.151577    6224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca0e2ee4-723d-4fc1-b13e-72b13c83d280-workload-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:01:42.151617 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:42.151587    6224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca0e2ee4-723d-4fc1-b13e-72b13c83d280-credential-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:01:42.151617 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:42.151596    6224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca0e2ee4-723d-4fc1-b13e-72b13c83d280-istiod-ca-cert\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:01:42.151617 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:42.151605    6224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca0e2ee4-723d-4fc1-b13e-72b13c83d280-istio-data\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:01:42.153081 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:42.153058    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ca0e2ee4-723d-4fc1-b13e-72b13c83d280-istio-podinfo" pod "ca0e2ee4-723d-4fc1-b13e-72b13c83d280" (UID: "ca0e2ee4-723d-4fc1-b13e-72b13c83d280"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 08:01:42.153344 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:42.153321    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca0e2ee4-723d-4fc1-b13e-72b13c83d280-kube-api-access-vhmmt" pod "ca0e2ee4-723d-4fc1-b13e-72b13c83d280" (UID: "ca0e2ee4-723d-4fc1-b13e-72b13c83d280"). InnerVolumeSpecName "kube-api-access-vhmmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:01:42.153398 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:42.153364    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca0e2ee4-723d-4fc1-b13e-72b13c83d280-istio-envoy" pod "ca0e2ee4-723d-4fc1-b13e-72b13c83d280" (UID: "ca0e2ee4-723d-4fc1-b13e-72b13c83d280"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:01:42.153398 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:42.153387    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca0e2ee4-723d-4fc1-b13e-72b13c83d280-istio-token" pod "ca0e2ee4-723d-4fc1-b13e-72b13c83d280" (UID: "ca0e2ee4-723d-4fc1-b13e-72b13c83d280"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:01:42.253359 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:42.253322    6224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca0e2ee4-723d-4fc1-b13e-72b13c83d280-istio-podinfo\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:01:42.253359 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:42.253358    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhmmt\" (UniqueName: \"kubernetes.io/projected/ca0e2ee4-723d-4fc1-b13e-72b13c83d280-kube-api-access-vhmmt\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:01:42.253582 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:42.253371    6224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca0e2ee4-723d-4fc1-b13e-72b13c83d280-istio-envoy\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:01:42.253582 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:42.253385    6224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca0e2ee4-723d-4fc1-b13e-72b13c83d280-istio-token\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:01:42.643795 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:42.643750    6224 generic.go:356] "Generic (PLEG): container finished" podID="ca0e2ee4-723d-4fc1-b13e-72b13c83d280" containerID="34bc958339bd74659bdfd09f020156ffdb2ed2a83c22f7d7651934a04be40cd2" exitCode=0
Oct 30 08:01:42.644007 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:42.643785    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj" event={"ID":"ca0e2ee4-723d-4fc1-b13e-72b13c83d280","Type":"ContainerDied","Data":"34bc958339bd74659bdfd09f020156ffdb2ed2a83c22f7d7651934a04be40cd2"}
Oct 30 08:01:42.644007 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:42.643873    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj" event={"ID":"ca0e2ee4-723d-4fc1-b13e-72b13c83d280","Type":"ContainerDied","Data":"f3c6151e9bb7712464317e38333daf2aec76a0fba50949ae5c74532c4287b6d5"}
Oct 30 08:01:42.644007 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:42.643900    6224 scope.go:122] "RemoveContainer" containerID="34bc958339bd74659bdfd09f020156ffdb2ed2a83c22f7d7651934a04be40cd2"
Oct 30 08:01:42.644007 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:42.643817    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj"
Oct 30 08:01:42.659162 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:42.659137    6224 scope.go:122] "RemoveContainer" containerID="5b53486850f289a4bf760dee28e46d37112e535569df409558a611e76a20b4c9"
Oct 30 08:01:42.669088 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:42.669072    6224 scope.go:122] "RemoveContainer" containerID="34bc958339bd74659bdfd09f020156ffdb2ed2a83c22f7d7651934a04be40cd2"
Oct 30 08:01:42.669381 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:01:42.669357    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34bc958339bd74659bdfd09f020156ffdb2ed2a83c22f7d7651934a04be40cd2\": container with ID starting with 34bc958339bd74659bdfd09f020156ffdb2ed2a83c22f7d7651934a04be40cd2 not found: ID does not exist" containerID="34bc958339bd74659bdfd09f020156ffdb2ed2a83c22f7d7651934a04be40cd2"
Oct 30 08:01:42.669452 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:42.669395    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34bc958339bd74659bdfd09f020156ffdb2ed2a83c22f7d7651934a04be40cd2"} err="failed to get container status \"34bc958339bd74659bdfd09f020156ffdb2ed2a83c22f7d7651934a04be40cd2\": rpc error: code = NotFound desc = could not find container \"34bc958339bd74659bdfd09f020156ffdb2ed2a83c22f7d7651934a04be40cd2\": container with ID starting with 34bc958339bd74659bdfd09f020156ffdb2ed2a83c22f7d7651934a04be40cd2 not found: ID does not exist"
Oct 30 08:01:42.669452 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:42.669418    6224 scope.go:122] "RemoveContainer" containerID="5b53486850f289a4bf760dee28e46d37112e535569df409558a611e76a20b4c9"
Oct 30 08:01:42.669707 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:01:42.669683    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b53486850f289a4bf760dee28e46d37112e535569df409558a611e76a20b4c9\": container with ID starting with 5b53486850f289a4bf760dee28e46d37112e535569df409558a611e76a20b4c9 not found: ID does not exist" containerID="5b53486850f289a4bf760dee28e46d37112e535569df409558a611e76a20b4c9"
Oct 30 08:01:42.669769 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:42.669717    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b53486850f289a4bf760dee28e46d37112e535569df409558a611e76a20b4c9"} err="failed to get container status \"5b53486850f289a4bf760dee28e46d37112e535569df409558a611e76a20b4c9\": rpc error: code = NotFound desc = could not find container \"5b53486850f289a4bf760dee28e46d37112e535569df409558a611e76a20b4c9\": container with ID starting with 5b53486850f289a4bf760dee28e46d37112e535569df409558a611e76a20b4c9 not found: ID does not exist"
Oct 30 08:01:42.703846 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:42.703800    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj"]
Oct 30 08:01:42.718404 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:42.718382    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-083c13ed-openshift-default-6586cd86df-qphzj"]
Oct 30 08:01:43.084503 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:43.084460    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f1f700-cccdc9f5-wn4nt" podUID="0d55b266-9ae5-4b44-85e3-ea14553e4e27" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:01:43.084503 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:43.084468    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-f1f700-6db48b9ff4-chndk" podUID="1825dd2c-f162-4d90-8ebd-a6923e990abd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:01:43.504635 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:43.504552    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca0e2ee4-723d-4fc1-b13e-72b13c83d280" path="/var/lib/kubelet/pods/ca0e2ee4-723d-4fc1-b13e-72b13c83d280/volumes"
Oct 30 08:01:44.015396 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:44.015342    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-f1f700-6db48b9ff4-chndk"
Oct 30 08:01:44.018784 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:44.018764    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f1f700-cccdc9f5-wn4nt"
Oct 30 08:01:44.176852 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:44.176815    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1825dd2c-f162-4d90-8ebd-a6923e990abd-kube-api-access-gpvs7\" (UniqueName: \"kubernetes.io/projected/1825dd2c-f162-4d90-8ebd-a6923e990abd-kube-api-access-gpvs7\") pod \"1825dd2c-f162-4d90-8ebd-a6923e990abd\" (UID: \"1825dd2c-f162-4d90-8ebd-a6923e990abd\") "
Oct 30 08:01:44.177289 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:44.176916    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d55b266-9ae5-4b44-85e3-ea14553e4e27-kube-api-access-4mv6k\" (UniqueName: \"kubernetes.io/projected/0d55b266-9ae5-4b44-85e3-ea14553e4e27-kube-api-access-4mv6k\") pod \"0d55b266-9ae5-4b44-85e3-ea14553e4e27\" (UID: \"0d55b266-9ae5-4b44-85e3-ea14553e4e27\") "
Oct 30 08:01:44.177289 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:44.177027    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1825dd2c-f162-4d90-8ebd-a6923e990abd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1825dd2c-f162-4d90-8ebd-a6923e990abd-plugins-config-volume\") pod \"1825dd2c-f162-4d90-8ebd-a6923e990abd\" (UID: \"1825dd2c-f162-4d90-8ebd-a6923e990abd\") "
Oct 30 08:01:44.177289 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:44.177069    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d55b266-9ae5-4b44-85e3-ea14553e4e27-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d55b266-9ae5-4b44-85e3-ea14553e4e27-plugins-config-volume\") pod \"0d55b266-9ae5-4b44-85e3-ea14553e4e27\" (UID: \"0d55b266-9ae5-4b44-85e3-ea14553e4e27\") "
Oct 30 08:01:44.177477 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:44.177422    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1825dd2c-f162-4d90-8ebd-a6923e990abd-plugins-config-volume" pod "1825dd2c-f162-4d90-8ebd-a6923e990abd" (UID: "1825dd2c-f162-4d90-8ebd-a6923e990abd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:01:44.177477 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:44.177453    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d55b266-9ae5-4b44-85e3-ea14553e4e27-plugins-config-volume" pod "0d55b266-9ae5-4b44-85e3-ea14553e4e27" (UID: "0d55b266-9ae5-4b44-85e3-ea14553e4e27"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:01:44.178970 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:44.178948    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1825dd2c-f162-4d90-8ebd-a6923e990abd-kube-api-access-gpvs7" pod "1825dd2c-f162-4d90-8ebd-a6923e990abd" (UID: "1825dd2c-f162-4d90-8ebd-a6923e990abd"). InnerVolumeSpecName "kube-api-access-gpvs7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:01:44.179078 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:44.179060    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d55b266-9ae5-4b44-85e3-ea14553e4e27-kube-api-access-4mv6k" pod "0d55b266-9ae5-4b44-85e3-ea14553e4e27" (UID: "0d55b266-9ae5-4b44-85e3-ea14553e4e27"). InnerVolumeSpecName "kube-api-access-4mv6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:01:44.278406 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:44.278329    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mv6k\" (UniqueName: \"kubernetes.io/projected/0d55b266-9ae5-4b44-85e3-ea14553e4e27-kube-api-access-4mv6k\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:01:44.278406 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:44.278360    6224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1825dd2c-f162-4d90-8ebd-a6923e990abd-plugins-config-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:01:44.278406 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:44.278382    6224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d55b266-9ae5-4b44-85e3-ea14553e4e27-plugins-config-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:01:44.278406 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:44.278392    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpvs7\" (UniqueName: \"kubernetes.io/projected/1825dd2c-f162-4d90-8ebd-a6923e990abd-kube-api-access-gpvs7\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:01:44.660286 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:44.660247    6224 generic.go:356] "Generic (PLEG): container finished" podID="1825dd2c-f162-4d90-8ebd-a6923e990abd" containerID="d3ebd0183d13705987a8aea6c939610fd9a359c1a85393f2dd0e50b5044a5f2e" exitCode=0
Oct 30 08:01:44.660511 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:44.660306    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-f1f700-6db48b9ff4-chndk"
Oct 30 08:01:44.660511 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:44.660322    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-f1f700-6db48b9ff4-chndk" event={"ID":"1825dd2c-f162-4d90-8ebd-a6923e990abd","Type":"ContainerDied","Data":"d3ebd0183d13705987a8aea6c939610fd9a359c1a85393f2dd0e50b5044a5f2e"}
Oct 30 08:01:44.660511 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:44.660379    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-f1f700-6db48b9ff4-chndk" event={"ID":"1825dd2c-f162-4d90-8ebd-a6923e990abd","Type":"ContainerDied","Data":"bb0d12de5e286e32e80a5fc564567f997c3f0eed00673a2ce2e54145fbcaf841"}
Oct 30 08:01:44.660511 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:44.660403    6224 scope.go:122] "RemoveContainer" containerID="d3ebd0183d13705987a8aea6c939610fd9a359c1a85393f2dd0e50b5044a5f2e"
Oct 30 08:01:44.662143 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:44.662116    6224 generic.go:356] "Generic (PLEG): container finished" podID="0d55b266-9ae5-4b44-85e3-ea14553e4e27" containerID="40c1243ac13cc5bd03a5727bed1733e95368ba674d9c8b98b1c4041449f77b3f" exitCode=0
Oct 30 08:01:44.662268 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:44.662179    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f1f700-cccdc9f5-wn4nt"
Oct 30 08:01:44.662268 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:44.662227    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f1f700-cccdc9f5-wn4nt" event={"ID":"0d55b266-9ae5-4b44-85e3-ea14553e4e27","Type":"ContainerDied","Data":"40c1243ac13cc5bd03a5727bed1733e95368ba674d9c8b98b1c4041449f77b3f"}
Oct 30 08:01:44.662357 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:44.662269    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f1f700-cccdc9f5-wn4nt" event={"ID":"0d55b266-9ae5-4b44-85e3-ea14553e4e27","Type":"ContainerDied","Data":"c6efc81f1f2bbf26200a3d715a5f7e509eea535f87447b035f0c31f9e192a7f1"}
Oct 30 08:01:44.673524 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:44.673505    6224 scope.go:122] "RemoveContainer" containerID="d3ebd0183d13705987a8aea6c939610fd9a359c1a85393f2dd0e50b5044a5f2e"
Oct 30 08:01:44.673817 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:01:44.673785    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3ebd0183d13705987a8aea6c939610fd9a359c1a85393f2dd0e50b5044a5f2e\": container with ID starting with d3ebd0183d13705987a8aea6c939610fd9a359c1a85393f2dd0e50b5044a5f2e not found: ID does not exist" containerID="d3ebd0183d13705987a8aea6c939610fd9a359c1a85393f2dd0e50b5044a5f2e"
Oct 30 08:01:44.673904 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:44.673829    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3ebd0183d13705987a8aea6c939610fd9a359c1a85393f2dd0e50b5044a5f2e"} err="failed to get container status \"d3ebd0183d13705987a8aea6c939610fd9a359c1a85393f2dd0e50b5044a5f2e\": rpc error: code = NotFound desc = could not find container \"d3ebd0183d13705987a8aea6c939610fd9a359c1a85393f2dd0e50b5044a5f2e\": container with ID starting with d3ebd0183d13705987a8aea6c939610fd9a359c1a85393f2dd0e50b5044a5f2e not found: ID does not exist"
Oct 30 08:01:44.673904 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:44.673854    6224 scope.go:122] "RemoveContainer" containerID="40c1243ac13cc5bd03a5727bed1733e95368ba674d9c8b98b1c4041449f77b3f"
Oct 30 08:01:44.691006 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:44.690975    6224 scope.go:122] "RemoveContainer" containerID="40c1243ac13cc5bd03a5727bed1733e95368ba674d9c8b98b1c4041449f77b3f"
Oct 30 08:01:44.691313 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:01:44.691293    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40c1243ac13cc5bd03a5727bed1733e95368ba674d9c8b98b1c4041449f77b3f\": container with ID starting with 40c1243ac13cc5bd03a5727bed1733e95368ba674d9c8b98b1c4041449f77b3f not found: ID does not exist" containerID="40c1243ac13cc5bd03a5727bed1733e95368ba674d9c8b98b1c4041449f77b3f"
Oct 30 08:01:44.691382 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:44.691322    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40c1243ac13cc5bd03a5727bed1733e95368ba674d9c8b98b1c4041449f77b3f"} err="failed to get container status \"40c1243ac13cc5bd03a5727bed1733e95368ba674d9c8b98b1c4041449f77b3f\": rpc error: code = NotFound desc = could not find container \"40c1243ac13cc5bd03a5727bed1733e95368ba674d9c8b98b1c4041449f77b3f\": container with ID starting with 40c1243ac13cc5bd03a5727bed1733e95368ba674d9c8b98b1c4041449f77b3f not found: ID does not exist"
Oct 30 08:01:44.734471 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:44.734445    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-f1f700-6db48b9ff4-chndk"]
Oct 30 08:01:44.741980 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:44.741954    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-f1f700-6db48b9ff4-chndk"]
Oct 30 08:01:44.800500 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:44.800465    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-f1f700-cccdc9f5-wn4nt"]
Oct 30 08:01:44.808521 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:44.808494    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-f1f700-cccdc9f5-wn4nt"]
Oct 30 08:01:45.504661 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:45.504631    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d55b266-9ae5-4b44-85e3-ea14553e4e27" path="/var/lib/kubelet/pods/0d55b266-9ae5-4b44-85e3-ea14553e4e27/volumes"
Oct 30 08:01:45.505088 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:45.505072    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1825dd2c-f162-4d90-8ebd-a6923e990abd" path="/var/lib/kubelet/pods/1825dd2c-f162-4d90-8ebd-a6923e990abd/volumes"
Oct 30 08:01:46.260537 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:46.260502    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-137122/rate-test-model-137122-kserve-6bb88d4545-29s55"]
Oct 30 08:01:46.948501 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:46.948456    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-932c00/rate-test-model-932c00-kserve-59bbdb5c89-vgxtp" podUID="6ad02f3d-27a3-4934-b4eb-60b39ab74aa8" containerName="main" containerID="cri-o://0a7c406a5b15f9173bb5674e56ea47849b8e054828945d956120d7f9c7497d6c" gracePeriod=45
Oct 30 08:01:47.077353 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:47.077321    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-932c00/rate-test-model-932c00-kserve-59bbdb5c89-vgxtp" podUID="6ad02f3d-27a3-4934-b4eb-60b39ab74aa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.217:8000/ready\": dial tcp 10.134.0.217:8000: connect: connection refused"
Oct 30 08:01:47.223434 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:47.223348    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-932c00/rate-test-model-932c00-kserve-59bbdb5c89-vgxtp"
Oct 30 08:01:47.320594 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:47.320553    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ad02f3d-27a3-4934-b4eb-60b39ab74aa8-home\" (UniqueName: \"kubernetes.io/empty-dir/6ad02f3d-27a3-4934-b4eb-60b39ab74aa8-home\") pod \"6ad02f3d-27a3-4934-b4eb-60b39ab74aa8\" (UID: \"6ad02f3d-27a3-4934-b4eb-60b39ab74aa8\") "
Oct 30 08:01:47.320795 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:47.320623    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ad02f3d-27a3-4934-b4eb-60b39ab74aa8-kube-api-access-8btw4\" (UniqueName: \"kubernetes.io/projected/6ad02f3d-27a3-4934-b4eb-60b39ab74aa8-kube-api-access-8btw4\") pod \"6ad02f3d-27a3-4934-b4eb-60b39ab74aa8\" (UID: \"6ad02f3d-27a3-4934-b4eb-60b39ab74aa8\") "
Oct 30 08:01:47.320795 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:47.320655    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ad02f3d-27a3-4934-b4eb-60b39ab74aa8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ad02f3d-27a3-4934-b4eb-60b39ab74aa8-tmp-dir\") pod \"6ad02f3d-27a3-4934-b4eb-60b39ab74aa8\" (UID: \"6ad02f3d-27a3-4934-b4eb-60b39ab74aa8\") "
Oct 30 08:01:47.320795 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:47.320713    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ad02f3d-27a3-4934-b4eb-60b39ab74aa8-tls-certs\" (UniqueName: \"kubernetes.io/secret/6ad02f3d-27a3-4934-b4eb-60b39ab74aa8-tls-certs\") pod \"6ad02f3d-27a3-4934-b4eb-60b39ab74aa8\" (UID: \"6ad02f3d-27a3-4934-b4eb-60b39ab74aa8\") "
Oct 30 08:01:47.321008 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:47.320829    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ad02f3d-27a3-4934-b4eb-60b39ab74aa8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ad02f3d-27a3-4934-b4eb-60b39ab74aa8-dshm\") pod \"6ad02f3d-27a3-4934-b4eb-60b39ab74aa8\" (UID: \"6ad02f3d-27a3-4934-b4eb-60b39ab74aa8\") "
Oct 30 08:01:47.321008 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:47.320872    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ad02f3d-27a3-4934-b4eb-60b39ab74aa8-home" pod "6ad02f3d-27a3-4934-b4eb-60b39ab74aa8" (UID: "6ad02f3d-27a3-4934-b4eb-60b39ab74aa8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:01:47.321008 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:47.320892    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ad02f3d-27a3-4934-b4eb-60b39ab74aa8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ad02f3d-27a3-4934-b4eb-60b39ab74aa8-model-cache\") pod \"6ad02f3d-27a3-4934-b4eb-60b39ab74aa8\" (UID: \"6ad02f3d-27a3-4934-b4eb-60b39ab74aa8\") "
Oct 30 08:01:47.321193 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:47.321037    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ad02f3d-27a3-4934-b4eb-60b39ab74aa8-tmp-dir" pod "6ad02f3d-27a3-4934-b4eb-60b39ab74aa8" (UID: "6ad02f3d-27a3-4934-b4eb-60b39ab74aa8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:01:47.321193 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:47.321168    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ad02f3d-27a3-4934-b4eb-60b39ab74aa8-model-cache" pod "6ad02f3d-27a3-4934-b4eb-60b39ab74aa8" (UID: "6ad02f3d-27a3-4934-b4eb-60b39ab74aa8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:01:47.321538 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:47.321515    6224 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ad02f3d-27a3-4934-b4eb-60b39ab74aa8-home\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:01:47.321538 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:47.321537    6224 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ad02f3d-27a3-4934-b4eb-60b39ab74aa8-tmp-dir\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:01:47.321723 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:47.321548    6224 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ad02f3d-27a3-4934-b4eb-60b39ab74aa8-model-cache\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:01:47.322982 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:47.322953    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ad02f3d-27a3-4934-b4eb-60b39ab74aa8-dshm" pod "6ad02f3d-27a3-4934-b4eb-60b39ab74aa8" (UID: "6ad02f3d-27a3-4934-b4eb-60b39ab74aa8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:01:47.322982 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:47.322965    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ad02f3d-27a3-4934-b4eb-60b39ab74aa8-kube-api-access-8btw4" pod "6ad02f3d-27a3-4934-b4eb-60b39ab74aa8" (UID: "6ad02f3d-27a3-4934-b4eb-60b39ab74aa8"). InnerVolumeSpecName "kube-api-access-8btw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:01:47.323145 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:47.323102    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ad02f3d-27a3-4934-b4eb-60b39ab74aa8-tls-certs" pod "6ad02f3d-27a3-4934-b4eb-60b39ab74aa8" (UID: "6ad02f3d-27a3-4934-b4eb-60b39ab74aa8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:01:47.423303 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:47.423268    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8btw4\" (UniqueName: \"kubernetes.io/projected/6ad02f3d-27a3-4934-b4eb-60b39ab74aa8-kube-api-access-8btw4\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:01:47.423303 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:47.423297    6224 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ad02f3d-27a3-4934-b4eb-60b39ab74aa8-tls-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:01:47.423303 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:47.423306    6224 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ad02f3d-27a3-4934-b4eb-60b39ab74aa8-dshm\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:01:47.684396 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:47.684360    6224 generic.go:356] "Generic (PLEG): container finished" podID="6ad02f3d-27a3-4934-b4eb-60b39ab74aa8" containerID="0a7c406a5b15f9173bb5674e56ea47849b8e054828945d956120d7f9c7497d6c" exitCode=0
Oct 30 08:01:47.684592 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:47.684402    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-932c00/rate-test-model-932c00-kserve-59bbdb5c89-vgxtp"
Oct 30 08:01:47.684592 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:47.684446    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-932c00/rate-test-model-932c00-kserve-59bbdb5c89-vgxtp" event={"ID":"6ad02f3d-27a3-4934-b4eb-60b39ab74aa8","Type":"ContainerDied","Data":"0a7c406a5b15f9173bb5674e56ea47849b8e054828945d956120d7f9c7497d6c"}
Oct 30 08:01:47.684592 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:47.684486    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-932c00/rate-test-model-932c00-kserve-59bbdb5c89-vgxtp" event={"ID":"6ad02f3d-27a3-4934-b4eb-60b39ab74aa8","Type":"ContainerDied","Data":"d35dd2083e3d9eb848fd75fa8bc120678a12f058cf3e11b8169a0fd4cdc68cbf"}
Oct 30 08:01:47.684592 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:47.684503    6224 scope.go:122] "RemoveContainer" containerID="0a7c406a5b15f9173bb5674e56ea47849b8e054828945d956120d7f9c7497d6c"
Oct 30 08:01:47.694100 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:47.694084    6224 scope.go:122] "RemoveContainer" containerID="0a7c406a5b15f9173bb5674e56ea47849b8e054828945d956120d7f9c7497d6c"
Oct 30 08:01:47.694347 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:01:47.694328    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a7c406a5b15f9173bb5674e56ea47849b8e054828945d956120d7f9c7497d6c\": container with ID starting with 0a7c406a5b15f9173bb5674e56ea47849b8e054828945d956120d7f9c7497d6c not found: ID does not exist" containerID="0a7c406a5b15f9173bb5674e56ea47849b8e054828945d956120d7f9c7497d6c"
Oct 30 08:01:47.694399 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:47.694354    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a7c406a5b15f9173bb5674e56ea47849b8e054828945d956120d7f9c7497d6c"} err="failed to get container status \"0a7c406a5b15f9173bb5674e56ea47849b8e054828945d956120d7f9c7497d6c\": rpc error: code = NotFound desc = could not find container \"0a7c406a5b15f9173bb5674e56ea47849b8e054828945d956120d7f9c7497d6c\": container with ID starting with 0a7c406a5b15f9173bb5674e56ea47849b8e054828945d956120d7f9c7497d6c not found: ID does not exist"
Oct 30 08:01:47.745082 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:47.745048    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-932c00/rate-test-model-932c00-kserve-59bbdb5c89-vgxtp"]
Oct 30 08:01:47.751588 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:47.751565    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-932c00/rate-test-model-932c00-kserve-59bbdb5c89-vgxtp"]
Oct 30 08:01:49.504504 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:49.504470    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ad02f3d-27a3-4934-b4eb-60b39ab74aa8" path="/var/lib/kubelet/pods/6ad02f3d-27a3-4934-b4eb-60b39ab74aa8/volumes"
Oct 30 08:01:59.024068 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:59.024033    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh"]
Oct 30 08:01:59.024523 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:01:59.024246    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh" podUID="6f5bab32-7cdb-471b-9bd0-38b205ace2e3" containerName="istio-proxy" containerID="cri-o://1961f45606306292f536917d1a90ba1d51c7d2670ab4a44d2ad2baa82f946ba7" gracePeriod=30
Oct 30 08:02:01.282360 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:01.282317    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-137122/rate-test-model-137122-kserve-6bb88d4545-29s55" podUID="85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477" containerName="main" containerID="cri-o://dac9882b70ea007883f5f3bfa68c81654ccddd49ab0f459c024d08bdf13eacb7" gracePeriod=45
Oct 30 08:02:01.328866 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:01.328838    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 08:02:01.566005 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:01.565975    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-137122/rate-test-model-137122-kserve-6bb88d4545-29s55"
Oct 30 08:02:01.711469 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:01.711436    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477-dshm\" (UniqueName: \"kubernetes.io/empty-dir/85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477-dshm\") pod \"85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477\" (UID: \"85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477\") "
Oct 30 08:02:01.711649 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:01.711505    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477-kube-api-access-chmzx\" (UniqueName: \"kubernetes.io/projected/85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477-kube-api-access-chmzx\") pod \"85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477\" (UID: \"85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477\") "
Oct 30 08:02:01.711649 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:01.711558    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477-tls-certs\" (UniqueName: \"kubernetes.io/secret/85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477-tls-certs\") pod \"85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477\" (UID: \"85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477\") "
Oct 30 08:02:01.711649 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:01.711625    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477-model-cache\") pod \"85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477\" (UID: \"85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477\") "
Oct 30 08:02:01.711649 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:01.711645    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477-tmp-dir\") pod \"85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477\" (UID: \"85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477\") "
Oct 30 08:02:01.711979 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:01.711708    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477-home\" (UniqueName: \"kubernetes.io/empty-dir/85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477-home\") pod \"85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477\" (UID: \"85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477\") "
Oct 30 08:02:01.711979 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:01.711908    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477-model-cache" pod "85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477" (UID: "85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:02:01.712097 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:01.711982    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477-tmp-dir" pod "85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477" (UID: "85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:02:01.712097 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:01.712039    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477-home" pod "85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477" (UID: "85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:02:01.712306 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:01.712285    6224 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477-model-cache\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:02:01.712391 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:01.712310    6224 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477-tmp-dir\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:02:01.712391 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:01.712337    6224 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477-home\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:02:01.713863 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:01.713836    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477-dshm" pod "85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477" (UID: "85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:02:01.713965 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:01.713864    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477-tls-certs" pod "85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477" (UID: "85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:02:01.713965 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:01.713877    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477-kube-api-access-chmzx" pod "85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477" (UID: "85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477"). InnerVolumeSpecName "kube-api-access-chmzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:02:01.814426 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:01.814250    6224 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477-dshm\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:02:01.814426 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:01.814288    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chmzx\" (UniqueName: \"kubernetes.io/projected/85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477-kube-api-access-chmzx\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:02:01.814426 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:01.814302    6224 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477-tls-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:02:01.831827 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:01.828414    6224 generic.go:356] "Generic (PLEG): container finished" podID="85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477" containerID="dac9882b70ea007883f5f3bfa68c81654ccddd49ab0f459c024d08bdf13eacb7" exitCode=0
Oct 30 08:02:01.831827 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:01.828451    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-137122/rate-test-model-137122-kserve-6bb88d4545-29s55" event={"ID":"85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477","Type":"ContainerDied","Data":"dac9882b70ea007883f5f3bfa68c81654ccddd49ab0f459c024d08bdf13eacb7"}
Oct 30 08:02:01.831827 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:01.828473    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-137122/rate-test-model-137122-kserve-6bb88d4545-29s55"
Oct 30 08:02:01.831827 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:01.828498    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-137122/rate-test-model-137122-kserve-6bb88d4545-29s55" event={"ID":"85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477","Type":"ContainerDied","Data":"e1c52bf236d7ad0d64e33d08061d9151e289b76c2a099fefc151386da351f886"}
Oct 30 08:02:01.831827 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:01.828520    6224 scope.go:122] "RemoveContainer" containerID="dac9882b70ea007883f5f3bfa68c81654ccddd49ab0f459c024d08bdf13eacb7"
Oct 30 08:02:01.850888 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:01.850866    6224 scope.go:122] "RemoveContainer" containerID="dac9882b70ea007883f5f3bfa68c81654ccddd49ab0f459c024d08bdf13eacb7"
Oct 30 08:02:01.851219 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:02:01.851187    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dac9882b70ea007883f5f3bfa68c81654ccddd49ab0f459c024d08bdf13eacb7\": container with ID starting with dac9882b70ea007883f5f3bfa68c81654ccddd49ab0f459c024d08bdf13eacb7 not found: ID does not exist" containerID="dac9882b70ea007883f5f3bfa68c81654ccddd49ab0f459c024d08bdf13eacb7"
Oct 30 08:02:01.851330 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:01.851227    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dac9882b70ea007883f5f3bfa68c81654ccddd49ab0f459c024d08bdf13eacb7"} err="failed to get container status \"dac9882b70ea007883f5f3bfa68c81654ccddd49ab0f459c024d08bdf13eacb7\": rpc error: code = NotFound desc = could not find container \"dac9882b70ea007883f5f3bfa68c81654ccddd49ab0f459c024d08bdf13eacb7\": container with ID starting with dac9882b70ea007883f5f3bfa68c81654ccddd49ab0f459c024d08bdf13eacb7 not found: ID does not exist"
Oct 30 08:02:01.869751 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:02:01.869719    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 08:02:01.887611 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:01.887585    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-137122/rate-test-model-137122-kserve-6bb88d4545-29s55"]
Oct 30 08:02:01.901959 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:01.901934    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-137122/rate-test-model-137122-kserve-6bb88d4545-29s55"]
Oct 30 08:02:03.503364 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:03.503289    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477" path="/var/lib/kubelet/pods/85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477/volumes"
Oct 30 08:02:03.681017 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:03.680982    6224 patch_prober.go:29] interesting pod/e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 08:02:03.681211 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:03.681044    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh" podUID="6f5bab32-7cdb-471b-9bd0-38b205ace2e3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 08:02:04.308902 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:04.308878    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh"
Oct 30 08:02:04.443268 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:04.443232    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f5bab32-7cdb-471b-9bd0-38b205ace2e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f5bab32-7cdb-471b-9bd0-38b205ace2e3-istio-data\") pod \"6f5bab32-7cdb-471b-9bd0-38b205ace2e3\" (UID: \"6f5bab32-7cdb-471b-9bd0-38b205ace2e3\") "
Oct 30 08:02:04.443481 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:04.443290    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f5bab32-7cdb-471b-9bd0-38b205ace2e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f5bab32-7cdb-471b-9bd0-38b205ace2e3-istiod-ca-cert\") pod \"6f5bab32-7cdb-471b-9bd0-38b205ace2e3\" (UID: \"6f5bab32-7cdb-471b-9bd0-38b205ace2e3\") "
Oct 30 08:02:04.443481 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:04.443349    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f5bab32-7cdb-471b-9bd0-38b205ace2e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f5bab32-7cdb-471b-9bd0-38b205ace2e3-istio-envoy\") pod \"6f5bab32-7cdb-471b-9bd0-38b205ace2e3\" (UID: \"6f5bab32-7cdb-471b-9bd0-38b205ace2e3\") "
Oct 30 08:02:04.443618 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:04.443504    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f5bab32-7cdb-471b-9bd0-38b205ace2e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f5bab32-7cdb-471b-9bd0-38b205ace2e3-workload-socket\") pod \"6f5bab32-7cdb-471b-9bd0-38b205ace2e3\" (UID: \"6f5bab32-7cdb-471b-9bd0-38b205ace2e3\") "
Oct 30 08:02:04.443618 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:04.443554    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f5bab32-7cdb-471b-9bd0-38b205ace2e3-istio-data" pod "6f5bab32-7cdb-471b-9bd0-38b205ace2e3" (UID: "6f5bab32-7cdb-471b-9bd0-38b205ace2e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:02:04.443618 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:04.443583    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f5bab32-7cdb-471b-9bd0-38b205ace2e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f5bab32-7cdb-471b-9bd0-38b205ace2e3-workload-certs\") pod \"6f5bab32-7cdb-471b-9bd0-38b205ace2e3\" (UID: \"6f5bab32-7cdb-471b-9bd0-38b205ace2e3\") "
Oct 30 08:02:04.443784 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:04.443626    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f5bab32-7cdb-471b-9bd0-38b205ace2e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f5bab32-7cdb-471b-9bd0-38b205ace2e3-istio-podinfo\") pod \"6f5bab32-7cdb-471b-9bd0-38b205ace2e3\" (UID: \"6f5bab32-7cdb-471b-9bd0-38b205ace2e3\") "
Oct 30 08:02:04.443784 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:04.443655    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f5bab32-7cdb-471b-9bd0-38b205ace2e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f5bab32-7cdb-471b-9bd0-38b205ace2e3-credential-socket\") pod \"6f5bab32-7cdb-471b-9bd0-38b205ace2e3\" (UID: \"6f5bab32-7cdb-471b-9bd0-38b205ace2e3\") "
Oct 30 08:02:04.443784 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:04.443704    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f5bab32-7cdb-471b-9bd0-38b205ace2e3-istio-token\" (UniqueName: \"kubernetes.io/projected/6f5bab32-7cdb-471b-9bd0-38b205ace2e3-istio-token\") pod \"6f5bab32-7cdb-471b-9bd0-38b205ace2e3\" (UID: \"6f5bab32-7cdb-471b-9bd0-38b205ace2e3\") "
Oct 30 08:02:04.443784 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:04.443742    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f5bab32-7cdb-471b-9bd0-38b205ace2e3-kube-api-access-t6cv4\" (UniqueName: \"kubernetes.io/projected/6f5bab32-7cdb-471b-9bd0-38b205ace2e3-kube-api-access-t6cv4\") pod \"6f5bab32-7cdb-471b-9bd0-38b205ace2e3\" (UID: \"6f5bab32-7cdb-471b-9bd0-38b205ace2e3\") "
Oct 30 08:02:04.443784 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:04.443755    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f5bab32-7cdb-471b-9bd0-38b205ace2e3-istiod-ca-cert" pod "6f5bab32-7cdb-471b-9bd0-38b205ace2e3" (UID: "6f5bab32-7cdb-471b-9bd0-38b205ace2e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:02:04.444089 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:04.443961    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f5bab32-7cdb-471b-9bd0-38b205ace2e3-workload-certs" pod "6f5bab32-7cdb-471b-9bd0-38b205ace2e3" (UID: "6f5bab32-7cdb-471b-9bd0-38b205ace2e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:02:04.444452 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:04.444286    6224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f5bab32-7cdb-471b-9bd0-38b205ace2e3-workload-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:02:04.444452 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:04.444328    6224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f5bab32-7cdb-471b-9bd0-38b205ace2e3-istio-data\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:02:04.444452 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:04.444342    6224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f5bab32-7cdb-471b-9bd0-38b205ace2e3-istiod-ca-cert\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:02:04.445105 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:04.444659    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f5bab32-7cdb-471b-9bd0-38b205ace2e3-credential-socket" pod "6f5bab32-7cdb-471b-9bd0-38b205ace2e3" (UID: "6f5bab32-7cdb-471b-9bd0-38b205ace2e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:02:04.445105 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:04.444763    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f5bab32-7cdb-471b-9bd0-38b205ace2e3-workload-socket" pod "6f5bab32-7cdb-471b-9bd0-38b205ace2e3" (UID: "6f5bab32-7cdb-471b-9bd0-38b205ace2e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:02:04.446250 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:04.446226    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f5bab32-7cdb-471b-9bd0-38b205ace2e3-istio-envoy" pod "6f5bab32-7cdb-471b-9bd0-38b205ace2e3" (UID: "6f5bab32-7cdb-471b-9bd0-38b205ace2e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:02:04.446338 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:04.446258    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f5bab32-7cdb-471b-9bd0-38b205ace2e3-istio-token" pod "6f5bab32-7cdb-471b-9bd0-38b205ace2e3" (UID: "6f5bab32-7cdb-471b-9bd0-38b205ace2e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:02:04.446396 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:04.446345    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f5bab32-7cdb-471b-9bd0-38b205ace2e3-istio-podinfo" pod "6f5bab32-7cdb-471b-9bd0-38b205ace2e3" (UID: "6f5bab32-7cdb-471b-9bd0-38b205ace2e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 08:02:04.446455 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:04.446412    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f5bab32-7cdb-471b-9bd0-38b205ace2e3-kube-api-access-t6cv4" pod "6f5bab32-7cdb-471b-9bd0-38b205ace2e3" (UID: "6f5bab32-7cdb-471b-9bd0-38b205ace2e3"). InnerVolumeSpecName "kube-api-access-t6cv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:02:04.545360 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:04.545281    6224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f5bab32-7cdb-471b-9bd0-38b205ace2e3-workload-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:02:04.545360 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:04.545312    6224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f5bab32-7cdb-471b-9bd0-38b205ace2e3-istio-podinfo\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:02:04.545360 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:04.545323    6224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f5bab32-7cdb-471b-9bd0-38b205ace2e3-credential-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:02:04.545360 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:04.545335    6224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f5bab32-7cdb-471b-9bd0-38b205ace2e3-istio-token\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:02:04.545360 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:04.545348    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6cv4\" (UniqueName: \"kubernetes.io/projected/6f5bab32-7cdb-471b-9bd0-38b205ace2e3-kube-api-access-t6cv4\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:02:04.545360 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:04.545356    6224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f5bab32-7cdb-471b-9bd0-38b205ace2e3-istio-envoy\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:02:04.853095 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:04.853009    6224 generic.go:356] "Generic (PLEG): container finished" podID="6f5bab32-7cdb-471b-9bd0-38b205ace2e3" containerID="1961f45606306292f536917d1a90ba1d51c7d2670ab4a44d2ad2baa82f946ba7" exitCode=0
Oct 30 08:02:04.853095 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:04.853066    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh"
Oct 30 08:02:04.853298 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:04.853097    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh" event={"ID":"6f5bab32-7cdb-471b-9bd0-38b205ace2e3","Type":"ContainerDied","Data":"1961f45606306292f536917d1a90ba1d51c7d2670ab4a44d2ad2baa82f946ba7"}
Oct 30 08:02:04.853298 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:04.853163    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh" event={"ID":"6f5bab32-7cdb-471b-9bd0-38b205ace2e3","Type":"ContainerDied","Data":"431abbf7a4f5d83a5934f24b81cb09bfb4b86da9b5dd7c1da6371979e8571b29"}
Oct 30 08:02:04.853298 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:04.853190    6224 scope.go:122] "RemoveContainer" containerID="1961f45606306292f536917d1a90ba1d51c7d2670ab4a44d2ad2baa82f946ba7"
Oct 30 08:02:04.864986 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:04.864967    6224 scope.go:122] "RemoveContainer" containerID="cb68c675228dc2925720b0e9ca5511cb2a4d461c920b79a8684c7137f0385a91"
Oct 30 08:02:04.875135 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:04.875119    6224 scope.go:122] "RemoveContainer" containerID="1961f45606306292f536917d1a90ba1d51c7d2670ab4a44d2ad2baa82f946ba7"
Oct 30 08:02:04.875377 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:02:04.875359    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1961f45606306292f536917d1a90ba1d51c7d2670ab4a44d2ad2baa82f946ba7\": container with ID starting with 1961f45606306292f536917d1a90ba1d51c7d2670ab4a44d2ad2baa82f946ba7 not found: ID does not exist" containerID="1961f45606306292f536917d1a90ba1d51c7d2670ab4a44d2ad2baa82f946ba7"
Oct 30 08:02:04.875439 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:04.875385    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1961f45606306292f536917d1a90ba1d51c7d2670ab4a44d2ad2baa82f946ba7"} err="failed to get container status \"1961f45606306292f536917d1a90ba1d51c7d2670ab4a44d2ad2baa82f946ba7\": rpc error: code = NotFound desc = could not find container \"1961f45606306292f536917d1a90ba1d51c7d2670ab4a44d2ad2baa82f946ba7\": container with ID starting with 1961f45606306292f536917d1a90ba1d51c7d2670ab4a44d2ad2baa82f946ba7 not found: ID does not exist"
Oct 30 08:02:04.875439 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:04.875402    6224 scope.go:122] "RemoveContainer" containerID="cb68c675228dc2925720b0e9ca5511cb2a4d461c920b79a8684c7137f0385a91"
Oct 30 08:02:04.875650 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:02:04.875630    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb68c675228dc2925720b0e9ca5511cb2a4d461c920b79a8684c7137f0385a91\": container with ID starting with cb68c675228dc2925720b0e9ca5511cb2a4d461c920b79a8684c7137f0385a91 not found: ID does not exist" containerID="cb68c675228dc2925720b0e9ca5511cb2a4d461c920b79a8684c7137f0385a91"
Oct 30 08:02:04.875705 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:04.875659    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb68c675228dc2925720b0e9ca5511cb2a4d461c920b79a8684c7137f0385a91"} err="failed to get container status \"cb68c675228dc2925720b0e9ca5511cb2a4d461c920b79a8684c7137f0385a91\": rpc error: code = NotFound desc = could not find container \"cb68c675228dc2925720b0e9ca5511cb2a4d461c920b79a8684c7137f0385a91\": container with ID starting with cb68c675228dc2925720b0e9ca5511cb2a4d461c920b79a8684c7137f0385a91 not found: ID does not exist"
Oct 30 08:02:04.915159 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:04.915135    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh"]
Oct 30 08:02:04.919830 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:04.919775    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-f1f700-openshift-default-6f8bcfd4b9-jz8lh"]
Oct 30 08:02:05.505141 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:05.505096    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f5bab32-7cdb-471b-9bd0-38b205ace2e3" path="/var/lib/kubelet/pods/6f5bab32-7cdb-471b-9bd0-38b205ace2e3/volumes"
Oct 30 08:02:16.640888 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.640853    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-f5ceca/rate-test-model-f5ceca-kserve-845bf4bfdf-ck76g"]
Oct 30 08:02:16.641436 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641407    6224 cpu_manager.go:414] "containerMap: removing container" podUID="da938ecc-f439-4873-b964-8102a4d0dfb5" containerName="cleanup"
Oct 30 08:02:16.641436 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641420    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da938ecc-f439-4873-b964-8102a4d0dfb5" containerName="cleanup"
Oct 30 08:02:16.641436 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641431    6224 cpu_manager.go:414] "containerMap: removing container" podUID="da7c6736-ac92-44ec-a1cf-5f835eb860b1" containerName="cleanup"
Oct 30 08:02:16.641436 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641436    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da7c6736-ac92-44ec-a1cf-5f835eb860b1" containerName="cleanup"
Oct 30 08:02:16.641589 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641445    6224 cpu_manager.go:414] "containerMap: removing container" podUID="85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477" containerName="main"
Oct 30 08:02:16.641589 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641450    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477" containerName="main"
Oct 30 08:02:16.641589 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641462    6224 cpu_manager.go:414] "containerMap: removing container" podUID="fb6ead4a-d324-450f-adb6-dbbb75afc174" containerName="cleanup"
Oct 30 08:02:16.641589 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641466    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb6ead4a-d324-450f-adb6-dbbb75afc174" containerName="cleanup"
Oct 30 08:02:16.641589 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641475    6224 cpu_manager.go:414] "containerMap: removing container" podUID="ca0e2ee4-723d-4fc1-b13e-72b13c83d280" containerName="istio-proxy"
Oct 30 08:02:16.641589 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641479    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca0e2ee4-723d-4fc1-b13e-72b13c83d280" containerName="istio-proxy"
Oct 30 08:02:16.641589 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641488    6224 cpu_manager.go:414] "containerMap: removing container" podUID="2eec27f2-9ce9-4269-9c99-0ab3839cf2ee" containerName="payload-processing"
Oct 30 08:02:16.641589 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641492    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2eec27f2-9ce9-4269-9c99-0ab3839cf2ee" containerName="payload-processing"
Oct 30 08:02:16.641589 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641499    6224 cpu_manager.go:414] "containerMap: removing container" podUID="38de25ac-6fa2-4650-9cfa-ec23daf2ec76" containerName="cleanup"
Oct 30 08:02:16.641589 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641503    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38de25ac-6fa2-4650-9cfa-ec23daf2ec76" containerName="cleanup"
Oct 30 08:02:16.641589 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641514    6224 cpu_manager.go:414] "containerMap: removing container" podUID="1ea09206-913c-471c-b942-4a26878ee5e6" containerName="payload-pre-processing"
Oct 30 08:02:16.641589 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641519    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ea09206-913c-471c-b942-4a26878ee5e6" containerName="payload-pre-processing"
Oct 30 08:02:16.641589 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641528    6224 cpu_manager.go:414] "containerMap: removing container" podUID="ca0e2ee4-723d-4fc1-b13e-72b13c83d280" containerName="istio-proxy"
Oct 30 08:02:16.641589 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641533    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca0e2ee4-723d-4fc1-b13e-72b13c83d280" containerName="istio-proxy"
Oct 30 08:02:16.641589 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641540    6224 cpu_manager.go:414] "containerMap: removing container" podUID="3a639cc2-221b-4e62-86c7-64ae4f7c43f7" containerName="cleanup"
Oct 30 08:02:16.641589 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641544    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a639cc2-221b-4e62-86c7-64ae4f7c43f7" containerName="cleanup"
Oct 30 08:02:16.641589 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641550    6224 cpu_manager.go:414] "containerMap: removing container" podUID="f68a7a87-6354-4456-ab2c-17cf8995865f" containerName="cleanup"
Oct 30 08:02:16.641589 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641554    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f68a7a87-6354-4456-ab2c-17cf8995865f" containerName="cleanup"
Oct 30 08:02:16.641589 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641560    6224 cpu_manager.go:414] "containerMap: removing container" podUID="38de25ac-6fa2-4650-9cfa-ec23daf2ec76" containerName="cleanup"
Oct 30 08:02:16.641589 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641564    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38de25ac-6fa2-4650-9cfa-ec23daf2ec76" containerName="cleanup"
Oct 30 08:02:16.641589 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641570    6224 cpu_manager.go:414] "containerMap: removing container" podUID="e8936cc2-40e5-4d2e-81b0-1ad1ca9e5a2a" containerName="cleanup"
Oct 30 08:02:16.641589 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641574    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8936cc2-40e5-4d2e-81b0-1ad1ca9e5a2a" containerName="cleanup"
Oct 30 08:02:16.641589 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641583    6224 cpu_manager.go:414] "containerMap: removing container" podUID="3a639cc2-221b-4e62-86c7-64ae4f7c43f7" containerName="cleanup"
Oct 30 08:02:16.641589 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641587    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a639cc2-221b-4e62-86c7-64ae4f7c43f7" containerName="cleanup"
Oct 30 08:02:16.641589 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641593    6224 cpu_manager.go:414] "containerMap: removing container" podUID="246d2cbc-c719-46fd-a84d-0e47a21358fe" containerName="cleanup"
Oct 30 08:02:16.641589 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641597    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="246d2cbc-c719-46fd-a84d-0e47a21358fe" containerName="cleanup"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641603    6224 cpu_manager.go:414] "containerMap: removing container" podUID="fb6ead4a-d324-450f-adb6-dbbb75afc174" containerName="cleanup"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641607    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb6ead4a-d324-450f-adb6-dbbb75afc174" containerName="cleanup"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641614    6224 cpu_manager.go:414] "containerMap: removing container" podUID="246d2cbc-c719-46fd-a84d-0e47a21358fe" containerName="cleanup"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641618    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="246d2cbc-c719-46fd-a84d-0e47a21358fe" containerName="cleanup"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641625    6224 cpu_manager.go:414] "containerMap: removing container" podUID="6f5bab32-7cdb-471b-9bd0-38b205ace2e3" containerName="istio-proxy"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641630    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f5bab32-7cdb-471b-9bd0-38b205ace2e3" containerName="istio-proxy"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641637    6224 cpu_manager.go:414] "containerMap: removing container" podUID="271eca48-7f0b-4619-914c-bbf966c47e3d" containerName="payload-pre-processing"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641642    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="271eca48-7f0b-4619-914c-bbf966c47e3d" containerName="payload-pre-processing"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641649    6224 cpu_manager.go:414] "containerMap: removing container" podUID="fb6ead4a-d324-450f-adb6-dbbb75afc174" containerName="cleanup"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641653    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb6ead4a-d324-450f-adb6-dbbb75afc174" containerName="cleanup"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641661    6224 cpu_manager.go:414] "containerMap: removing container" podUID="246d2cbc-c719-46fd-a84d-0e47a21358fe" containerName="cleanup"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641665    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="246d2cbc-c719-46fd-a84d-0e47a21358fe" containerName="cleanup"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641672    6224 cpu_manager.go:414] "containerMap: removing container" podUID="a0e05a96-75ea-4636-b9d9-f08b5c1ba1ad" containerName="cleanup"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641676    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0e05a96-75ea-4636-b9d9-f08b5c1ba1ad" containerName="cleanup"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641682    6224 cpu_manager.go:414] "containerMap: removing container" podUID="1cd532fb-2369-434f-9622-d6792f8ee7a0" containerName="cleanup"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641687    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cd532fb-2369-434f-9622-d6792f8ee7a0" containerName="cleanup"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641695    6224 cpu_manager.go:414] "containerMap: removing container" podUID="e8936cc2-40e5-4d2e-81b0-1ad1ca9e5a2a" containerName="cleanup"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641700    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8936cc2-40e5-4d2e-81b0-1ad1ca9e5a2a" containerName="cleanup"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641707    6224 cpu_manager.go:414] "containerMap: removing container" podUID="6f5bab32-7cdb-471b-9bd0-38b205ace2e3" containerName="istio-proxy"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641711    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f5bab32-7cdb-471b-9bd0-38b205ace2e3" containerName="istio-proxy"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641719    6224 cpu_manager.go:414] "containerMap: removing container" podUID="bc9e4c3b-9885-4706-817b-4da954d01dfb" containerName="maas-api"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641723    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc9e4c3b-9885-4706-817b-4da954d01dfb" containerName="maas-api"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641729    6224 cpu_manager.go:414] "containerMap: removing container" podUID="f68a7a87-6354-4456-ab2c-17cf8995865f" containerName="cleanup"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641734    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f68a7a87-6354-4456-ab2c-17cf8995865f" containerName="cleanup"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641742    6224 cpu_manager.go:414] "containerMap: removing container" podUID="1cd532fb-2369-434f-9622-d6792f8ee7a0" containerName="cleanup"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641746    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cd532fb-2369-434f-9622-d6792f8ee7a0" containerName="cleanup"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641751    6224 cpu_manager.go:414] "containerMap: removing container" podUID="da7c6736-ac92-44ec-a1cf-5f835eb860b1" containerName="cleanup"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641755    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da7c6736-ac92-44ec-a1cf-5f835eb860b1" containerName="cleanup"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641761    6224 cpu_manager.go:414] "containerMap: removing container" podUID="da938ecc-f439-4873-b964-8102a4d0dfb5" containerName="cleanup"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641766    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da938ecc-f439-4873-b964-8102a4d0dfb5" containerName="cleanup"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641773    6224 cpu_manager.go:414] "containerMap: removing container" podUID="64034fab-2b81-4f54-9d0c-5f2360ab6274" containerName="payload-processing"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641778    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64034fab-2b81-4f54-9d0c-5f2360ab6274" containerName="payload-processing"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641785    6224 cpu_manager.go:414] "containerMap: removing container" podUID="a0e05a96-75ea-4636-b9d9-f08b5c1ba1ad" containerName="cleanup"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641789    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0e05a96-75ea-4636-b9d9-f08b5c1ba1ad" containerName="cleanup"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641796    6224 cpu_manager.go:414] "containerMap: removing container" podUID="da938ecc-f439-4873-b964-8102a4d0dfb5" containerName="cleanup"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641800    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da938ecc-f439-4873-b964-8102a4d0dfb5" containerName="cleanup"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641819    6224 cpu_manager.go:414] "containerMap: removing container" podUID="6ad02f3d-27a3-4934-b4eb-60b39ab74aa8" containerName="main"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641824    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ad02f3d-27a3-4934-b4eb-60b39ab74aa8" containerName="main"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641831    6224 cpu_manager.go:414] "containerMap: removing container" podUID="1825dd2c-f162-4d90-8ebd-a6923e990abd" containerName="payload-processing"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641835    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1825dd2c-f162-4d90-8ebd-a6923e990abd" containerName="payload-processing"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641844    6224 cpu_manager.go:414] "containerMap: removing container" podUID="a0e05a96-75ea-4636-b9d9-f08b5c1ba1ad" containerName="cleanup"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641848    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0e05a96-75ea-4636-b9d9-f08b5c1ba1ad" containerName="cleanup"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641858    6224 cpu_manager.go:414] "containerMap: removing container" podUID="0d55b266-9ae5-4b44-85e3-ea14553e4e27" containerName="payload-pre-processing"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641863    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d55b266-9ae5-4b44-85e3-ea14553e4e27" containerName="payload-pre-processing"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641956    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a639cc2-221b-4e62-86c7-64ae4f7c43f7" containerName="cleanup"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641963    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="246d2cbc-c719-46fd-a84d-0e47a21358fe" containerName="cleanup"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641969    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb6ead4a-d324-450f-adb6-dbbb75afc174" containerName="cleanup"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641975    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="da7c6736-ac92-44ec-a1cf-5f835eb860b1" containerName="cleanup"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641980    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="246d2cbc-c719-46fd-a84d-0e47a21358fe" containerName="cleanup"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641986    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f5bab32-7cdb-471b-9bd0-38b205ace2e3" containerName="istio-proxy"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641992    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="271eca48-7f0b-4619-914c-bbf966c47e3d" containerName="payload-pre-processing"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.641999    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0e05a96-75ea-4636-b9d9-f08b5c1ba1ad" containerName="cleanup"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.642004    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cd532fb-2369-434f-9622-d6792f8ee7a0" containerName="cleanup"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.642009    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8936cc2-40e5-4d2e-81b0-1ad1ca9e5a2a" containerName="cleanup"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.642015    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f5bab32-7cdb-471b-9bd0-38b205ace2e3" containerName="istio-proxy"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.642019    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="f68a7a87-6354-4456-ab2c-17cf8995865f" containerName="cleanup"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.642035    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc9e4c3b-9885-4706-817b-4da954d01dfb" containerName="maas-api"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.642040    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="38de25ac-6fa2-4650-9cfa-ec23daf2ec76" containerName="cleanup"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.642045    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="f68a7a87-6354-4456-ab2c-17cf8995865f" containerName="cleanup"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.642051    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cd532fb-2369-434f-9622-d6792f8ee7a0" containerName="cleanup"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.642057    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="da7c6736-ac92-44ec-a1cf-5f835eb860b1" containerName="cleanup"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.642063    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="da938ecc-f439-4873-b964-8102a4d0dfb5" containerName="cleanup"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.642069    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="64034fab-2b81-4f54-9d0c-5f2360ab6274" containerName="payload-processing"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.642074    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0e05a96-75ea-4636-b9d9-f08b5c1ba1ad" containerName="cleanup"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.642079    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="da938ecc-f439-4873-b964-8102a4d0dfb5" containerName="cleanup"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.642084    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ad02f3d-27a3-4934-b4eb-60b39ab74aa8" containerName="main"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.642090    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="1825dd2c-f162-4d90-8ebd-a6923e990abd" containerName="payload-processing"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.642097    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0e05a96-75ea-4636-b9d9-f08b5c1ba1ad" containerName="cleanup"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.642103    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8936cc2-40e5-4d2e-81b0-1ad1ca9e5a2a" containerName="cleanup"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.642108    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d55b266-9ae5-4b44-85e3-ea14553e4e27" containerName="payload-pre-processing"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.642113    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="da938ecc-f439-4873-b964-8102a4d0dfb5" containerName="cleanup"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.642118    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="85dfc83f-9c0c-4bc6-a4ab-998d6c2d1477" containerName="main"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.642124    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb6ead4a-d324-450f-adb6-dbbb75afc174" containerName="cleanup"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.642128    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca0e2ee4-723d-4fc1-b13e-72b13c83d280" containerName="istio-proxy"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.642133    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="2eec27f2-9ce9-4269-9c99-0ab3839cf2ee" containerName="payload-processing"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.642138    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ea09206-913c-471c-b942-4a26878ee5e6" containerName="payload-pre-processing"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.642144    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca0e2ee4-723d-4fc1-b13e-72b13c83d280" containerName="istio-proxy"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.642150    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a639cc2-221b-4e62-86c7-64ae4f7c43f7" containerName="cleanup"
Oct 30 08:02:16.642587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.642156    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="38de25ac-6fa2-4650-9cfa-ec23daf2ec76" containerName="cleanup"
Oct 30 08:02:16.647303 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.647275    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f5ceca/rate-test-model-f5ceca-kserve-845bf4bfdf-ck76g"
Oct 30 08:02:16.668680 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.668653    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f5ceca/rate-test-model-f5ceca-kserve-845bf4bfdf-ck76g"]
Oct 30 08:02:16.679881 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.679855    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-f5ceca\"/\"kube-root-ca.crt\""
Oct 30 08:02:16.679881 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.679857    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-f5ceca\"/\"openshift-service-ca.crt\""
Oct 30 08:02:16.682478 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.682450    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-f5ceca\"/\"rate-test-model-f5ceca-kserve-self-signed-certs\""
Oct 30 08:02:16.682478 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.682446    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-f5ceca\"/\"default-dockercfg-p7lgh\""
Oct 30 08:02:16.699621 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.699588    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9d4bb88-868c-4767-be0a-08f21d7a1f8c-model-cache\") pod \"rate-test-model-f5ceca-kserve-845bf4bfdf-ck76g\" (UID: \"f9d4bb88-868c-4767-be0a-08f21d7a1f8c\") " pod="ai-tenant-e2e-shared-b-w3-f5ceca/rate-test-model-f5ceca-kserve-845bf4bfdf-ck76g"
Oct 30 08:02:16.699827 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.699706    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9d4bb88-868c-4767-be0a-08f21d7a1f8c-dshm\") pod \"rate-test-model-f5ceca-kserve-845bf4bfdf-ck76g\" (UID: \"f9d4bb88-868c-4767-be0a-08f21d7a1f8c\") " pod="ai-tenant-e2e-shared-b-w3-f5ceca/rate-test-model-f5ceca-kserve-845bf4bfdf-ck76g"
Oct 30 08:02:16.699827 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.699738    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9d4bb88-868c-4767-be0a-08f21d7a1f8c-home\") pod \"rate-test-model-f5ceca-kserve-845bf4bfdf-ck76g\" (UID: \"f9d4bb88-868c-4767-be0a-08f21d7a1f8c\") " pod="ai-tenant-e2e-shared-b-w3-f5ceca/rate-test-model-f5ceca-kserve-845bf4bfdf-ck76g"
Oct 30 08:02:16.699827 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.699771    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9d4bb88-868c-4767-be0a-08f21d7a1f8c-tls-certs\") pod \"rate-test-model-f5ceca-kserve-845bf4bfdf-ck76g\" (UID: \"f9d4bb88-868c-4767-be0a-08f21d7a1f8c\") " pod="ai-tenant-e2e-shared-b-w3-f5ceca/rate-test-model-f5ceca-kserve-845bf4bfdf-ck76g"
Oct 30 08:02:16.700010 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.699960    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jflkt\" (UniqueName: \"kubernetes.io/projected/f9d4bb88-868c-4767-be0a-08f21d7a1f8c-kube-api-access-jflkt\") pod \"rate-test-model-f5ceca-kserve-845bf4bfdf-ck76g\" (UID: \"f9d4bb88-868c-4767-be0a-08f21d7a1f8c\") " pod="ai-tenant-e2e-shared-b-w3-f5ceca/rate-test-model-f5ceca-kserve-845bf4bfdf-ck76g"
Oct 30 08:02:16.700080 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.700011    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9d4bb88-868c-4767-be0a-08f21d7a1f8c-tmp-dir\") pod \"rate-test-model-f5ceca-kserve-845bf4bfdf-ck76g\" (UID: \"f9d4bb88-868c-4767-be0a-08f21d7a1f8c\") " pod="ai-tenant-e2e-shared-b-w3-f5ceca/rate-test-model-f5ceca-kserve-845bf4bfdf-ck76g"
Oct 30 08:02:16.801622 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.801580    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jflkt\" (UniqueName: \"kubernetes.io/projected/f9d4bb88-868c-4767-be0a-08f21d7a1f8c-kube-api-access-jflkt\") pod \"rate-test-model-f5ceca-kserve-845bf4bfdf-ck76g\" (UID: \"f9d4bb88-868c-4767-be0a-08f21d7a1f8c\") " pod="ai-tenant-e2e-shared-b-w3-f5ceca/rate-test-model-f5ceca-kserve-845bf4bfdf-ck76g"
Oct 30 08:02:16.801622 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.801629    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9d4bb88-868c-4767-be0a-08f21d7a1f8c-tmp-dir\") pod \"rate-test-model-f5ceca-kserve-845bf4bfdf-ck76g\" (UID: \"f9d4bb88-868c-4767-be0a-08f21d7a1f8c\") " pod="ai-tenant-e2e-shared-b-w3-f5ceca/rate-test-model-f5ceca-kserve-845bf4bfdf-ck76g"
Oct 30 08:02:16.801945 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.801665    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9d4bb88-868c-4767-be0a-08f21d7a1f8c-model-cache\") pod \"rate-test-model-f5ceca-kserve-845bf4bfdf-ck76g\" (UID: \"f9d4bb88-868c-4767-be0a-08f21d7a1f8c\") " pod="ai-tenant-e2e-shared-b-w3-f5ceca/rate-test-model-f5ceca-kserve-845bf4bfdf-ck76g"
Oct 30 08:02:16.801945 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.801715    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9d4bb88-868c-4767-be0a-08f21d7a1f8c-dshm\") pod \"rate-test-model-f5ceca-kserve-845bf4bfdf-ck76g\" (UID: \"f9d4bb88-868c-4767-be0a-08f21d7a1f8c\") " pod="ai-tenant-e2e-shared-b-w3-f5ceca/rate-test-model-f5ceca-kserve-845bf4bfdf-ck76g"
Oct 30 08:02:16.801945 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.801740    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9d4bb88-868c-4767-be0a-08f21d7a1f8c-home\") pod \"rate-test-model-f5ceca-kserve-845bf4bfdf-ck76g\" (UID: \"f9d4bb88-868c-4767-be0a-08f21d7a1f8c\") " pod="ai-tenant-e2e-shared-b-w3-f5ceca/rate-test-model-f5ceca-kserve-845bf4bfdf-ck76g"
Oct 30 08:02:16.801945 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.801762    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9d4bb88-868c-4767-be0a-08f21d7a1f8c-tls-certs\") pod \"rate-test-model-f5ceca-kserve-845bf4bfdf-ck76g\" (UID: \"f9d4bb88-868c-4767-be0a-08f21d7a1f8c\") " pod="ai-tenant-e2e-shared-b-w3-f5ceca/rate-test-model-f5ceca-kserve-845bf4bfdf-ck76g"
Oct 30 08:02:16.802172 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.802101    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9d4bb88-868c-4767-be0a-08f21d7a1f8c-tmp-dir\") pod \"rate-test-model-f5ceca-kserve-845bf4bfdf-ck76g\" (UID: \"f9d4bb88-868c-4767-be0a-08f21d7a1f8c\") " pod="ai-tenant-e2e-shared-b-w3-f5ceca/rate-test-model-f5ceca-kserve-845bf4bfdf-ck76g"
Oct 30 08:02:16.802172 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.802160    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9d4bb88-868c-4767-be0a-08f21d7a1f8c-model-cache\") pod \"rate-test-model-f5ceca-kserve-845bf4bfdf-ck76g\" (UID: \"f9d4bb88-868c-4767-be0a-08f21d7a1f8c\") " pod="ai-tenant-e2e-shared-b-w3-f5ceca/rate-test-model-f5ceca-kserve-845bf4bfdf-ck76g"
Oct 30 08:02:16.802288 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.802268    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9d4bb88-868c-4767-be0a-08f21d7a1f8c-home\") pod \"rate-test-model-f5ceca-kserve-845bf4bfdf-ck76g\" (UID: \"f9d4bb88-868c-4767-be0a-08f21d7a1f8c\") " pod="ai-tenant-e2e-shared-b-w3-f5ceca/rate-test-model-f5ceca-kserve-845bf4bfdf-ck76g"
Oct 30 08:02:16.804324 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.804295    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9d4bb88-868c-4767-be0a-08f21d7a1f8c-dshm\") pod \"rate-test-model-f5ceca-kserve-845bf4bfdf-ck76g\" (UID: \"f9d4bb88-868c-4767-be0a-08f21d7a1f8c\") " pod="ai-tenant-e2e-shared-b-w3-f5ceca/rate-test-model-f5ceca-kserve-845bf4bfdf-ck76g"
Oct 30 08:02:16.805074 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.805049    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9d4bb88-868c-4767-be0a-08f21d7a1f8c-tls-certs\") pod \"rate-test-model-f5ceca-kserve-845bf4bfdf-ck76g\" (UID: \"f9d4bb88-868c-4767-be0a-08f21d7a1f8c\") " pod="ai-tenant-e2e-shared-b-w3-f5ceca/rate-test-model-f5ceca-kserve-845bf4bfdf-ck76g"
Oct 30 08:02:16.827563 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.827530    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jflkt\" (UniqueName: \"kubernetes.io/projected/f9d4bb88-868c-4767-be0a-08f21d7a1f8c-kube-api-access-jflkt\") pod \"rate-test-model-f5ceca-kserve-845bf4bfdf-ck76g\" (UID: \"f9d4bb88-868c-4767-be0a-08f21d7a1f8c\") " pod="ai-tenant-e2e-shared-b-w3-f5ceca/rate-test-model-f5ceca-kserve-845bf4bfdf-ck76g"
Oct 30 08:02:16.963866 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:16.963759    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f5ceca/rate-test-model-f5ceca-kserve-845bf4bfdf-ck76g"
Oct 30 08:02:17.300383 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:17.300338    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f5ceca/rate-test-model-f5ceca-kserve-845bf4bfdf-ck76g"]
Oct 30 08:02:17.304898 ip-10-0-128-125 kubenswrapper[6224]: W1030 08:02:17.304869    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf9d4bb88_868c_4767_be0a_08f21d7a1f8c.slice/crio-95cbdc209e22a3cdc087dbffb99c7c10a41f0a9d1fae23b028efc31b5828865d WatchSource:0}: Error finding container 95cbdc209e22a3cdc087dbffb99c7c10a41f0a9d1fae23b028efc31b5828865d: Status 404 returned error can't find the container with id 95cbdc209e22a3cdc087dbffb99c7c10a41f0a9d1fae23b028efc31b5828865d
Oct 30 08:02:17.398304 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:17.398273    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf"]
Oct 30 08:02:17.398509 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:17.398484    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf" podUID="89326bce-8281-4747-aa3f-c1d37c615234" containerName="istio-proxy" containerID="cri-o://72cd4cbefbae77bd6a6bdd3ce4ee12ffabdc8d855eaebfa80a3ba7bca32fd5a7" gracePeriod=30
Oct 30 08:02:17.946850 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:17.946793    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f5ceca/rate-test-model-f5ceca-kserve-845bf4bfdf-ck76g" event={"ID":"f9d4bb88-868c-4767-be0a-08f21d7a1f8c","Type":"ContainerStarted","Data":"60801af8224f24604a725d97b39820ec79c9a67252a5f5d62c8fb11c80043998"}
Oct 30 08:02:17.947245 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:17.946858    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f5ceca/rate-test-model-f5ceca-kserve-845bf4bfdf-ck76g" event={"ID":"f9d4bb88-868c-4767-be0a-08f21d7a1f8c","Type":"ContainerStarted","Data":"95cbdc209e22a3cdc087dbffb99c7c10a41f0a9d1fae23b028efc31b5828865d"}
Oct 30 08:02:17.964065 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:17.964039    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-f5ceca/rate-test-model-f5ceca-kserve-845bf4bfdf-ck76g"
Oct 30 08:02:17.996550 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:17.996497    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-f5ceca/rate-test-model-f5ceca-kserve-845bf4bfdf-ck76g" podStartSLOduration=1.996479362 podStartE2EDuration="1.996479362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:02:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:02:17.992156345 +0000 UTC m=+3896.601563315" watchObservedRunningTime="2026-10-30 08:02:17.996479362 +0000 UTC m=+3896.605886338"
Oct 30 08:02:22.666265 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:22.666193    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf"
Oct 30 08:02:22.776620 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:22.776574    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89326bce-8281-4747-aa3f-c1d37c615234-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89326bce-8281-4747-aa3f-c1d37c615234-workload-socket\") pod \"89326bce-8281-4747-aa3f-c1d37c615234\" (UID: \"89326bce-8281-4747-aa3f-c1d37c615234\") "
Oct 30 08:02:22.776844 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:22.776670    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89326bce-8281-4747-aa3f-c1d37c615234-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89326bce-8281-4747-aa3f-c1d37c615234-istio-envoy\") pod \"89326bce-8281-4747-aa3f-c1d37c615234\" (UID: \"89326bce-8281-4747-aa3f-c1d37c615234\") "
Oct 30 08:02:22.776844 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:22.776753    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/89326bce-8281-4747-aa3f-c1d37c615234-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89326bce-8281-4747-aa3f-c1d37c615234-istio-podinfo\") pod \"89326bce-8281-4747-aa3f-c1d37c615234\" (UID: \"89326bce-8281-4747-aa3f-c1d37c615234\") "
Oct 30 08:02:22.776844 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:22.776841    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89326bce-8281-4747-aa3f-c1d37c615234-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89326bce-8281-4747-aa3f-c1d37c615234-istio-data\") pod \"89326bce-8281-4747-aa3f-c1d37c615234\" (UID: \"89326bce-8281-4747-aa3f-c1d37c615234\") "
Oct 30 08:02:22.776987 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:22.776889    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89326bce-8281-4747-aa3f-c1d37c615234-kube-api-access-9rqh2\" (UniqueName: \"kubernetes.io/projected/89326bce-8281-4747-aa3f-c1d37c615234-kube-api-access-9rqh2\") pod \"89326bce-8281-4747-aa3f-c1d37c615234\" (UID: \"89326bce-8281-4747-aa3f-c1d37c615234\") "
Oct 30 08:02:22.776987 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:22.776923    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89326bce-8281-4747-aa3f-c1d37c615234-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89326bce-8281-4747-aa3f-c1d37c615234-credential-socket\") pod \"89326bce-8281-4747-aa3f-c1d37c615234\" (UID: \"89326bce-8281-4747-aa3f-c1d37c615234\") "
Oct 30 08:02:22.776987 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:22.776976    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89326bce-8281-4747-aa3f-c1d37c615234-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89326bce-8281-4747-aa3f-c1d37c615234-istiod-ca-cert\") pod \"89326bce-8281-4747-aa3f-c1d37c615234\" (UID: \"89326bce-8281-4747-aa3f-c1d37c615234\") "
Oct 30 08:02:22.777108 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:22.776887    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89326bce-8281-4747-aa3f-c1d37c615234-workload-socket" pod "89326bce-8281-4747-aa3f-c1d37c615234" (UID: "89326bce-8281-4747-aa3f-c1d37c615234"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:02:22.777108 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:22.777092    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89326bce-8281-4747-aa3f-c1d37c615234-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89326bce-8281-4747-aa3f-c1d37c615234-workload-certs\") pod \"89326bce-8281-4747-aa3f-c1d37c615234\" (UID: \"89326bce-8281-4747-aa3f-c1d37c615234\") "
Oct 30 08:02:22.777227 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:22.777104    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89326bce-8281-4747-aa3f-c1d37c615234-istio-data" pod "89326bce-8281-4747-aa3f-c1d37c615234" (UID: "89326bce-8281-4747-aa3f-c1d37c615234"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:02:22.777227 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:22.777172    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89326bce-8281-4747-aa3f-c1d37c615234-istio-token\" (UniqueName: \"kubernetes.io/projected/89326bce-8281-4747-aa3f-c1d37c615234-istio-token\") pod \"89326bce-8281-4747-aa3f-c1d37c615234\" (UID: \"89326bce-8281-4747-aa3f-c1d37c615234\") "
Oct 30 08:02:22.777377 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:22.777351    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89326bce-8281-4747-aa3f-c1d37c615234-workload-certs" pod "89326bce-8281-4747-aa3f-c1d37c615234" (UID: "89326bce-8281-4747-aa3f-c1d37c615234"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:02:22.777441 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:22.777379    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89326bce-8281-4747-aa3f-c1d37c615234-istiod-ca-cert" pod "89326bce-8281-4747-aa3f-c1d37c615234" (UID: "89326bce-8281-4747-aa3f-c1d37c615234"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:02:22.777712 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:22.777667    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89326bce-8281-4747-aa3f-c1d37c615234-credential-socket" pod "89326bce-8281-4747-aa3f-c1d37c615234" (UID: "89326bce-8281-4747-aa3f-c1d37c615234"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:02:22.777842 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:22.777790    6224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89326bce-8281-4747-aa3f-c1d37c615234-istio-data\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:02:22.777918 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:22.777845    6224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89326bce-8281-4747-aa3f-c1d37c615234-credential-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:02:22.777918 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:22.777860    6224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89326bce-8281-4747-aa3f-c1d37c615234-istiod-ca-cert\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:02:22.777918 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:22.777874    6224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89326bce-8281-4747-aa3f-c1d37c615234-workload-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:02:22.777918 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:22.777890    6224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89326bce-8281-4747-aa3f-c1d37c615234-workload-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:02:22.780840 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:22.779430    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89326bce-8281-4747-aa3f-c1d37c615234-kube-api-access-9rqh2" pod "89326bce-8281-4747-aa3f-c1d37c615234" (UID: "89326bce-8281-4747-aa3f-c1d37c615234"). InnerVolumeSpecName "kube-api-access-9rqh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:02:22.780840 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:22.779481    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89326bce-8281-4747-aa3f-c1d37c615234-istio-envoy" pod "89326bce-8281-4747-aa3f-c1d37c615234" (UID: "89326bce-8281-4747-aa3f-c1d37c615234"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:02:22.780840 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:22.779570    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89326bce-8281-4747-aa3f-c1d37c615234-istio-podinfo" pod "89326bce-8281-4747-aa3f-c1d37c615234" (UID: "89326bce-8281-4747-aa3f-c1d37c615234"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 08:02:22.780840 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:22.779696    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89326bce-8281-4747-aa3f-c1d37c615234-istio-token" pod "89326bce-8281-4747-aa3f-c1d37c615234" (UID: "89326bce-8281-4747-aa3f-c1d37c615234"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:02:22.878801 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:22.878769    6224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89326bce-8281-4747-aa3f-c1d37c615234-istio-envoy\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:02:22.878801 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:22.878798    6224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89326bce-8281-4747-aa3f-c1d37c615234-istio-podinfo\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:02:22.878801 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:22.878822    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rqh2\" (UniqueName: \"kubernetes.io/projected/89326bce-8281-4747-aa3f-c1d37c615234-kube-api-access-9rqh2\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:02:22.879053 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:22.878834    6224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89326bce-8281-4747-aa3f-c1d37c615234-istio-token\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:02:22.982047 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:22.981948    6224 generic.go:356] "Generic (PLEG): container finished" podID="89326bce-8281-4747-aa3f-c1d37c615234" containerID="72cd4cbefbae77bd6a6bdd3ce4ee12ffabdc8d855eaebfa80a3ba7bca32fd5a7" exitCode=0
Oct 30 08:02:22.982047 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:22.981998    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf"
Oct 30 08:02:22.982047 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:22.982032    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf" event={"ID":"89326bce-8281-4747-aa3f-c1d37c615234","Type":"ContainerDied","Data":"72cd4cbefbae77bd6a6bdd3ce4ee12ffabdc8d855eaebfa80a3ba7bca32fd5a7"}
Oct 30 08:02:22.982339 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:22.982095    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf" event={"ID":"89326bce-8281-4747-aa3f-c1d37c615234","Type":"ContainerDied","Data":"89c6d5643fc0e465fee073874ef2e0236df3fa2465dcba3d516bd03d8e0940f8"}
Oct 30 08:02:22.982339 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:22.982120    6224 scope.go:122] "RemoveContainer" containerID="72cd4cbefbae77bd6a6bdd3ce4ee12ffabdc8d855eaebfa80a3ba7bca32fd5a7"
Oct 30 08:02:22.996621 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:22.996602    6224 scope.go:122] "RemoveContainer" containerID="f83e71bb373bc44313b4a523c72c94150cc2ef2427a9d57a48c1de416ed47e29"
Oct 30 08:02:23.009038 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:23.009019    6224 scope.go:122] "RemoveContainer" containerID="72cd4cbefbae77bd6a6bdd3ce4ee12ffabdc8d855eaebfa80a3ba7bca32fd5a7"
Oct 30 08:02:23.009397 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:02:23.009370    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72cd4cbefbae77bd6a6bdd3ce4ee12ffabdc8d855eaebfa80a3ba7bca32fd5a7\": container with ID starting with 72cd4cbefbae77bd6a6bdd3ce4ee12ffabdc8d855eaebfa80a3ba7bca32fd5a7 not found: ID does not exist" containerID="72cd4cbefbae77bd6a6bdd3ce4ee12ffabdc8d855eaebfa80a3ba7bca32fd5a7"
Oct 30 08:02:23.009531 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:23.009410    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72cd4cbefbae77bd6a6bdd3ce4ee12ffabdc8d855eaebfa80a3ba7bca32fd5a7"} err="failed to get container status \"72cd4cbefbae77bd6a6bdd3ce4ee12ffabdc8d855eaebfa80a3ba7bca32fd5a7\": rpc error: code = NotFound desc = could not find container \"72cd4cbefbae77bd6a6bdd3ce4ee12ffabdc8d855eaebfa80a3ba7bca32fd5a7\": container with ID starting with 72cd4cbefbae77bd6a6bdd3ce4ee12ffabdc8d855eaebfa80a3ba7bca32fd5a7 not found: ID does not exist"
Oct 30 08:02:23.009531 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:23.009436    6224 scope.go:122] "RemoveContainer" containerID="f83e71bb373bc44313b4a523c72c94150cc2ef2427a9d57a48c1de416ed47e29"
Oct 30 08:02:23.010195 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:02:23.010168    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f83e71bb373bc44313b4a523c72c94150cc2ef2427a9d57a48c1de416ed47e29\": container with ID starting with f83e71bb373bc44313b4a523c72c94150cc2ef2427a9d57a48c1de416ed47e29 not found: ID does not exist" containerID="f83e71bb373bc44313b4a523c72c94150cc2ef2427a9d57a48c1de416ed47e29"
Oct 30 08:02:23.010315 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:23.010199    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f83e71bb373bc44313b4a523c72c94150cc2ef2427a9d57a48c1de416ed47e29"} err="failed to get container status \"f83e71bb373bc44313b4a523c72c94150cc2ef2427a9d57a48c1de416ed47e29\": rpc error: code = NotFound desc = could not find container \"f83e71bb373bc44313b4a523c72c94150cc2ef2427a9d57a48c1de416ed47e29\": container with ID starting with f83e71bb373bc44313b4a523c72c94150cc2ef2427a9d57a48c1de416ed47e29 not found: ID does not exist"
Oct 30 08:02:23.013887 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:02:23.013863    6224 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="get container status: runtime container status: rpc error: code = NotFound desc = could not find container \"f83e71bb373bc44313b4a523c72c94150cc2ef2427a9d57a48c1de416ed47e29\": container with ID starting with f83e71bb373bc44313b4a523c72c94150cc2ef2427a9d57a48c1de416ed47e29 not found: ID does not exist"
Oct 30 08:02:23.053341 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:23.053311    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf"]
Oct 30 08:02:23.071166 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:23.071134    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-9bec5c-openshift-default-5bf79bcf45-8tmtf"]
Oct 30 08:02:23.506578 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:23.506542    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89326bce-8281-4747-aa3f-c1d37c615234" path="/var/lib/kubelet/pods/89326bce-8281-4747-aa3f-c1d37c615234/volumes"
Oct 30 08:02:26.964167 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:26.964123    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-f5ceca/rate-test-model-f5ceca-kserve-845bf4bfdf-ck76g"
Oct 30 08:02:26.977304 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:26.977270    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-f5ceca/rate-test-model-f5ceca-kserve-845bf4bfdf-ck76g"
Oct 30 08:02:27.021623 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:27.021593    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-f5ceca/rate-test-model-f5ceca-kserve-845bf4bfdf-ck76g"
Oct 30 08:02:28.419037 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:28.418990    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c1b5d7-6cb579c4-9h8vp"]
Oct 30 08:02:28.452217 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:28.452179    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c1b5d7-677bbc4b95-vlwb7"]
Oct 30 08:02:30.054182 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:30.054151    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-265384-7f894cf8b5-wccfq"]
Oct 30 08:02:30.055157 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:30.054368    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-265384-7f894cf8b5-wccfq" podUID="dda66230-cf6c-4f92-a50a-2edf07a1b0b8" containerName="maas-api" containerID="cri-o://b0f51c3703bb788b066ccf04d98c0c6bb7c65aac6432d2165cf0f3ac6c6cafa4" gracePeriod=30
Oct 30 08:02:30.392034 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:30.392002    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-265384-7f894cf8b5-wccfq"
Oct 30 08:02:30.577861 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:30.577819    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dda66230-cf6c-4f92-a50a-2edf07a1b0b8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dda66230-cf6c-4f92-a50a-2edf07a1b0b8-openshift-service-ca\") pod \"dda66230-cf6c-4f92-a50a-2edf07a1b0b8\" (UID: \"dda66230-cf6c-4f92-a50a-2edf07a1b0b8\") "
Oct 30 08:02:30.578046 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:30.577886    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dda66230-cf6c-4f92-a50a-2edf07a1b0b8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dda66230-cf6c-4f92-a50a-2edf07a1b0b8-metrics-certs\") pod \"dda66230-cf6c-4f92-a50a-2edf07a1b0b8\" (UID: \"dda66230-cf6c-4f92-a50a-2edf07a1b0b8\") "
Oct 30 08:02:30.578046 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:30.577951    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dda66230-cf6c-4f92-a50a-2edf07a1b0b8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dda66230-cf6c-4f92-a50a-2edf07a1b0b8-maas-api-tls\") pod \"dda66230-cf6c-4f92-a50a-2edf07a1b0b8\" (UID: \"dda66230-cf6c-4f92-a50a-2edf07a1b0b8\") "
Oct 30 08:02:30.578151 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:30.578047    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dda66230-cf6c-4f92-a50a-2edf07a1b0b8-kube-api-access-tb47v\" (UniqueName: \"kubernetes.io/projected/dda66230-cf6c-4f92-a50a-2edf07a1b0b8-kube-api-access-tb47v\") pod \"dda66230-cf6c-4f92-a50a-2edf07a1b0b8\" (UID: \"dda66230-cf6c-4f92-a50a-2edf07a1b0b8\") "
Oct 30 08:02:30.578257 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:30.578229    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dda66230-cf6c-4f92-a50a-2edf07a1b0b8-openshift-service-ca" pod "dda66230-cf6c-4f92-a50a-2edf07a1b0b8" (UID: "dda66230-cf6c-4f92-a50a-2edf07a1b0b8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:02:30.578549 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:30.578526    6224 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dda66230-cf6c-4f92-a50a-2edf07a1b0b8-openshift-service-ca\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:02:30.580149 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:30.580114    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dda66230-cf6c-4f92-a50a-2edf07a1b0b8-metrics-certs" pod "dda66230-cf6c-4f92-a50a-2edf07a1b0b8" (UID: "dda66230-cf6c-4f92-a50a-2edf07a1b0b8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:02:30.580249 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:30.580220    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dda66230-cf6c-4f92-a50a-2edf07a1b0b8-kube-api-access-tb47v" pod "dda66230-cf6c-4f92-a50a-2edf07a1b0b8" (UID: "dda66230-cf6c-4f92-a50a-2edf07a1b0b8"). InnerVolumeSpecName "kube-api-access-tb47v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:02:30.580300 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:30.580261    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dda66230-cf6c-4f92-a50a-2edf07a1b0b8-maas-api-tls" pod "dda66230-cf6c-4f92-a50a-2edf07a1b0b8" (UID: "dda66230-cf6c-4f92-a50a-2edf07a1b0b8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:02:30.687375 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:30.680898    6224 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dda66230-cf6c-4f92-a50a-2edf07a1b0b8-metrics-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:02:30.687375 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:30.680942    6224 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dda66230-cf6c-4f92-a50a-2edf07a1b0b8-maas-api-tls\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:02:30.687375 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:30.680956    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tb47v\" (UniqueName: \"kubernetes.io/projected/dda66230-cf6c-4f92-a50a-2edf07a1b0b8-kube-api-access-tb47v\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:02:31.041755 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:31.041720    6224 generic.go:356] "Generic (PLEG): container finished" podID="dda66230-cf6c-4f92-a50a-2edf07a1b0b8" containerID="b0f51c3703bb788b066ccf04d98c0c6bb7c65aac6432d2165cf0f3ac6c6cafa4" exitCode=0
Oct 30 08:02:31.041995 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:31.041785    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-265384-7f894cf8b5-wccfq"
Oct 30 08:02:31.041995 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:31.041795    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-265384-7f894cf8b5-wccfq" event={"ID":"dda66230-cf6c-4f92-a50a-2edf07a1b0b8","Type":"ContainerDied","Data":"b0f51c3703bb788b066ccf04d98c0c6bb7c65aac6432d2165cf0f3ac6c6cafa4"}
Oct 30 08:02:31.041995 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:31.041914    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-265384-7f894cf8b5-wccfq" event={"ID":"dda66230-cf6c-4f92-a50a-2edf07a1b0b8","Type":"ContainerDied","Data":"f139f30555d1ffeb17b7e3410b582089a33a71b102c8914b371c9953740aad49"}
Oct 30 08:02:31.041995 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:31.041945    6224 scope.go:122] "RemoveContainer" containerID="b0f51c3703bb788b066ccf04d98c0c6bb7c65aac6432d2165cf0f3ac6c6cafa4"
Oct 30 08:02:31.053063 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:31.053048    6224 scope.go:122] "RemoveContainer" containerID="b0f51c3703bb788b066ccf04d98c0c6bb7c65aac6432d2165cf0f3ac6c6cafa4"
Oct 30 08:02:31.053350 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:02:31.053333    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0f51c3703bb788b066ccf04d98c0c6bb7c65aac6432d2165cf0f3ac6c6cafa4\": container with ID starting with b0f51c3703bb788b066ccf04d98c0c6bb7c65aac6432d2165cf0f3ac6c6cafa4 not found: ID does not exist" containerID="b0f51c3703bb788b066ccf04d98c0c6bb7c65aac6432d2165cf0f3ac6c6cafa4"
Oct 30 08:02:31.053402 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:31.053358    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0f51c3703bb788b066ccf04d98c0c6bb7c65aac6432d2165cf0f3ac6c6cafa4"} err="failed to get container status \"b0f51c3703bb788b066ccf04d98c0c6bb7c65aac6432d2165cf0f3ac6c6cafa4\": rpc error: code = NotFound desc = could not find container \"b0f51c3703bb788b066ccf04d98c0c6bb7c65aac6432d2165cf0f3ac6c6cafa4\": container with ID starting with b0f51c3703bb788b066ccf04d98c0c6bb7c65aac6432d2165cf0f3ac6c6cafa4 not found: ID does not exist"
Oct 30 08:02:31.099433 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:31.099405    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-265384-7f894cf8b5-wccfq"]
Oct 30 08:02:31.113576 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:31.113551    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-265384-7f894cf8b5-wccfq"]
Oct 30 08:02:31.504265 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:31.504197    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dda66230-cf6c-4f92-a50a-2edf07a1b0b8" path="/var/lib/kubelet/pods/dda66230-cf6c-4f92-a50a-2edf07a1b0b8/volumes"
Oct 30 08:02:31.873257 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:02:31.873193    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 08:02:33.420111 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:33.420071    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-c1b5d7-6cb579c4-9h8vp" podUID="1313ec66-e8a0-4845-9181-cf9fe5b2d8c9" containerName="payload-processing" containerID="cri-o://4aaf9f666652e5731d15fa960fdc01b22c7ecea4c6a2f18aac1ee473d8690d1a" gracePeriod=25
Oct 30 08:02:33.453686 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:33.453633    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c1b5d7-677bbc4b95-vlwb7" podUID="ff56f991-7e93-4d06-8d87-c42cb72465e3" containerName="payload-pre-processing" containerID="cri-o://f77e0f6b2929f081d3d174b5dbb4ba39ee8f761c12c8aaf16bc41d13541891d5" gracePeriod=25
Oct 30 08:02:37.644159 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:37.644120    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-2bab4c/distinct-w4-2bab4c-kserve-6868d767c9-j7wlm"]
Oct 30 08:02:37.660343 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:37.660312    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-2bab4c/premium-w4-2bab4c-kserve-665887fc68-c6dgf"]
Oct 30 08:02:37.670228 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:37.670198    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-2bab4c/simulator-w4-2bab4c-kserve-6db9444549-7nrk8"]
Oct 30 08:02:39.372465 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:39.372426    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-c1b5d7-6cb579c4-9h8vp" podUID="1313ec66-e8a0-4845-9181-cf9fe5b2d8c9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:02:39.372906 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:39.372426    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c1b5d7-677bbc4b95-vlwb7" podUID="ff56f991-7e93-4d06-8d87-c42cb72465e3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:02:39.438470 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:39.438433    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9"]
Oct 30 08:02:39.438971 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:39.438935    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9" podUID="a3d25e61-890d-45bd-957a-6690c5f26a08" containerName="istio-proxy" containerID="cri-o://7c12c72888c54b6f84c4bd64842231316058fc7987d9af92fb1f91c229c4ddaa" gracePeriod=30
Oct 30 08:02:43.593252 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:43.593214    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-2bab4c-56fcccb486-2wtg7"]
Oct 30 08:02:44.713944 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:44.713875    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9"
Oct 30 08:02:44.851752 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:44.851718    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3d25e61-890d-45bd-957a-6690c5f26a08-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3d25e61-890d-45bd-957a-6690c5f26a08-workload-socket\") pod \"a3d25e61-890d-45bd-957a-6690c5f26a08\" (UID: \"a3d25e61-890d-45bd-957a-6690c5f26a08\") "
Oct 30 08:02:44.851752 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:44.851765    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3d25e61-890d-45bd-957a-6690c5f26a08-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3d25e61-890d-45bd-957a-6690c5f26a08-credential-socket\") pod \"a3d25e61-890d-45bd-957a-6690c5f26a08\" (UID: \"a3d25e61-890d-45bd-957a-6690c5f26a08\") "
Oct 30 08:02:44.852042 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:44.851789    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3d25e61-890d-45bd-957a-6690c5f26a08-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3d25e61-890d-45bd-957a-6690c5f26a08-istio-podinfo\") pod \"a3d25e61-890d-45bd-957a-6690c5f26a08\" (UID: \"a3d25e61-890d-45bd-957a-6690c5f26a08\") "
Oct 30 08:02:44.852042 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:44.851915    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3d25e61-890d-45bd-957a-6690c5f26a08-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3d25e61-890d-45bd-957a-6690c5f26a08-workload-certs\") pod \"a3d25e61-890d-45bd-957a-6690c5f26a08\" (UID: \"a3d25e61-890d-45bd-957a-6690c5f26a08\") "
Oct 30 08:02:44.852042 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:44.851950    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3d25e61-890d-45bd-957a-6690c5f26a08-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3d25e61-890d-45bd-957a-6690c5f26a08-istio-envoy\") pod \"a3d25e61-890d-45bd-957a-6690c5f26a08\" (UID: \"a3d25e61-890d-45bd-957a-6690c5f26a08\") "
Oct 30 08:02:44.852042 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:44.852010    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3d25e61-890d-45bd-957a-6690c5f26a08-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3d25e61-890d-45bd-957a-6690c5f26a08-istiod-ca-cert\") pod \"a3d25e61-890d-45bd-957a-6690c5f26a08\" (UID: \"a3d25e61-890d-45bd-957a-6690c5f26a08\") "
Oct 30 08:02:44.852263 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:44.852046    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3d25e61-890d-45bd-957a-6690c5f26a08-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3d25e61-890d-45bd-957a-6690c5f26a08-istio-data\") pod \"a3d25e61-890d-45bd-957a-6690c5f26a08\" (UID: \"a3d25e61-890d-45bd-957a-6690c5f26a08\") "
Oct 30 08:02:44.852263 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:44.852080    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3d25e61-890d-45bd-957a-6690c5f26a08-kube-api-access-g5rn5\" (UniqueName: \"kubernetes.io/projected/a3d25e61-890d-45bd-957a-6690c5f26a08-kube-api-access-g5rn5\") pod \"a3d25e61-890d-45bd-957a-6690c5f26a08\" (UID: \"a3d25e61-890d-45bd-957a-6690c5f26a08\") "
Oct 30 08:02:44.852263 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:44.852099    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3d25e61-890d-45bd-957a-6690c5f26a08-credential-socket" pod "a3d25e61-890d-45bd-957a-6690c5f26a08" (UID: "a3d25e61-890d-45bd-957a-6690c5f26a08"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:02:44.852263 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:44.852118    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3d25e61-890d-45bd-957a-6690c5f26a08-workload-socket" pod "a3d25e61-890d-45bd-957a-6690c5f26a08" (UID: "a3d25e61-890d-45bd-957a-6690c5f26a08"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:02:44.852263 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:44.852125    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3d25e61-890d-45bd-957a-6690c5f26a08-istio-token\" (UniqueName: \"kubernetes.io/projected/a3d25e61-890d-45bd-957a-6690c5f26a08-istio-token\") pod \"a3d25e61-890d-45bd-957a-6690c5f26a08\" (UID: \"a3d25e61-890d-45bd-957a-6690c5f26a08\") "
Oct 30 08:02:44.852263 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:44.852252    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3d25e61-890d-45bd-957a-6690c5f26a08-workload-certs" pod "a3d25e61-890d-45bd-957a-6690c5f26a08" (UID: "a3d25e61-890d-45bd-957a-6690c5f26a08"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:02:44.852750 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:44.852379    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3d25e61-890d-45bd-957a-6690c5f26a08-istio-data" pod "a3d25e61-890d-45bd-957a-6690c5f26a08" (UID: "a3d25e61-890d-45bd-957a-6690c5f26a08"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:02:44.852750 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:44.852505    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3d25e61-890d-45bd-957a-6690c5f26a08-istiod-ca-cert" pod "a3d25e61-890d-45bd-957a-6690c5f26a08" (UID: "a3d25e61-890d-45bd-957a-6690c5f26a08"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:02:44.852877 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:44.852751    6224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3d25e61-890d-45bd-957a-6690c5f26a08-workload-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:02:44.852877 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:44.852767    6224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3d25e61-890d-45bd-957a-6690c5f26a08-istiod-ca-cert\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:02:44.852877 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:44.852776    6224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3d25e61-890d-45bd-957a-6690c5f26a08-istio-data\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:02:44.852877 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:44.852788    6224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3d25e61-890d-45bd-957a-6690c5f26a08-workload-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:02:44.852877 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:44.852797    6224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3d25e61-890d-45bd-957a-6690c5f26a08-credential-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:02:44.854194 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:44.854163    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3d25e61-890d-45bd-957a-6690c5f26a08-istio-podinfo" pod "a3d25e61-890d-45bd-957a-6690c5f26a08" (UID: "a3d25e61-890d-45bd-957a-6690c5f26a08"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 08:02:44.854508 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:44.854489    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3d25e61-890d-45bd-957a-6690c5f26a08-kube-api-access-g5rn5" pod "a3d25e61-890d-45bd-957a-6690c5f26a08" (UID: "a3d25e61-890d-45bd-957a-6690c5f26a08"). InnerVolumeSpecName "kube-api-access-g5rn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:02:44.854647 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:44.854627    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3d25e61-890d-45bd-957a-6690c5f26a08-istio-envoy" pod "a3d25e61-890d-45bd-957a-6690c5f26a08" (UID: "a3d25e61-890d-45bd-957a-6690c5f26a08"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:02:44.854694 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:44.854633    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3d25e61-890d-45bd-957a-6690c5f26a08-istio-token" pod "a3d25e61-890d-45bd-957a-6690c5f26a08" (UID: "a3d25e61-890d-45bd-957a-6690c5f26a08"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:02:44.954110 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:44.954078    6224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3d25e61-890d-45bd-957a-6690c5f26a08-istio-podinfo\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:02:44.954110 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:44.954104    6224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3d25e61-890d-45bd-957a-6690c5f26a08-istio-envoy\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:02:44.954110 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:44.954113    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5rn5\" (UniqueName: \"kubernetes.io/projected/a3d25e61-890d-45bd-957a-6690c5f26a08-kube-api-access-g5rn5\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:02:44.954338 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:44.954124    6224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3d25e61-890d-45bd-957a-6690c5f26a08-istio-token\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:02:45.137518 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:45.137474    6224 generic.go:356] "Generic (PLEG): container finished" podID="a3d25e61-890d-45bd-957a-6690c5f26a08" containerID="7c12c72888c54b6f84c4bd64842231316058fc7987d9af92fb1f91c229c4ddaa" exitCode=0
Oct 30 08:02:45.137715 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:45.137541    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9"
Oct 30 08:02:45.137715 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:45.137545    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9" event={"ID":"a3d25e61-890d-45bd-957a-6690c5f26a08","Type":"ContainerDied","Data":"7c12c72888c54b6f84c4bd64842231316058fc7987d9af92fb1f91c229c4ddaa"}
Oct 30 08:02:45.137715 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:45.137641    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9" event={"ID":"a3d25e61-890d-45bd-957a-6690c5f26a08","Type":"ContainerDied","Data":"7c53bed9ecdf33a3285a4579ee9714c6bba180040e46f2cfae17060395e6b115"}
Oct 30 08:02:45.137715 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:45.137674    6224 scope.go:122] "RemoveContainer" containerID="7c12c72888c54b6f84c4bd64842231316058fc7987d9af92fb1f91c229c4ddaa"
Oct 30 08:02:45.148487 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:45.148472    6224 scope.go:122] "RemoveContainer" containerID="7c12c72888c54b6f84c4bd64842231316058fc7987d9af92fb1f91c229c4ddaa"
Oct 30 08:02:45.148748 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:02:45.148730    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c12c72888c54b6f84c4bd64842231316058fc7987d9af92fb1f91c229c4ddaa\": container with ID starting with 7c12c72888c54b6f84c4bd64842231316058fc7987d9af92fb1f91c229c4ddaa not found: ID does not exist" containerID="7c12c72888c54b6f84c4bd64842231316058fc7987d9af92fb1f91c229c4ddaa"
Oct 30 08:02:45.148848 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:45.148760    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c12c72888c54b6f84c4bd64842231316058fc7987d9af92fb1f91c229c4ddaa"} err="failed to get container status \"7c12c72888c54b6f84c4bd64842231316058fc7987d9af92fb1f91c229c4ddaa\": rpc error: code = NotFound desc = could not find container \"7c12c72888c54b6f84c4bd64842231316058fc7987d9af92fb1f91c229c4ddaa\": container with ID starting with 7c12c72888c54b6f84c4bd64842231316058fc7987d9af92fb1f91c229c4ddaa not found: ID does not exist"
Oct 30 08:02:45.196792 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:45.196764    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9"]
Oct 30 08:02:45.210890 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:45.210863    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-265384-openshift-default-6f9c67dbfc-669j9"]
Oct 30 08:02:45.502828 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:45.502741    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3d25e61-890d-45bd-957a-6690c5f26a08" path="/var/lib/kubelet/pods/a3d25e61-890d-45bd-957a-6690c5f26a08/volumes"
Oct 30 08:02:48.594542 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:48.594473    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-2bab4c-56fcccb486-2wtg7" podUID="1f058cca-8314-4ff2-aa1f-3169f7d770fa" containerName="payload-processing" containerID="cri-o://e04889e3495adfb22152b033b96fa006a7ba5ad401b5bafdfa44d7730f33f9ee" gracePeriod=25
Oct 30 08:02:48.872049 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:48.872022    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-2bab4c-56fcccb486-2wtg7"
Oct 30 08:02:49.001112 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:49.001077    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f058cca-8314-4ff2-aa1f-3169f7d770fa-kube-api-access-r8dcq\" (UniqueName: \"kubernetes.io/projected/1f058cca-8314-4ff2-aa1f-3169f7d770fa-kube-api-access-r8dcq\") pod \"1f058cca-8314-4ff2-aa1f-3169f7d770fa\" (UID: \"1f058cca-8314-4ff2-aa1f-3169f7d770fa\") "
Oct 30 08:02:49.001303 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:49.001197    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f058cca-8314-4ff2-aa1f-3169f7d770fa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f058cca-8314-4ff2-aa1f-3169f7d770fa-plugins-config-volume\") pod \"1f058cca-8314-4ff2-aa1f-3169f7d770fa\" (UID: \"1f058cca-8314-4ff2-aa1f-3169f7d770fa\") "
Oct 30 08:02:49.001587 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:49.001562    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f058cca-8314-4ff2-aa1f-3169f7d770fa-plugins-config-volume" pod "1f058cca-8314-4ff2-aa1f-3169f7d770fa" (UID: "1f058cca-8314-4ff2-aa1f-3169f7d770fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:02:49.003213 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:49.003193    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f058cca-8314-4ff2-aa1f-3169f7d770fa-kube-api-access-r8dcq" pod "1f058cca-8314-4ff2-aa1f-3169f7d770fa" (UID: "1f058cca-8314-4ff2-aa1f-3169f7d770fa"). InnerVolumeSpecName "kube-api-access-r8dcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:02:49.102756 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:49.102693    6224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f058cca-8314-4ff2-aa1f-3169f7d770fa-plugins-config-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:02:49.102756 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:49.102721    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8dcq\" (UniqueName: \"kubernetes.io/projected/1f058cca-8314-4ff2-aa1f-3169f7d770fa-kube-api-access-r8dcq\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:02:49.164442 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:49.164414    6224 generic.go:356] "Generic (PLEG): container finished" podID="1f058cca-8314-4ff2-aa1f-3169f7d770fa" containerID="e04889e3495adfb22152b033b96fa006a7ba5ad401b5bafdfa44d7730f33f9ee" exitCode=0
Oct 30 08:02:49.164576 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:49.164461    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-2bab4c-56fcccb486-2wtg7"
Oct 30 08:02:49.164576 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:49.164503    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-2bab4c-56fcccb486-2wtg7" event={"ID":"1f058cca-8314-4ff2-aa1f-3169f7d770fa","Type":"ContainerDied","Data":"e04889e3495adfb22152b033b96fa006a7ba5ad401b5bafdfa44d7730f33f9ee"}
Oct 30 08:02:49.164576 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:49.164561    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-2bab4c-56fcccb486-2wtg7" event={"ID":"1f058cca-8314-4ff2-aa1f-3169f7d770fa","Type":"ContainerDied","Data":"4d5ef980043d360149ac5018defd22a061352eb92e72377da09e7a7b6da23855"}
Oct 30 08:02:49.164705 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:49.164591    6224 scope.go:122] "RemoveContainer" containerID="e04889e3495adfb22152b033b96fa006a7ba5ad401b5bafdfa44d7730f33f9ee"
Oct 30 08:02:49.175328 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:49.175311    6224 scope.go:122] "RemoveContainer" containerID="e04889e3495adfb22152b033b96fa006a7ba5ad401b5bafdfa44d7730f33f9ee"
Oct 30 08:02:49.175596 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:02:49.175575    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e04889e3495adfb22152b033b96fa006a7ba5ad401b5bafdfa44d7730f33f9ee\": container with ID starting with e04889e3495adfb22152b033b96fa006a7ba5ad401b5bafdfa44d7730f33f9ee not found: ID does not exist" containerID="e04889e3495adfb22152b033b96fa006a7ba5ad401b5bafdfa44d7730f33f9ee"
Oct 30 08:02:49.175646 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:49.175604    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e04889e3495adfb22152b033b96fa006a7ba5ad401b5bafdfa44d7730f33f9ee"} err="failed to get container status \"e04889e3495adfb22152b033b96fa006a7ba5ad401b5bafdfa44d7730f33f9ee\": rpc error: code = NotFound desc = could not find container \"e04889e3495adfb22152b033b96fa006a7ba5ad401b5bafdfa44d7730f33f9ee\": container with ID starting with e04889e3495adfb22152b033b96fa006a7ba5ad401b5bafdfa44d7730f33f9ee not found: ID does not exist"
Oct 30 08:02:49.223036 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:49.223014    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-2bab4c-56fcccb486-2wtg7"]
Oct 30 08:02:49.237204 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:49.237182    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-2bab4c-56fcccb486-2wtg7"]
Oct 30 08:02:49.298452 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:49.298415    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c1b5d7-666cdbf94c-kgdnv"]
Oct 30 08:02:49.298721 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:49.298694    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c1b5d7-666cdbf94c-kgdnv" podUID="4444b346-84b4-4ac2-b445-0b43ec4d6fda" containerName="maas-api" containerID="cri-o://b44fa25c51d0540cb14035f5750c6c2b54b46351097175bd776336a1471720e5" gracePeriod=30
Oct 30 08:02:49.371775 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:49.371686    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c1b5d7-677bbc4b95-vlwb7" podUID="ff56f991-7e93-4d06-8d87-c42cb72465e3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:02:49.371963 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:49.371897    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-c1b5d7-6cb579c4-9h8vp" podUID="1313ec66-e8a0-4845-9181-cf9fe5b2d8c9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:02:49.506467 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:49.506432    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f058cca-8314-4ff2-aa1f-3169f7d770fa" path="/var/lib/kubelet/pods/1f058cca-8314-4ff2-aa1f-3169f7d770fa/volumes"
Oct 30 08:02:49.554078 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:49.554050    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c1b5d7-666cdbf94c-kgdnv"
Oct 30 08:02:49.712143 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:49.712062    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4444b346-84b4-4ac2-b445-0b43ec4d6fda-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4444b346-84b4-4ac2-b445-0b43ec4d6fda-maas-api-tls\") pod \"4444b346-84b4-4ac2-b445-0b43ec4d6fda\" (UID: \"4444b346-84b4-4ac2-b445-0b43ec4d6fda\") "
Oct 30 08:02:49.712143 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:49.712132    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4444b346-84b4-4ac2-b445-0b43ec4d6fda-kube-api-access-x46cm\" (UniqueName: \"kubernetes.io/projected/4444b346-84b4-4ac2-b445-0b43ec4d6fda-kube-api-access-x46cm\") pod \"4444b346-84b4-4ac2-b445-0b43ec4d6fda\" (UID: \"4444b346-84b4-4ac2-b445-0b43ec4d6fda\") "
Oct 30 08:02:49.712705 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:49.712172    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4444b346-84b4-4ac2-b445-0b43ec4d6fda-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4444b346-84b4-4ac2-b445-0b43ec4d6fda-openshift-service-ca\") pod \"4444b346-84b4-4ac2-b445-0b43ec4d6fda\" (UID: \"4444b346-84b4-4ac2-b445-0b43ec4d6fda\") "
Oct 30 08:02:49.712705 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:49.712268    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4444b346-84b4-4ac2-b445-0b43ec4d6fda-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4444b346-84b4-4ac2-b445-0b43ec4d6fda-metrics-certs\") pod \"4444b346-84b4-4ac2-b445-0b43ec4d6fda\" (UID: \"4444b346-84b4-4ac2-b445-0b43ec4d6fda\") "
Oct 30 08:02:49.712705 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:49.712625    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4444b346-84b4-4ac2-b445-0b43ec4d6fda-openshift-service-ca" pod "4444b346-84b4-4ac2-b445-0b43ec4d6fda" (UID: "4444b346-84b4-4ac2-b445-0b43ec4d6fda"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:02:49.714576 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:49.714543    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4444b346-84b4-4ac2-b445-0b43ec4d6fda-metrics-certs" pod "4444b346-84b4-4ac2-b445-0b43ec4d6fda" (UID: "4444b346-84b4-4ac2-b445-0b43ec4d6fda"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:02:49.714783 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:49.714759    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4444b346-84b4-4ac2-b445-0b43ec4d6fda-maas-api-tls" pod "4444b346-84b4-4ac2-b445-0b43ec4d6fda" (UID: "4444b346-84b4-4ac2-b445-0b43ec4d6fda"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:02:49.714892 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:49.714784    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4444b346-84b4-4ac2-b445-0b43ec4d6fda-kube-api-access-x46cm" pod "4444b346-84b4-4ac2-b445-0b43ec4d6fda" (UID: "4444b346-84b4-4ac2-b445-0b43ec4d6fda"). InnerVolumeSpecName "kube-api-access-x46cm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:02:49.814336 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:49.814300    6224 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4444b346-84b4-4ac2-b445-0b43ec4d6fda-metrics-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:02:49.814336 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:49.814330    6224 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4444b346-84b4-4ac2-b445-0b43ec4d6fda-maas-api-tls\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:02:49.814336 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:49.814340    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x46cm\" (UniqueName: \"kubernetes.io/projected/4444b346-84b4-4ac2-b445-0b43ec4d6fda-kube-api-access-x46cm\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:02:49.814577 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:49.814348    6224 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4444b346-84b4-4ac2-b445-0b43ec4d6fda-openshift-service-ca\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:02:50.173194 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:50.173153    6224 generic.go:356] "Generic (PLEG): container finished" podID="4444b346-84b4-4ac2-b445-0b43ec4d6fda" containerID="b44fa25c51d0540cb14035f5750c6c2b54b46351097175bd776336a1471720e5" exitCode=0
Oct 30 08:02:50.173194 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:50.173198    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c1b5d7-666cdbf94c-kgdnv"
Oct 30 08:02:50.173431 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:50.173217    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c1b5d7-666cdbf94c-kgdnv" event={"ID":"4444b346-84b4-4ac2-b445-0b43ec4d6fda","Type":"ContainerDied","Data":"b44fa25c51d0540cb14035f5750c6c2b54b46351097175bd776336a1471720e5"}
Oct 30 08:02:50.173431 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:50.173272    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c1b5d7-666cdbf94c-kgdnv" event={"ID":"4444b346-84b4-4ac2-b445-0b43ec4d6fda","Type":"ContainerDied","Data":"54836bac72de78c116ed0feff61a9be7c8c64a85e1437da1cba560419cd10c3c"}
Oct 30 08:02:50.173431 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:50.173293    6224 scope.go:122] "RemoveContainer" containerID="b44fa25c51d0540cb14035f5750c6c2b54b46351097175bd776336a1471720e5"
Oct 30 08:02:50.184015 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:50.183993    6224 scope.go:122] "RemoveContainer" containerID="b44fa25c51d0540cb14035f5750c6c2b54b46351097175bd776336a1471720e5"
Oct 30 08:02:50.184311 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:02:50.184284    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b44fa25c51d0540cb14035f5750c6c2b54b46351097175bd776336a1471720e5\": container with ID starting with b44fa25c51d0540cb14035f5750c6c2b54b46351097175bd776336a1471720e5 not found: ID does not exist" containerID="b44fa25c51d0540cb14035f5750c6c2b54b46351097175bd776336a1471720e5"
Oct 30 08:02:50.184424 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:50.184320    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b44fa25c51d0540cb14035f5750c6c2b54b46351097175bd776336a1471720e5"} err="failed to get container status \"b44fa25c51d0540cb14035f5750c6c2b54b46351097175bd776336a1471720e5\": rpc error: code = NotFound desc = could not find container \"b44fa25c51d0540cb14035f5750c6c2b54b46351097175bd776336a1471720e5\": container with ID starting with b44fa25c51d0540cb14035f5750c6c2b54b46351097175bd776336a1471720e5 not found: ID does not exist"
Oct 30 08:02:50.235751 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:50.234832    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c1b5d7-666cdbf94c-kgdnv"]
Oct 30 08:02:50.248289 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:50.248256    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c1b5d7-666cdbf94c-kgdnv"]
Oct 30 08:02:50.688120 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:50.688082    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-06e19a-579597b4c-wlbjb"]
Oct 30 08:02:51.504849 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:51.504797    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4444b346-84b4-4ac2-b445-0b43ec4d6fda" path="/var/lib/kubelet/pods/4444b346-84b4-4ac2-b445-0b43ec4d6fda/volumes"
Oct 30 08:02:52.667885 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:52.667848    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-2bab4c/distinct-w4-2bab4c-kserve-6868d767c9-j7wlm" podUID="11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00" containerName="main" containerID="cri-o://8fb2e28712f58d1bccf7d6e1685849c57c8975d11b5199fed4fe3ab2bb128e58" gracePeriod=45
Oct 30 08:02:52.681638 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:52.681604    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-2bab4c/premium-w4-2bab4c-kserve-665887fc68-c6dgf" podUID="f9463476-fcf5-4d66-bfd2-b85ce4ed651f" containerName="main" containerID="cri-o://6cb9fbd119431132e53c8a595bdba5447d6cad23fc4ec2e667058fecbec124a5" gracePeriod=45
Oct 30 08:02:52.696165 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:52.696135    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-2bab4c/simulator-w4-2bab4c-kserve-6db9444549-7nrk8" podUID="c7b09178-9ac7-492f-890b-5ff3da130981" containerName="main" containerID="cri-o://cf869217a4c504119fbe41e156c00e8d160a9434870ff7842b165aa5285dff99" gracePeriod=45
Oct 30 08:02:52.768186 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:52.768158    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2bab4c/simulator-w4-2bab4c-kserve-6db9444549-7nrk8" podUID="c7b09178-9ac7-492f-890b-5ff3da130981" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": dial tcp 10.134.0.46:8000: connect: connection refused"
Oct 30 08:02:52.856091 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:52.856055    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2bab4c/distinct-w4-2bab4c-kserve-6868d767c9-j7wlm" podUID="11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": dial tcp 10.134.0.52:8000: connect: connection refused"
Oct 30 08:02:52.967356 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:52.967330    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2bab4c/premium-w4-2bab4c-kserve-665887fc68-c6dgf"
Oct 30 08:02:53.003589 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.003565    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2bab4c/simulator-w4-2bab4c-kserve-6db9444549-7nrk8"
Oct 30 08:02:53.008165 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.008146    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2bab4c/distinct-w4-2bab4c-kserve-6868d767c9-j7wlm"
Oct 30 08:02:53.047968 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.047940    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9463476-fcf5-4d66-bfd2-b85ce4ed651f-kube-api-access-j2bg2\" (UniqueName: \"kubernetes.io/projected/f9463476-fcf5-4d66-bfd2-b85ce4ed651f-kube-api-access-j2bg2\") pod \"f9463476-fcf5-4d66-bfd2-b85ce4ed651f\" (UID: \"f9463476-fcf5-4d66-bfd2-b85ce4ed651f\") "
Oct 30 08:02:53.048123 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.047982    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9463476-fcf5-4d66-bfd2-b85ce4ed651f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9463476-fcf5-4d66-bfd2-b85ce4ed651f-model-cache\") pod \"f9463476-fcf5-4d66-bfd2-b85ce4ed651f\" (UID: \"f9463476-fcf5-4d66-bfd2-b85ce4ed651f\") "
Oct 30 08:02:53.048123 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.048060    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9463476-fcf5-4d66-bfd2-b85ce4ed651f-home\" (UniqueName: \"kubernetes.io/empty-dir/f9463476-fcf5-4d66-bfd2-b85ce4ed651f-home\") pod \"f9463476-fcf5-4d66-bfd2-b85ce4ed651f\" (UID: \"f9463476-fcf5-4d66-bfd2-b85ce4ed651f\") "
Oct 30 08:02:53.048239 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.048174    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9463476-fcf5-4d66-bfd2-b85ce4ed651f-tls-certs\" (UniqueName: \"kubernetes.io/secret/f9463476-fcf5-4d66-bfd2-b85ce4ed651f-tls-certs\") pod \"f9463476-fcf5-4d66-bfd2-b85ce4ed651f\" (UID: \"f9463476-fcf5-4d66-bfd2-b85ce4ed651f\") "
Oct 30 08:02:53.048299 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.048277    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9463476-fcf5-4d66-bfd2-b85ce4ed651f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9463476-fcf5-4d66-bfd2-b85ce4ed651f-tmp-dir\") pod \"f9463476-fcf5-4d66-bfd2-b85ce4ed651f\" (UID: \"f9463476-fcf5-4d66-bfd2-b85ce4ed651f\") "
Oct 30 08:02:53.048360 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.048312    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9463476-fcf5-4d66-bfd2-b85ce4ed651f-model-cache" pod "f9463476-fcf5-4d66-bfd2-b85ce4ed651f" (UID: "f9463476-fcf5-4d66-bfd2-b85ce4ed651f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:02:53.048360 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.048334    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9463476-fcf5-4d66-bfd2-b85ce4ed651f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9463476-fcf5-4d66-bfd2-b85ce4ed651f-dshm\") pod \"f9463476-fcf5-4d66-bfd2-b85ce4ed651f\" (UID: \"f9463476-fcf5-4d66-bfd2-b85ce4ed651f\") "
Oct 30 08:02:53.048360 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.048320    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9463476-fcf5-4d66-bfd2-b85ce4ed651f-home" pod "f9463476-fcf5-4d66-bfd2-b85ce4ed651f" (UID: "f9463476-fcf5-4d66-bfd2-b85ce4ed651f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:02:53.048570 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.048545    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9463476-fcf5-4d66-bfd2-b85ce4ed651f-tmp-dir" pod "f9463476-fcf5-4d66-bfd2-b85ce4ed651f" (UID: "f9463476-fcf5-4d66-bfd2-b85ce4ed651f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:02:53.048827 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.048792    6224 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9463476-fcf5-4d66-bfd2-b85ce4ed651f-tmp-dir\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:02:53.048827 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.048829    6224 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9463476-fcf5-4d66-bfd2-b85ce4ed651f-model-cache\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:02:53.048991 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.048838    6224 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9463476-fcf5-4d66-bfd2-b85ce4ed651f-home\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:02:53.050321 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.050291    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9463476-fcf5-4d66-bfd2-b85ce4ed651f-tls-certs" pod "f9463476-fcf5-4d66-bfd2-b85ce4ed651f" (UID: "f9463476-fcf5-4d66-bfd2-b85ce4ed651f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:02:53.050321 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.050305    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9463476-fcf5-4d66-bfd2-b85ce4ed651f-kube-api-access-j2bg2" pod "f9463476-fcf5-4d66-bfd2-b85ce4ed651f" (UID: "f9463476-fcf5-4d66-bfd2-b85ce4ed651f"). InnerVolumeSpecName "kube-api-access-j2bg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:02:53.050644 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.050628    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9463476-fcf5-4d66-bfd2-b85ce4ed651f-dshm" pod "f9463476-fcf5-4d66-bfd2-b85ce4ed651f" (UID: "f9463476-fcf5-4d66-bfd2-b85ce4ed651f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:02:53.149367 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.149330    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7b09178-9ac7-492f-890b-5ff3da130981-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7b09178-9ac7-492f-890b-5ff3da130981-model-cache\") pod \"c7b09178-9ac7-492f-890b-5ff3da130981\" (UID: \"c7b09178-9ac7-492f-890b-5ff3da130981\") "
Oct 30 08:02:53.149367 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.149365    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00-home\" (UniqueName: \"kubernetes.io/empty-dir/11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00-home\") pod \"11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00\" (UID: \"11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00\") "
Oct 30 08:02:53.149597 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.149445    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00-kube-api-access-j4sdz\" (UniqueName: \"kubernetes.io/projected/11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00-kube-api-access-j4sdz\") pod \"11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00\" (UID: \"11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00\") "
Oct 30 08:02:53.149597 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.149488    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00-tls-certs\" (UniqueName: \"kubernetes.io/secret/11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00-tls-certs\") pod \"11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00\" (UID: \"11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00\") "
Oct 30 08:02:53.149597 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.149523    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7b09178-9ac7-492f-890b-5ff3da130981-tls-certs\" (UniqueName: \"kubernetes.io/secret/c7b09178-9ac7-492f-890b-5ff3da130981-tls-certs\") pod \"c7b09178-9ac7-492f-890b-5ff3da130981\" (UID: \"c7b09178-9ac7-492f-890b-5ff3da130981\") "
Oct 30 08:02:53.149597 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.149574    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7b09178-9ac7-492f-890b-5ff3da130981-kube-api-access-gsxdh\" (UniqueName: \"kubernetes.io/projected/c7b09178-9ac7-492f-890b-5ff3da130981-kube-api-access-gsxdh\") pod \"c7b09178-9ac7-492f-890b-5ff3da130981\" (UID: \"c7b09178-9ac7-492f-890b-5ff3da130981\") "
Oct 30 08:02:53.149830 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.149628    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b09178-9ac7-492f-890b-5ff3da130981-model-cache" pod "c7b09178-9ac7-492f-890b-5ff3da130981" (UID: "c7b09178-9ac7-492f-890b-5ff3da130981"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:02:53.149830 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.149630    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00-dshm\" (UniqueName: \"kubernetes.io/empty-dir/11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00-dshm\") pod \"11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00\" (UID: \"11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00\") "
Oct 30 08:02:53.149830 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.149664    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00-home" pod "11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00" (UID: "11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:02:53.149830 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.149821    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00-model-cache\") pod \"11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00\" (UID: \"11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00\") "
Oct 30 08:02:53.150064 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.149883    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7b09178-9ac7-492f-890b-5ff3da130981-home\" (UniqueName: \"kubernetes.io/empty-dir/c7b09178-9ac7-492f-890b-5ff3da130981-home\") pod \"c7b09178-9ac7-492f-890b-5ff3da130981\" (UID: \"c7b09178-9ac7-492f-890b-5ff3da130981\") "
Oct 30 08:02:53.150064 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.149916    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00-tmp-dir\") pod \"11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00\" (UID: \"11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00\") "
Oct 30 08:02:53.150064 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.149987    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7b09178-9ac7-492f-890b-5ff3da130981-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7b09178-9ac7-492f-890b-5ff3da130981-dshm\") pod \"c7b09178-9ac7-492f-890b-5ff3da130981\" (UID: \"c7b09178-9ac7-492f-890b-5ff3da130981\") "
Oct 30 08:02:53.150064 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.150016    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7b09178-9ac7-492f-890b-5ff3da130981-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7b09178-9ac7-492f-890b-5ff3da130981-tmp-dir\") pod \"c7b09178-9ac7-492f-890b-5ff3da130981\" (UID: \"c7b09178-9ac7-492f-890b-5ff3da130981\") "
Oct 30 08:02:53.150283 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.150194    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b09178-9ac7-492f-890b-5ff3da130981-home" pod "c7b09178-9ac7-492f-890b-5ff3da130981" (UID: "c7b09178-9ac7-492f-890b-5ff3da130981"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:02:53.150626 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.150503    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b09178-9ac7-492f-890b-5ff3da130981-tmp-dir" pod "c7b09178-9ac7-492f-890b-5ff3da130981" (UID: "c7b09178-9ac7-492f-890b-5ff3da130981"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:02:53.150626 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.150563    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00-tmp-dir" pod "11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00" (UID: "11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:02:53.150626 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.150587    6224 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9463476-fcf5-4d66-bfd2-b85ce4ed651f-dshm\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:02:53.150626 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.150627    6224 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7b09178-9ac7-492f-890b-5ff3da130981-home\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:02:53.150994 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.150643    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2bg2\" (UniqueName: \"kubernetes.io/projected/f9463476-fcf5-4d66-bfd2-b85ce4ed651f-kube-api-access-j2bg2\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:02:53.150994 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.150659    6224 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7b09178-9ac7-492f-890b-5ff3da130981-model-cache\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:02:53.150994 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.150673    6224 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00-home\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:02:53.150994 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.150686    6224 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9463476-fcf5-4d66-bfd2-b85ce4ed651f-tls-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:02:53.150994 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.150632    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00-model-cache" pod "11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00" (UID: "11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:02:53.152154 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.152128    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7b09178-9ac7-492f-890b-5ff3da130981-tls-certs" pod "c7b09178-9ac7-492f-890b-5ff3da130981" (UID: "c7b09178-9ac7-492f-890b-5ff3da130981"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:02:53.152284 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.152272    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00-dshm" pod "11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00" (UID: "11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:02:53.152357 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.152326    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00-kube-api-access-j4sdz" pod "11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00" (UID: "11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00"). InnerVolumeSpecName "kube-api-access-j4sdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:02:53.152357 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.152335    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00-tls-certs" pod "11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00" (UID: "11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:02:53.152592 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.152573    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b09178-9ac7-492f-890b-5ff3da130981-dshm" pod "c7b09178-9ac7-492f-890b-5ff3da130981" (UID: "c7b09178-9ac7-492f-890b-5ff3da130981"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:02:53.153079 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.153060    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7b09178-9ac7-492f-890b-5ff3da130981-kube-api-access-gsxdh" pod "c7b09178-9ac7-492f-890b-5ff3da130981" (UID: "c7b09178-9ac7-492f-890b-5ff3da130981"). InnerVolumeSpecName "kube-api-access-gsxdh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:02:53.198986 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.196445    6224 generic.go:356] "Generic (PLEG): container finished" podID="f9463476-fcf5-4d66-bfd2-b85ce4ed651f" containerID="6cb9fbd119431132e53c8a595bdba5447d6cad23fc4ec2e667058fecbec124a5" exitCode=0
Oct 30 08:02:53.198986 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.196575    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2bab4c/premium-w4-2bab4c-kserve-665887fc68-c6dgf" event={"ID":"f9463476-fcf5-4d66-bfd2-b85ce4ed651f","Type":"ContainerDied","Data":"6cb9fbd119431132e53c8a595bdba5447d6cad23fc4ec2e667058fecbec124a5"}
Oct 30 08:02:53.198986 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.196613    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2bab4c/premium-w4-2bab4c-kserve-665887fc68-c6dgf" event={"ID":"f9463476-fcf5-4d66-bfd2-b85ce4ed651f","Type":"ContainerDied","Data":"b5655b958bde911e3d0417332d55188aa28181bcdd3eaaa914592156ddd92063"}
Oct 30 08:02:53.198986 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.196636    6224 scope.go:122] "RemoveContainer" containerID="6cb9fbd119431132e53c8a595bdba5447d6cad23fc4ec2e667058fecbec124a5"
Oct 30 08:02:53.198986 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.196778    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2bab4c/premium-w4-2bab4c-kserve-665887fc68-c6dgf"
Oct 30 08:02:53.201220 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.201192    6224 generic.go:356] "Generic (PLEG): container finished" podID="11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00" containerID="8fb2e28712f58d1bccf7d6e1685849c57c8975d11b5199fed4fe3ab2bb128e58" exitCode=0
Oct 30 08:02:53.201379 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.201357    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2bab4c/distinct-w4-2bab4c-kserve-6868d767c9-j7wlm" event={"ID":"11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00","Type":"ContainerDied","Data":"8fb2e28712f58d1bccf7d6e1685849c57c8975d11b5199fed4fe3ab2bb128e58"}
Oct 30 08:02:53.201457 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.201397    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2bab4c/distinct-w4-2bab4c-kserve-6868d767c9-j7wlm" event={"ID":"11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00","Type":"ContainerDied","Data":"35b7528dc6840b47d4a32d2b951f3aa1c075441f376307817a4c8a3ace5c6739"}
Oct 30 08:02:53.201524 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.201466    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2bab4c/distinct-w4-2bab4c-kserve-6868d767c9-j7wlm"
Oct 30 08:02:53.216836 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.216797    6224 scope.go:122] "RemoveContainer" containerID="6cb9fbd119431132e53c8a595bdba5447d6cad23fc4ec2e667058fecbec124a5"
Oct 30 08:02:53.217389 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.217349    6224 generic.go:356] "Generic (PLEG): container finished" podID="c7b09178-9ac7-492f-890b-5ff3da130981" containerID="cf869217a4c504119fbe41e156c00e8d160a9434870ff7842b165aa5285dff99" exitCode=0
Oct 30 08:02:53.217602 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.217568    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2bab4c/simulator-w4-2bab4c-kserve-6db9444549-7nrk8" event={"ID":"c7b09178-9ac7-492f-890b-5ff3da130981","Type":"ContainerDied","Data":"cf869217a4c504119fbe41e156c00e8d160a9434870ff7842b165aa5285dff99"}
Oct 30 08:02:53.217757 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.217727    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2bab4c/simulator-w4-2bab4c-kserve-6db9444549-7nrk8" event={"ID":"c7b09178-9ac7-492f-890b-5ff3da130981","Type":"ContainerDied","Data":"f33e7b1af29f8b5999a9768c230a15273afce33e54b33039439ca5ddeecdd3f5"}
Oct 30 08:02:53.217946 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.217931    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2bab4c/simulator-w4-2bab4c-kserve-6db9444549-7nrk8"
Oct 30 08:02:53.219002 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:02:53.218961    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cb9fbd119431132e53c8a595bdba5447d6cad23fc4ec2e667058fecbec124a5\": container with ID starting with 6cb9fbd119431132e53c8a595bdba5447d6cad23fc4ec2e667058fecbec124a5 not found: ID does not exist" containerID="6cb9fbd119431132e53c8a595bdba5447d6cad23fc4ec2e667058fecbec124a5"
Oct 30 08:02:53.219099 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.219013    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cb9fbd119431132e53c8a595bdba5447d6cad23fc4ec2e667058fecbec124a5"} err="failed to get container status \"6cb9fbd119431132e53c8a595bdba5447d6cad23fc4ec2e667058fecbec124a5\": rpc error: code = NotFound desc = could not find container \"6cb9fbd119431132e53c8a595bdba5447d6cad23fc4ec2e667058fecbec124a5\": container with ID starting with 6cb9fbd119431132e53c8a595bdba5447d6cad23fc4ec2e667058fecbec124a5 not found: ID does not exist"
Oct 30 08:02:53.219099 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.219036    6224 scope.go:122] "RemoveContainer" containerID="8fb2e28712f58d1bccf7d6e1685849c57c8975d11b5199fed4fe3ab2bb128e58"
Oct 30 08:02:53.233309 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.233285    6224 scope.go:122] "RemoveContainer" containerID="8fb2e28712f58d1bccf7d6e1685849c57c8975d11b5199fed4fe3ab2bb128e58"
Oct 30 08:02:53.235903 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:02:53.234022    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fb2e28712f58d1bccf7d6e1685849c57c8975d11b5199fed4fe3ab2bb128e58\": container with ID starting with 8fb2e28712f58d1bccf7d6e1685849c57c8975d11b5199fed4fe3ab2bb128e58 not found: ID does not exist" containerID="8fb2e28712f58d1bccf7d6e1685849c57c8975d11b5199fed4fe3ab2bb128e58"
Oct 30 08:02:53.235903 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.234061    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fb2e28712f58d1bccf7d6e1685849c57c8975d11b5199fed4fe3ab2bb128e58"} err="failed to get container status \"8fb2e28712f58d1bccf7d6e1685849c57c8975d11b5199fed4fe3ab2bb128e58\": rpc error: code = NotFound desc = could not find container \"8fb2e28712f58d1bccf7d6e1685849c57c8975d11b5199fed4fe3ab2bb128e58\": container with ID starting with 8fb2e28712f58d1bccf7d6e1685849c57c8975d11b5199fed4fe3ab2bb128e58 not found: ID does not exist"
Oct 30 08:02:53.235903 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.234087    6224 scope.go:122] "RemoveContainer" containerID="cf869217a4c504119fbe41e156c00e8d160a9434870ff7842b165aa5285dff99"
Oct 30 08:02:53.246036 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.246016    6224 scope.go:122] "RemoveContainer" containerID="cf869217a4c504119fbe41e156c00e8d160a9434870ff7842b165aa5285dff99"
Oct 30 08:02:53.246317 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:02:53.246296    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf869217a4c504119fbe41e156c00e8d160a9434870ff7842b165aa5285dff99\": container with ID starting with cf869217a4c504119fbe41e156c00e8d160a9434870ff7842b165aa5285dff99 not found: ID does not exist" containerID="cf869217a4c504119fbe41e156c00e8d160a9434870ff7842b165aa5285dff99"
Oct 30 08:02:53.246373 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.246324    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf869217a4c504119fbe41e156c00e8d160a9434870ff7842b165aa5285dff99"} err="failed to get container status \"cf869217a4c504119fbe41e156c00e8d160a9434870ff7842b165aa5285dff99\": rpc error: code = NotFound desc = could not find container \"cf869217a4c504119fbe41e156c00e8d160a9434870ff7842b165aa5285dff99\": container with ID starting with cf869217a4c504119fbe41e156c00e8d160a9434870ff7842b165aa5285dff99 not found: ID does not exist"
Oct 30 08:02:53.252129 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.252100    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4sdz\" (UniqueName: \"kubernetes.io/projected/11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00-kube-api-access-j4sdz\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:02:53.252231 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.252135    6224 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00-tls-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:02:53.252231 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.252150    6224 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7b09178-9ac7-492f-890b-5ff3da130981-tls-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:02:53.252231 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.252163    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsxdh\" (UniqueName: \"kubernetes.io/projected/c7b09178-9ac7-492f-890b-5ff3da130981-kube-api-access-gsxdh\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:02:53.252231 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.252175    6224 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00-dshm\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:02:53.252231 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.252188    6224 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00-model-cache\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:02:53.252231 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.252202    6224 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00-tmp-dir\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:02:53.252231 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.252214    6224 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7b09178-9ac7-492f-890b-5ff3da130981-dshm\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:02:53.252231 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.252225    6224 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7b09178-9ac7-492f-890b-5ff3da130981-tmp-dir\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:02:53.280659 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.280632    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-2bab4c/premium-w4-2bab4c-kserve-665887fc68-c6dgf"]
Oct 30 08:02:53.286905 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.286882    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-2bab4c/premium-w4-2bab4c-kserve-665887fc68-c6dgf"]
Oct 30 08:02:53.355908 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.355882    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-2bab4c/distinct-w4-2bab4c-kserve-6868d767c9-j7wlm"]
Oct 30 08:02:53.359437 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.359414    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-2bab4c/distinct-w4-2bab4c-kserve-6868d767c9-j7wlm"]
Oct 30 08:02:53.398559 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.398528    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-2bab4c/simulator-w4-2bab4c-kserve-6db9444549-7nrk8"]
Oct 30 08:02:53.402160 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.402133    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-2bab4c/simulator-w4-2bab4c-kserve-6db9444549-7nrk8"]
Oct 30 08:02:53.504178 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.504145    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00" path="/var/lib/kubelet/pods/11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00/volumes"
Oct 30 08:02:53.504949 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.504925    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7b09178-9ac7-492f-890b-5ff3da130981" path="/var/lib/kubelet/pods/c7b09178-9ac7-492f-890b-5ff3da130981/volumes"
Oct 30 08:02:53.505610 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.505589    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9463476-fcf5-4d66-bfd2-b85ce4ed651f" path="/var/lib/kubelet/pods/f9463476-fcf5-4d66-bfd2-b85ce4ed651f/volumes"
Oct 30 08:02:53.722309 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.722286    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-c1b5d7-6cb579c4-9h8vp"
Oct 30 08:02:53.754963 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.754941    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c1b5d7-677bbc4b95-vlwb7"
Oct 30 08:02:53.859118 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.858512    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff56f991-7e93-4d06-8d87-c42cb72465e3-kube-api-access-t6knn\" (UniqueName: \"kubernetes.io/projected/ff56f991-7e93-4d06-8d87-c42cb72465e3-kube-api-access-t6knn\") pod \"ff56f991-7e93-4d06-8d87-c42cb72465e3\" (UID: \"ff56f991-7e93-4d06-8d87-c42cb72465e3\") "
Oct 30 08:02:53.859118 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.858560    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1313ec66-e8a0-4845-9181-cf9fe5b2d8c9-kube-api-access-6g764\" (UniqueName: \"kubernetes.io/projected/1313ec66-e8a0-4845-9181-cf9fe5b2d8c9-kube-api-access-6g764\") pod \"1313ec66-e8a0-4845-9181-cf9fe5b2d8c9\" (UID: \"1313ec66-e8a0-4845-9181-cf9fe5b2d8c9\") "
Oct 30 08:02:53.859118 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.858648    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff56f991-7e93-4d06-8d87-c42cb72465e3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff56f991-7e93-4d06-8d87-c42cb72465e3-plugins-config-volume\") pod \"ff56f991-7e93-4d06-8d87-c42cb72465e3\" (UID: \"ff56f991-7e93-4d06-8d87-c42cb72465e3\") "
Oct 30 08:02:53.859118 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.858698    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1313ec66-e8a0-4845-9181-cf9fe5b2d8c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1313ec66-e8a0-4845-9181-cf9fe5b2d8c9-plugins-config-volume\") pod \"1313ec66-e8a0-4845-9181-cf9fe5b2d8c9\" (UID: \"1313ec66-e8a0-4845-9181-cf9fe5b2d8c9\") "
Oct 30 08:02:53.859639 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.859608    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1313ec66-e8a0-4845-9181-cf9fe5b2d8c9-plugins-config-volume" pod "1313ec66-e8a0-4845-9181-cf9fe5b2d8c9" (UID: "1313ec66-e8a0-4845-9181-cf9fe5b2d8c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:02:53.859913 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.859880    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff56f991-7e93-4d06-8d87-c42cb72465e3-plugins-config-volume" pod "ff56f991-7e93-4d06-8d87-c42cb72465e3" (UID: "ff56f991-7e93-4d06-8d87-c42cb72465e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:02:53.862210 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.862186    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff56f991-7e93-4d06-8d87-c42cb72465e3-kube-api-access-t6knn" pod "ff56f991-7e93-4d06-8d87-c42cb72465e3" (UID: "ff56f991-7e93-4d06-8d87-c42cb72465e3"). InnerVolumeSpecName "kube-api-access-t6knn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:02:53.863534 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.863510    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1313ec66-e8a0-4845-9181-cf9fe5b2d8c9-kube-api-access-6g764" pod "1313ec66-e8a0-4845-9181-cf9fe5b2d8c9" (UID: "1313ec66-e8a0-4845-9181-cf9fe5b2d8c9"). InnerVolumeSpecName "kube-api-access-6g764". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:02:53.960132 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.960093    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6knn\" (UniqueName: \"kubernetes.io/projected/ff56f991-7e93-4d06-8d87-c42cb72465e3-kube-api-access-t6knn\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:02:53.960132 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.960123    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6g764\" (UniqueName: \"kubernetes.io/projected/1313ec66-e8a0-4845-9181-cf9fe5b2d8c9-kube-api-access-6g764\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:02:53.960132 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.960132    6224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff56f991-7e93-4d06-8d87-c42cb72465e3-plugins-config-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:02:53.960397 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:53.960140    6224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1313ec66-e8a0-4845-9181-cf9fe5b2d8c9-plugins-config-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:02:54.226776 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:54.226738    6224 generic.go:356] "Generic (PLEG): container finished" podID="1313ec66-e8a0-4845-9181-cf9fe5b2d8c9" containerID="4aaf9f666652e5731d15fa960fdc01b22c7ecea4c6a2f18aac1ee473d8690d1a" exitCode=0
Oct 30 08:02:54.226985 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:54.226781    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-c1b5d7-6cb579c4-9h8vp"
Oct 30 08:02:54.226985 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:54.226825    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-c1b5d7-6cb579c4-9h8vp" event={"ID":"1313ec66-e8a0-4845-9181-cf9fe5b2d8c9","Type":"ContainerDied","Data":"4aaf9f666652e5731d15fa960fdc01b22c7ecea4c6a2f18aac1ee473d8690d1a"}
Oct 30 08:02:54.226985 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:54.226881    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-c1b5d7-6cb579c4-9h8vp" event={"ID":"1313ec66-e8a0-4845-9181-cf9fe5b2d8c9","Type":"ContainerDied","Data":"4aef7ebf3a3cca83dab2158ce4df236aa752a1c7dc3c83712e9c2d63556e609b"}
Oct 30 08:02:54.226985 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:54.226911    6224 scope.go:122] "RemoveContainer" containerID="4aaf9f666652e5731d15fa960fdc01b22c7ecea4c6a2f18aac1ee473d8690d1a"
Oct 30 08:02:54.229168 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:54.229147    6224 generic.go:356] "Generic (PLEG): container finished" podID="ff56f991-7e93-4d06-8d87-c42cb72465e3" containerID="f77e0f6b2929f081d3d174b5dbb4ba39ee8f761c12c8aaf16bc41d13541891d5" exitCode=0
Oct 30 08:02:54.229282 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:54.229174    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c1b5d7-677bbc4b95-vlwb7" event={"ID":"ff56f991-7e93-4d06-8d87-c42cb72465e3","Type":"ContainerDied","Data":"f77e0f6b2929f081d3d174b5dbb4ba39ee8f761c12c8aaf16bc41d13541891d5"}
Oct 30 08:02:54.229282 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:54.229203    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c1b5d7-677bbc4b95-vlwb7" event={"ID":"ff56f991-7e93-4d06-8d87-c42cb72465e3","Type":"ContainerDied","Data":"e71089215cce4a8195082c169fedca3d453ad22f732c7251f47100e4999c0d2a"}
Oct 30 08:02:54.229282 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:54.229258    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c1b5d7-677bbc4b95-vlwb7"
Oct 30 08:02:54.237204 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:54.237175    6224 scope.go:122] "RemoveContainer" containerID="4aaf9f666652e5731d15fa960fdc01b22c7ecea4c6a2f18aac1ee473d8690d1a"
Oct 30 08:02:54.237600 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:02:54.237462    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4aaf9f666652e5731d15fa960fdc01b22c7ecea4c6a2f18aac1ee473d8690d1a\": container with ID starting with 4aaf9f666652e5731d15fa960fdc01b22c7ecea4c6a2f18aac1ee473d8690d1a not found: ID does not exist" containerID="4aaf9f666652e5731d15fa960fdc01b22c7ecea4c6a2f18aac1ee473d8690d1a"
Oct 30 08:02:54.237600 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:54.237493    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4aaf9f666652e5731d15fa960fdc01b22c7ecea4c6a2f18aac1ee473d8690d1a"} err="failed to get container status \"4aaf9f666652e5731d15fa960fdc01b22c7ecea4c6a2f18aac1ee473d8690d1a\": rpc error: code = NotFound desc = could not find container \"4aaf9f666652e5731d15fa960fdc01b22c7ecea4c6a2f18aac1ee473d8690d1a\": container with ID starting with 4aaf9f666652e5731d15fa960fdc01b22c7ecea4c6a2f18aac1ee473d8690d1a not found: ID does not exist"
Oct 30 08:02:54.237600 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:54.237516    6224 scope.go:122] "RemoveContainer" containerID="f77e0f6b2929f081d3d174b5dbb4ba39ee8f761c12c8aaf16bc41d13541891d5"
Oct 30 08:02:54.249060 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:54.249016    6224 scope.go:122] "RemoveContainer" containerID="f77e0f6b2929f081d3d174b5dbb4ba39ee8f761c12c8aaf16bc41d13541891d5"
Oct 30 08:02:54.249434 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:02:54.249399    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f77e0f6b2929f081d3d174b5dbb4ba39ee8f761c12c8aaf16bc41d13541891d5\": container with ID starting with f77e0f6b2929f081d3d174b5dbb4ba39ee8f761c12c8aaf16bc41d13541891d5 not found: ID does not exist" containerID="f77e0f6b2929f081d3d174b5dbb4ba39ee8f761c12c8aaf16bc41d13541891d5"
Oct 30 08:02:54.249553 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:54.249441    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f77e0f6b2929f081d3d174b5dbb4ba39ee8f761c12c8aaf16bc41d13541891d5"} err="failed to get container status \"f77e0f6b2929f081d3d174b5dbb4ba39ee8f761c12c8aaf16bc41d13541891d5\": rpc error: code = NotFound desc = could not find container \"f77e0f6b2929f081d3d174b5dbb4ba39ee8f761c12c8aaf16bc41d13541891d5\": container with ID starting with f77e0f6b2929f081d3d174b5dbb4ba39ee8f761c12c8aaf16bc41d13541891d5 not found: ID does not exist"
Oct 30 08:02:54.285552 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:54.285518    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c1b5d7-6cb579c4-9h8vp"]
Oct 30 08:02:54.291206 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:54.291166    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c1b5d7-6cb579c4-9h8vp"]
Oct 30 08:02:54.329577 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:54.329544    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c1b5d7-677bbc4b95-vlwb7"]
Oct 30 08:02:54.333142 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:54.333114    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c1b5d7-677bbc4b95-vlwb7"]
Oct 30 08:02:54.747207 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:54.747171    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 08:02:55.502558 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:55.502526    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1313ec66-e8a0-4845-9181-cf9fe5b2d8c9" path="/var/lib/kubelet/pods/1313ec66-e8a0-4845-9181-cf9fe5b2d8c9/volumes"
Oct 30 08:02:55.502970 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:55.502955    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff56f991-7e93-4d06-8d87-c42cb72465e3" path="/var/lib/kubelet/pods/ff56f991-7e93-4d06-8d87-c42cb72465e3/volumes"
Oct 30 08:02:55.688836 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:02:55.688606    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-06e19a-579597b4c-wlbjb" podUID="7d82945b-c43f-413a-a2a4-941c9be966c4" containerName="payload-pre-processing" containerID="cri-o://f589568d6251293ab9f4a737e50212d59e2801afb5f1d338f830bbf49c7981eb" gracePeriod=25
Oct 30 08:03:01.857553 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:03:01.857519    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 08:03:05.601184 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:03:05.601144    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-06e19a-579597b4c-wlbjb" podUID="7d82945b-c43f-413a-a2a4-941c9be966c4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:03:09.108118 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:03:09.108084    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-932c00-79d49cb7d7-mpkdp"]
Oct 30 08:03:14.109234 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:03:14.109192    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-932c00-79d49cb7d7-mpkdp" podUID="e2a703c4-5066-4875-be2d-5fc882b38502" containerName="payload-pre-processing" containerID="cri-o://d61135b21220c000ef954bbee94eeda14dd755a6da49dc0d668914a7524a638a" gracePeriod=25
Oct 30 08:03:15.600704 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:03:15.600664    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-06e19a-579597b4c-wlbjb" podUID="7d82945b-c43f-413a-a2a4-941c9be966c4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:03:15.950880 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:03:15.950852    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-06e19a-579597b4c-wlbjb"
Oct 30 08:03:15.997479 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:03:15.997444    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d82945b-c43f-413a-a2a4-941c9be966c4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d82945b-c43f-413a-a2a4-941c9be966c4-plugins-config-volume\") pod \"7d82945b-c43f-413a-a2a4-941c9be966c4\" (UID: \"7d82945b-c43f-413a-a2a4-941c9be966c4\") "
Oct 30 08:03:15.997739 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:03:15.997597    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d82945b-c43f-413a-a2a4-941c9be966c4-kube-api-access-lt76t\" (UniqueName: \"kubernetes.io/projected/7d82945b-c43f-413a-a2a4-941c9be966c4-kube-api-access-lt76t\") pod \"7d82945b-c43f-413a-a2a4-941c9be966c4\" (UID: \"7d82945b-c43f-413a-a2a4-941c9be966c4\") "
Oct 30 08:03:16.015651 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:03:15.998980    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d82945b-c43f-413a-a2a4-941c9be966c4-plugins-config-volume" pod "7d82945b-c43f-413a-a2a4-941c9be966c4" (UID: "7d82945b-c43f-413a-a2a4-941c9be966c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:03:16.015651 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:03:16.009161    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d82945b-c43f-413a-a2a4-941c9be966c4-kube-api-access-lt76t" pod "7d82945b-c43f-413a-a2a4-941c9be966c4" (UID: "7d82945b-c43f-413a-a2a4-941c9be966c4"). InnerVolumeSpecName "kube-api-access-lt76t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:03:16.019757 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:03:16.019458    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-932c00-79d49cb7d7-mpkdp" podUID="e2a703c4-5066-4875-be2d-5fc882b38502" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:03:16.100125 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:03:16.100088    6224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d82945b-c43f-413a-a2a4-941c9be966c4-plugins-config-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:03:16.100125 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:03:16.100119    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt76t\" (UniqueName: \"kubernetes.io/projected/7d82945b-c43f-413a-a2a4-941c9be966c4-kube-api-access-lt76t\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:03:16.372964 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:03:16.372924    6224 generic.go:356] "Generic (PLEG): container finished" podID="7d82945b-c43f-413a-a2a4-941c9be966c4" containerID="f589568d6251293ab9f4a737e50212d59e2801afb5f1d338f830bbf49c7981eb" exitCode=0
Oct 30 08:03:16.373191 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:03:16.373113    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-06e19a-579597b4c-wlbjb" event={"ID":"7d82945b-c43f-413a-a2a4-941c9be966c4","Type":"ContainerDied","Data":"f589568d6251293ab9f4a737e50212d59e2801afb5f1d338f830bbf49c7981eb"}
Oct 30 08:03:16.373191 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:03:16.373174    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-06e19a-579597b4c-wlbjb" event={"ID":"7d82945b-c43f-413a-a2a4-941c9be966c4","Type":"ContainerDied","Data":"f9fdfa3d32bf77d53e7edc13bca1d909fa8ba505eca069e51efb0122e8107160"}
Oct 30 08:03:16.373340 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:03:16.373204    6224 scope.go:122] "RemoveContainer" containerID="f589568d6251293ab9f4a737e50212d59e2801afb5f1d338f830bbf49c7981eb"
Oct 30 08:03:16.373398 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:03:16.373371    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-06e19a-579597b4c-wlbjb"
Oct 30 08:03:16.392145 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:03:16.392121    6224 scope.go:122] "RemoveContainer" containerID="f589568d6251293ab9f4a737e50212d59e2801afb5f1d338f830bbf49c7981eb"
Oct 30 08:03:16.392505 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:03:16.392469    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f589568d6251293ab9f4a737e50212d59e2801afb5f1d338f830bbf49c7981eb\": container with ID starting with f589568d6251293ab9f4a737e50212d59e2801afb5f1d338f830bbf49c7981eb not found: ID does not exist" containerID="f589568d6251293ab9f4a737e50212d59e2801afb5f1d338f830bbf49c7981eb"
Oct 30 08:03:16.392574 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:03:16.392518    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f589568d6251293ab9f4a737e50212d59e2801afb5f1d338f830bbf49c7981eb"} err="failed to get container status \"f589568d6251293ab9f4a737e50212d59e2801afb5f1d338f830bbf49c7981eb\": rpc error: code = NotFound desc = could not find container \"f589568d6251293ab9f4a737e50212d59e2801afb5f1d338f830bbf49c7981eb\": container with ID starting with f589568d6251293ab9f4a737e50212d59e2801afb5f1d338f830bbf49c7981eb not found: ID does not exist"
Oct 30 08:03:16.448134 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:03:16.448083    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-06e19a-579597b4c-wlbjb"]
Oct 30 08:03:16.453145 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:03:16.453108    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-06e19a-579597b4c-wlbjb"]
Oct 30 08:03:17.504035 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:03:17.503998    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d82945b-c43f-413a-a2a4-941c9be966c4" path="/var/lib/kubelet/pods/7d82945b-c43f-413a-a2a4-941c9be966c4/volumes"
Oct 30 08:03:26.018753 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:03:26.018704    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-932c00-79d49cb7d7-mpkdp" podUID="e2a703c4-5066-4875-be2d-5fc882b38502" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:03:31.856672 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:03:31.856627    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 08:03:34.369438 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:03:34.369384    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-932c00-79d49cb7d7-mpkdp"
Oct 30 08:03:34.404459 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:03:34.404432    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2a703c4-5066-4875-be2d-5fc882b38502-kube-api-access-9m44j\" (UniqueName: \"kubernetes.io/projected/e2a703c4-5066-4875-be2d-5fc882b38502-kube-api-access-9m44j\") pod \"e2a703c4-5066-4875-be2d-5fc882b38502\" (UID: \"e2a703c4-5066-4875-be2d-5fc882b38502\") "
Oct 30 08:03:34.406588 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:03:34.406558    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2a703c4-5066-4875-be2d-5fc882b38502-kube-api-access-9m44j" pod "e2a703c4-5066-4875-be2d-5fc882b38502" (UID: "e2a703c4-5066-4875-be2d-5fc882b38502"). InnerVolumeSpecName "kube-api-access-9m44j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:03:34.496243 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:03:34.496207    6224 generic.go:356] "Generic (PLEG): container finished" podID="e2a703c4-5066-4875-be2d-5fc882b38502" containerID="d61135b21220c000ef954bbee94eeda14dd755a6da49dc0d668914a7524a638a" exitCode=0
Oct 30 08:03:34.496243 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:03:34.496242    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-932c00-79d49cb7d7-mpkdp"
Oct 30 08:03:34.496477 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:03:34.496257    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-932c00-79d49cb7d7-mpkdp" event={"ID":"e2a703c4-5066-4875-be2d-5fc882b38502","Type":"ContainerDied","Data":"d61135b21220c000ef954bbee94eeda14dd755a6da49dc0d668914a7524a638a"}
Oct 30 08:03:34.496477 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:03:34.496301    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-932c00-79d49cb7d7-mpkdp" event={"ID":"e2a703c4-5066-4875-be2d-5fc882b38502","Type":"ContainerDied","Data":"d153ad1827db43dd101030b2a3b4b3551bd1709b0fc2cec7322eda07a9ff1af5"}
Oct 30 08:03:34.496477 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:03:34.496331    6224 scope.go:122] "RemoveContainer" containerID="d61135b21220c000ef954bbee94eeda14dd755a6da49dc0d668914a7524a638a"
Oct 30 08:03:34.505115 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:03:34.505086    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2a703c4-5066-4875-be2d-5fc882b38502-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2a703c4-5066-4875-be2d-5fc882b38502-plugins-config-volume\") pod \"e2a703c4-5066-4875-be2d-5fc882b38502\" (UID: \"e2a703c4-5066-4875-be2d-5fc882b38502\") "
Oct 30 08:03:34.505378 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:03:34.505347    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9m44j\" (UniqueName: \"kubernetes.io/projected/e2a703c4-5066-4875-be2d-5fc882b38502-kube-api-access-9m44j\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:03:34.505510 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:03:34.505472    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2a703c4-5066-4875-be2d-5fc882b38502-plugins-config-volume" pod "e2a703c4-5066-4875-be2d-5fc882b38502" (UID: "e2a703c4-5066-4875-be2d-5fc882b38502"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:03:34.506031 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:03:34.506015    6224 scope.go:122] "RemoveContainer" containerID="d61135b21220c000ef954bbee94eeda14dd755a6da49dc0d668914a7524a638a"
Oct 30 08:03:34.506336 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:03:34.506315    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d61135b21220c000ef954bbee94eeda14dd755a6da49dc0d668914a7524a638a\": container with ID starting with d61135b21220c000ef954bbee94eeda14dd755a6da49dc0d668914a7524a638a not found: ID does not exist" containerID="d61135b21220c000ef954bbee94eeda14dd755a6da49dc0d668914a7524a638a"
Oct 30 08:03:34.506389 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:03:34.506344    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d61135b21220c000ef954bbee94eeda14dd755a6da49dc0d668914a7524a638a"} err="failed to get container status \"d61135b21220c000ef954bbee94eeda14dd755a6da49dc0d668914a7524a638a\": rpc error: code = NotFound desc = could not find container \"d61135b21220c000ef954bbee94eeda14dd755a6da49dc0d668914a7524a638a\": container with ID starting with d61135b21220c000ef954bbee94eeda14dd755a6da49dc0d668914a7524a638a not found: ID does not exist"
Oct 30 08:03:34.606140 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:03:34.606111    6224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2a703c4-5066-4875-be2d-5fc882b38502-plugins-config-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:03:34.846959 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:03:34.846925    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-932c00-79d49cb7d7-mpkdp"]
Oct 30 08:03:34.861270 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:03:34.861240    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-932c00-79d49cb7d7-mpkdp"]
Oct 30 08:03:35.217615 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:03:35.217531    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-137122-646879997d-8z24f"]
Oct 30 08:03:35.256763 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:03:35.256732    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-137122-7f8b9567cf-jmkdv"]
Oct 30 08:03:35.505630 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:03:35.505544    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2a703c4-5066-4875-be2d-5fc882b38502" path="/var/lib/kubelet/pods/e2a703c4-5066-4875-be2d-5fc882b38502/volumes"
Oct 30 08:03:40.218159 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:03:40.218116    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-137122-646879997d-8z24f" podUID="30fa2cf6-c67b-4be5-ab71-df50207183c5" containerName="payload-processing" containerID="cri-o://97a9e8d097ab24b618858cb8dcf8492d636dcbba860450c0a260708a7596cfe4" gracePeriod=25
Oct 30 08:03:40.257912 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:03:40.257875    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-137122-7f8b9567cf-jmkdv" podUID="7b8c015e-9234-4ecb-a908-9ce7eb600db5" containerName="payload-pre-processing" containerID="cri-o://1c6d9ff5eb3c83e6257554d95af42851649952b47d29f1b18af3286bcedef025" gracePeriod=25
Oct 30 08:03:41.386151 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:03:41.386113    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 08:03:44.679025 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:03:44.678989    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-137122-646879997d-8z24f" podUID="30fa2cf6-c67b-4be5-ab71-df50207183c5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:03:45.691059 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:03:45.691023    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-137122-7f8b9567cf-jmkdv" podUID="7b8c015e-9234-4ecb-a908-9ce7eb600db5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:03:54.678768 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:03:54.678730    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-137122-646879997d-8z24f" podUID="30fa2cf6-c67b-4be5-ab71-df50207183c5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:03:55.691094 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:03:55.691047    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-137122-7f8b9567cf-jmkdv" podUID="7b8c015e-9234-4ecb-a908-9ce7eb600db5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:04:00.518900 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:00.518867    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-137122-646879997d-8z24f"
Oct 30 08:04:00.545118 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:00.545095    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-137122-7f8b9567cf-jmkdv"
Oct 30 08:04:00.652339 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:00.652306    6224 generic.go:356] "Generic (PLEG): container finished" podID="7b8c015e-9234-4ecb-a908-9ce7eb600db5" containerID="1c6d9ff5eb3c83e6257554d95af42851649952b47d29f1b18af3286bcedef025" exitCode=0
Oct 30 08:04:00.652531 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:00.652358    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-137122-7f8b9567cf-jmkdv"
Oct 30 08:04:00.652531 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:00.652393    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-137122-7f8b9567cf-jmkdv" event={"ID":"7b8c015e-9234-4ecb-a908-9ce7eb600db5","Type":"ContainerDied","Data":"1c6d9ff5eb3c83e6257554d95af42851649952b47d29f1b18af3286bcedef025"}
Oct 30 08:04:00.652531 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:00.652452    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-137122-7f8b9567cf-jmkdv" event={"ID":"7b8c015e-9234-4ecb-a908-9ce7eb600db5","Type":"ContainerDied","Data":"c44ef6663fafb44f9260275a1d3ad4636ff32e78eb481c5633c4248d48ac1972"}
Oct 30 08:04:00.652531 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:00.652480    6224 scope.go:122] "RemoveContainer" containerID="1c6d9ff5eb3c83e6257554d95af42851649952b47d29f1b18af3286bcedef025"
Oct 30 08:04:00.653951 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:00.653927    6224 generic.go:356] "Generic (PLEG): container finished" podID="30fa2cf6-c67b-4be5-ab71-df50207183c5" containerID="97a9e8d097ab24b618858cb8dcf8492d636dcbba860450c0a260708a7596cfe4" exitCode=0
Oct 30 08:04:00.654049 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:00.653976    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-137122-646879997d-8z24f"
Oct 30 08:04:00.654049 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:00.654014    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-137122-646879997d-8z24f" event={"ID":"30fa2cf6-c67b-4be5-ab71-df50207183c5","Type":"ContainerDied","Data":"97a9e8d097ab24b618858cb8dcf8492d636dcbba860450c0a260708a7596cfe4"}
Oct 30 08:04:00.654182 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:00.654058    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-137122-646879997d-8z24f" event={"ID":"30fa2cf6-c67b-4be5-ab71-df50207183c5","Type":"ContainerDied","Data":"7476c4bc695c8d5839c17e53470d909778247993aa94f1177cf343127f648548"}
Oct 30 08:04:00.662340 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:00.662319    6224 scope.go:122] "RemoveContainer" containerID="1c6d9ff5eb3c83e6257554d95af42851649952b47d29f1b18af3286bcedef025"
Oct 30 08:04:00.662651 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:04:00.662634    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c6d9ff5eb3c83e6257554d95af42851649952b47d29f1b18af3286bcedef025\": container with ID starting with 1c6d9ff5eb3c83e6257554d95af42851649952b47d29f1b18af3286bcedef025 not found: ID does not exist" containerID="1c6d9ff5eb3c83e6257554d95af42851649952b47d29f1b18af3286bcedef025"
Oct 30 08:04:00.662705 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:00.662660    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c6d9ff5eb3c83e6257554d95af42851649952b47d29f1b18af3286bcedef025"} err="failed to get container status \"1c6d9ff5eb3c83e6257554d95af42851649952b47d29f1b18af3286bcedef025\": rpc error: code = NotFound desc = could not find container \"1c6d9ff5eb3c83e6257554d95af42851649952b47d29f1b18af3286bcedef025\": container with ID starting with 1c6d9ff5eb3c83e6257554d95af42851649952b47d29f1b18af3286bcedef025 not found: ID does not exist"
Oct 30 08:04:00.662705 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:00.662678    6224 scope.go:122] "RemoveContainer" containerID="97a9e8d097ab24b618858cb8dcf8492d636dcbba860450c0a260708a7596cfe4"
Oct 30 08:04:00.670990 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:00.670975    6224 scope.go:122] "RemoveContainer" containerID="97a9e8d097ab24b618858cb8dcf8492d636dcbba860450c0a260708a7596cfe4"
Oct 30 08:04:00.671241 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:04:00.671225    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97a9e8d097ab24b618858cb8dcf8492d636dcbba860450c0a260708a7596cfe4\": container with ID starting with 97a9e8d097ab24b618858cb8dcf8492d636dcbba860450c0a260708a7596cfe4 not found: ID does not exist" containerID="97a9e8d097ab24b618858cb8dcf8492d636dcbba860450c0a260708a7596cfe4"
Oct 30 08:04:00.671292 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:00.671248    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97a9e8d097ab24b618858cb8dcf8492d636dcbba860450c0a260708a7596cfe4"} err="failed to get container status \"97a9e8d097ab24b618858cb8dcf8492d636dcbba860450c0a260708a7596cfe4\": rpc error: code = NotFound desc = could not find container \"97a9e8d097ab24b618858cb8dcf8492d636dcbba860450c0a260708a7596cfe4\": container with ID starting with 97a9e8d097ab24b618858cb8dcf8492d636dcbba860450c0a260708a7596cfe4 not found: ID does not exist"
Oct 30 08:04:00.682693 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:00.682673    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30fa2cf6-c67b-4be5-ab71-df50207183c5-kube-api-access-p9s5g\" (UniqueName: \"kubernetes.io/projected/30fa2cf6-c67b-4be5-ab71-df50207183c5-kube-api-access-p9s5g\") pod \"30fa2cf6-c67b-4be5-ab71-df50207183c5\" (UID: \"30fa2cf6-c67b-4be5-ab71-df50207183c5\") "
Oct 30 08:04:00.682782 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:00.682770    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b8c015e-9234-4ecb-a908-9ce7eb600db5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b8c015e-9234-4ecb-a908-9ce7eb600db5-plugins-config-volume\") pod \"7b8c015e-9234-4ecb-a908-9ce7eb600db5\" (UID: \"7b8c015e-9234-4ecb-a908-9ce7eb600db5\") "
Oct 30 08:04:00.682861 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:00.682845    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30fa2cf6-c67b-4be5-ab71-df50207183c5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30fa2cf6-c67b-4be5-ab71-df50207183c5-plugins-config-volume\") pod \"30fa2cf6-c67b-4be5-ab71-df50207183c5\" (UID: \"30fa2cf6-c67b-4be5-ab71-df50207183c5\") "
Oct 30 08:04:00.683010 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:00.682993    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b8c015e-9234-4ecb-a908-9ce7eb600db5-kube-api-access-c7sqd\" (UniqueName: \"kubernetes.io/projected/7b8c015e-9234-4ecb-a908-9ce7eb600db5-kube-api-access-c7sqd\") pod \"7b8c015e-9234-4ecb-a908-9ce7eb600db5\" (UID: \"7b8c015e-9234-4ecb-a908-9ce7eb600db5\") "
Oct 30 08:04:00.683174 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:00.683154    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b8c015e-9234-4ecb-a908-9ce7eb600db5-plugins-config-volume" pod "7b8c015e-9234-4ecb-a908-9ce7eb600db5" (UID: "7b8c015e-9234-4ecb-a908-9ce7eb600db5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:04:00.683241 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:00.683196    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30fa2cf6-c67b-4be5-ab71-df50207183c5-plugins-config-volume" pod "30fa2cf6-c67b-4be5-ab71-df50207183c5" (UID: "30fa2cf6-c67b-4be5-ab71-df50207183c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:04:00.683427 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:00.683409    6224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b8c015e-9234-4ecb-a908-9ce7eb600db5-plugins-config-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:04:00.683480 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:00.683432    6224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30fa2cf6-c67b-4be5-ab71-df50207183c5-plugins-config-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:04:00.684886 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:00.684860    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30fa2cf6-c67b-4be5-ab71-df50207183c5-kube-api-access-p9s5g" pod "30fa2cf6-c67b-4be5-ab71-df50207183c5" (UID: "30fa2cf6-c67b-4be5-ab71-df50207183c5"). InnerVolumeSpecName "kube-api-access-p9s5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:04:00.685015 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:00.684996    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b8c015e-9234-4ecb-a908-9ce7eb600db5-kube-api-access-c7sqd" pod "7b8c015e-9234-4ecb-a908-9ce7eb600db5" (UID: "7b8c015e-9234-4ecb-a908-9ce7eb600db5"). InnerVolumeSpecName "kube-api-access-c7sqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:04:00.785289 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:00.785250    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7sqd\" (UniqueName: \"kubernetes.io/projected/7b8c015e-9234-4ecb-a908-9ce7eb600db5-kube-api-access-c7sqd\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:04:00.785491 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:00.785348    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9s5g\" (UniqueName: \"kubernetes.io/projected/30fa2cf6-c67b-4be5-ab71-df50207183c5-kube-api-access-p9s5g\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:04:01.005940 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:01.005910    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-137122-646879997d-8z24f"]
Oct 30 08:04:01.020423 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:01.020393    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-137122-646879997d-8z24f"]
Oct 30 08:04:01.062713 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:01.062614    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-137122-7f8b9567cf-jmkdv"]
Oct 30 08:04:01.070251 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:01.070221    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-137122-7f8b9567cf-jmkdv"]
Oct 30 08:04:01.502822 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:01.502734    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30fa2cf6-c67b-4be5-ab71-df50207183c5" path="/var/lib/kubelet/pods/30fa2cf6-c67b-4be5-ab71-df50207183c5/volumes"
Oct 30 08:04:01.503173 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:01.503158    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b8c015e-9234-4ecb-a908-9ce7eb600db5" path="/var/lib/kubelet/pods/7b8c015e-9234-4ecb-a908-9ce7eb600db5/volumes"
Oct 30 08:04:01.856829 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:04:01.856768    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 08:04:02.004527 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:02.004492    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-bb6be4/premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t"]
Oct 30 08:04:02.020492 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:02.020460    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-bb6be4/simulator-w2-bb6be4-kserve-8485bbf954-dnbbz"]
Oct 30 08:04:10.273323 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:10.273288    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-4fe2ed-676bcd8cf6-pw5b5"]
Oct 30 08:04:10.274008 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:10.273533    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4fe2ed-676bcd8cf6-pw5b5" podUID="a4d66eef-bccf-4b9f-8c10-d4ac990c531f" containerName="maas-api" containerID="cri-o://7841af5f024f4c00960cd337b56179b015fffb0ea49405a0eba36bb07fa23523" gracePeriod=30
Oct 30 08:04:10.578215 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:10.578176    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4fe2ed-676bcd8cf6-pw5b5"
Oct 30 08:04:10.603612 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:10.603574    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4d66eef-bccf-4b9f-8c10-d4ac990c531f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4d66eef-bccf-4b9f-8c10-d4ac990c531f-maas-api-tls\") pod \"a4d66eef-bccf-4b9f-8c10-d4ac990c531f\" (UID: \"a4d66eef-bccf-4b9f-8c10-d4ac990c531f\") "
Oct 30 08:04:10.603870 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:10.603706    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4d66eef-bccf-4b9f-8c10-d4ac990c531f-kube-api-access-zc2nc\" (UniqueName: \"kubernetes.io/projected/a4d66eef-bccf-4b9f-8c10-d4ac990c531f-kube-api-access-zc2nc\") pod \"a4d66eef-bccf-4b9f-8c10-d4ac990c531f\" (UID: \"a4d66eef-bccf-4b9f-8c10-d4ac990c531f\") "
Oct 30 08:04:10.603870 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:10.603852    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4d66eef-bccf-4b9f-8c10-d4ac990c531f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4d66eef-bccf-4b9f-8c10-d4ac990c531f-metrics-certs\") pod \"a4d66eef-bccf-4b9f-8c10-d4ac990c531f\" (UID: \"a4d66eef-bccf-4b9f-8c10-d4ac990c531f\") "
Oct 30 08:04:10.604044 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:10.603927    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4d66eef-bccf-4b9f-8c10-d4ac990c531f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4d66eef-bccf-4b9f-8c10-d4ac990c531f-openshift-service-ca\") pod \"a4d66eef-bccf-4b9f-8c10-d4ac990c531f\" (UID: \"a4d66eef-bccf-4b9f-8c10-d4ac990c531f\") "
Oct 30 08:04:10.604632 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:10.604596    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4d66eef-bccf-4b9f-8c10-d4ac990c531f-openshift-service-ca" pod "a4d66eef-bccf-4b9f-8c10-d4ac990c531f" (UID: "a4d66eef-bccf-4b9f-8c10-d4ac990c531f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:04:10.607570 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:10.607537    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4d66eef-bccf-4b9f-8c10-d4ac990c531f-maas-api-tls" pod "a4d66eef-bccf-4b9f-8c10-d4ac990c531f" (UID: "a4d66eef-bccf-4b9f-8c10-d4ac990c531f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:04:10.607695 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:10.607608    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4d66eef-bccf-4b9f-8c10-d4ac990c531f-metrics-certs" pod "a4d66eef-bccf-4b9f-8c10-d4ac990c531f" (UID: "a4d66eef-bccf-4b9f-8c10-d4ac990c531f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:04:10.608199 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:10.608159    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4d66eef-bccf-4b9f-8c10-d4ac990c531f-kube-api-access-zc2nc" pod "a4d66eef-bccf-4b9f-8c10-d4ac990c531f" (UID: "a4d66eef-bccf-4b9f-8c10-d4ac990c531f"). InnerVolumeSpecName "kube-api-access-zc2nc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:04:10.705062 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:10.705026    6224 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4d66eef-bccf-4b9f-8c10-d4ac990c531f-metrics-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:04:10.705062 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:10.705059    6224 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4d66eef-bccf-4b9f-8c10-d4ac990c531f-openshift-service-ca\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:04:10.705284 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:10.705072    6224 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4d66eef-bccf-4b9f-8c10-d4ac990c531f-maas-api-tls\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:04:10.705284 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:10.705084    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc2nc\" (UniqueName: \"kubernetes.io/projected/a4d66eef-bccf-4b9f-8c10-d4ac990c531f-kube-api-access-zc2nc\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:04:10.718001 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:10.717968    6224 generic.go:356] "Generic (PLEG): container finished" podID="a4d66eef-bccf-4b9f-8c10-d4ac990c531f" containerID="7841af5f024f4c00960cd337b56179b015fffb0ea49405a0eba36bb07fa23523" exitCode=0
Oct 30 08:04:10.718169 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:10.718023    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4fe2ed-676bcd8cf6-pw5b5"
Oct 30 08:04:10.718169 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:10.718050    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4fe2ed-676bcd8cf6-pw5b5" event={"ID":"a4d66eef-bccf-4b9f-8c10-d4ac990c531f","Type":"ContainerDied","Data":"7841af5f024f4c00960cd337b56179b015fffb0ea49405a0eba36bb07fa23523"}
Oct 30 08:04:10.718169 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:10.718111    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4fe2ed-676bcd8cf6-pw5b5" event={"ID":"a4d66eef-bccf-4b9f-8c10-d4ac990c531f","Type":"ContainerDied","Data":"c48491b4e3d26d5125336e56fa46997117daf44570ebebc45f4cc68f8181e960"}
Oct 30 08:04:10.718169 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:10.718135    6224 scope.go:122] "RemoveContainer" containerID="7841af5f024f4c00960cd337b56179b015fffb0ea49405a0eba36bb07fa23523"
Oct 30 08:04:10.729694 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:10.729679    6224 scope.go:122] "RemoveContainer" containerID="7841af5f024f4c00960cd337b56179b015fffb0ea49405a0eba36bb07fa23523"
Oct 30 08:04:10.729978 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:04:10.729961    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7841af5f024f4c00960cd337b56179b015fffb0ea49405a0eba36bb07fa23523\": container with ID starting with 7841af5f024f4c00960cd337b56179b015fffb0ea49405a0eba36bb07fa23523 not found: ID does not exist" containerID="7841af5f024f4c00960cd337b56179b015fffb0ea49405a0eba36bb07fa23523"
Oct 30 08:04:10.730027 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:10.729986    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7841af5f024f4c00960cd337b56179b015fffb0ea49405a0eba36bb07fa23523"} err="failed to get container status \"7841af5f024f4c00960cd337b56179b015fffb0ea49405a0eba36bb07fa23523\": rpc error: code = NotFound desc = could not find container \"7841af5f024f4c00960cd337b56179b015fffb0ea49405a0eba36bb07fa23523\": container with ID starting with 7841af5f024f4c00960cd337b56179b015fffb0ea49405a0eba36bb07fa23523 not found: ID does not exist"
Oct 30 08:04:10.777546 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:10.777473    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-4fe2ed-676bcd8cf6-pw5b5"]
Oct 30 08:04:10.780300 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:10.780272    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-4fe2ed-676bcd8cf6-pw5b5"]
Oct 30 08:04:11.360838 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.360767    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7427-eb12a0"]
Oct 30 08:04:11.361458 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.361437    6224 cpu_manager.go:414] "containerMap: removing container" podUID="30fa2cf6-c67b-4be5-ab71-df50207183c5" containerName="payload-processing"
Oct 30 08:04:11.361507 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.361462    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30fa2cf6-c67b-4be5-ab71-df50207183c5" containerName="payload-processing"
Oct 30 08:04:11.361507 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.361483    6224 cpu_manager.go:414] "containerMap: removing container" podUID="f9463476-fcf5-4d66-bfd2-b85ce4ed651f" containerName="main"
Oct 30 08:04:11.361507 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.361490    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9463476-fcf5-4d66-bfd2-b85ce4ed651f" containerName="main"
Oct 30 08:04:11.361507 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.361504    6224 cpu_manager.go:414] "containerMap: removing container" podUID="7b8c015e-9234-4ecb-a908-9ce7eb600db5" containerName="payload-pre-processing"
Oct 30 08:04:11.361639 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.361512    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b8c015e-9234-4ecb-a908-9ce7eb600db5" containerName="payload-pre-processing"
Oct 30 08:04:11.361639 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.361529    6224 cpu_manager.go:414] "containerMap: removing container" podUID="7d82945b-c43f-413a-a2a4-941c9be966c4" containerName="payload-pre-processing"
Oct 30 08:04:11.361639 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.361536    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d82945b-c43f-413a-a2a4-941c9be966c4" containerName="payload-pre-processing"
Oct 30 08:04:11.361639 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.361547    6224 cpu_manager.go:414] "containerMap: removing container" podUID="89326bce-8281-4747-aa3f-c1d37c615234" containerName="istio-proxy"
Oct 30 08:04:11.361639 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.361554    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89326bce-8281-4747-aa3f-c1d37c615234" containerName="istio-proxy"
Oct 30 08:04:11.361639 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.361566    6224 cpu_manager.go:414] "containerMap: removing container" podUID="a4d66eef-bccf-4b9f-8c10-d4ac990c531f" containerName="maas-api"
Oct 30 08:04:11.361639 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.361572    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4d66eef-bccf-4b9f-8c10-d4ac990c531f" containerName="maas-api"
Oct 30 08:04:11.361639 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.361582    6224 cpu_manager.go:414] "containerMap: removing container" podUID="dda66230-cf6c-4f92-a50a-2edf07a1b0b8" containerName="maas-api"
Oct 30 08:04:11.361639 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.361589    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dda66230-cf6c-4f92-a50a-2edf07a1b0b8" containerName="maas-api"
Oct 30 08:04:11.361639 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.361604    6224 cpu_manager.go:414] "containerMap: removing container" podUID="a3d25e61-890d-45bd-957a-6690c5f26a08" containerName="istio-proxy"
Oct 30 08:04:11.361639 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.361611    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3d25e61-890d-45bd-957a-6690c5f26a08" containerName="istio-proxy"
Oct 30 08:04:11.361639 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.361621    6224 cpu_manager.go:414] "containerMap: removing container" podUID="f68a7a87-6354-4456-ab2c-17cf8995865f" containerName="cleanup"
Oct 30 08:04:11.361639 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.361628    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f68a7a87-6354-4456-ab2c-17cf8995865f" containerName="cleanup"
Oct 30 08:04:11.361639 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.361638    6224 cpu_manager.go:414] "containerMap: removing container" podUID="1313ec66-e8a0-4845-9181-cf9fe5b2d8c9" containerName="payload-processing"
Oct 30 08:04:11.362236 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.361645    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1313ec66-e8a0-4845-9181-cf9fe5b2d8c9" containerName="payload-processing"
Oct 30 08:04:11.362236 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.361655    6224 cpu_manager.go:414] "containerMap: removing container" podUID="38de25ac-6fa2-4650-9cfa-ec23daf2ec76" containerName="cleanup"
Oct 30 08:04:11.362236 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.361662    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38de25ac-6fa2-4650-9cfa-ec23daf2ec76" containerName="cleanup"
Oct 30 08:04:11.362236 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.361673    6224 cpu_manager.go:414] "containerMap: removing container" podUID="ff56f991-7e93-4d06-8d87-c42cb72465e3" containerName="payload-pre-processing"
Oct 30 08:04:11.362236 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.361679    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff56f991-7e93-4d06-8d87-c42cb72465e3" containerName="payload-pre-processing"
Oct 30 08:04:11.362236 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.361691    6224 cpu_manager.go:414] "containerMap: removing container" podUID="3a639cc2-221b-4e62-86c7-64ae4f7c43f7" containerName="cleanup"
Oct 30 08:04:11.362236 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.361698    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a639cc2-221b-4e62-86c7-64ae4f7c43f7" containerName="cleanup"
Oct 30 08:04:11.362236 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.361708    6224 cpu_manager.go:414] "containerMap: removing container" podUID="89326bce-8281-4747-aa3f-c1d37c615234" containerName="istio-proxy"
Oct 30 08:04:11.362236 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.361715    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89326bce-8281-4747-aa3f-c1d37c615234" containerName="istio-proxy"
Oct 30 08:04:11.362236 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.361740    6224 cpu_manager.go:414] "containerMap: removing container" podUID="11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00" containerName="main"
Oct 30 08:04:11.362236 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.361746    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00" containerName="main"
Oct 30 08:04:11.362236 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.361755    6224 cpu_manager.go:414] "containerMap: removing container" podUID="e8936cc2-40e5-4d2e-81b0-1ad1ca9e5a2a" containerName="cleanup"
Oct 30 08:04:11.362236 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.361761    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8936cc2-40e5-4d2e-81b0-1ad1ca9e5a2a" containerName="cleanup"
Oct 30 08:04:11.362236 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.361769    6224 cpu_manager.go:414] "containerMap: removing container" podUID="4444b346-84b4-4ac2-b445-0b43ec4d6fda" containerName="maas-api"
Oct 30 08:04:11.362236 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.361775    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4444b346-84b4-4ac2-b445-0b43ec4d6fda" containerName="maas-api"
Oct 30 08:04:11.362236 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.361792    6224 cpu_manager.go:414] "containerMap: removing container" podUID="c7b09178-9ac7-492f-890b-5ff3da130981" containerName="main"
Oct 30 08:04:11.362236 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.361798    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7b09178-9ac7-492f-890b-5ff3da130981" containerName="main"
Oct 30 08:04:11.362236 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.361824    6224 cpu_manager.go:414] "containerMap: removing container" podUID="1cd532fb-2369-434f-9622-d6792f8ee7a0" containerName="cleanup"
Oct 30 08:04:11.362236 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.361831    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cd532fb-2369-434f-9622-d6792f8ee7a0" containerName="cleanup"
Oct 30 08:04:11.362236 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.361841    6224 cpu_manager.go:414] "containerMap: removing container" podUID="da7c6736-ac92-44ec-a1cf-5f835eb860b1" containerName="cleanup"
Oct 30 08:04:11.362236 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.361848    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da7c6736-ac92-44ec-a1cf-5f835eb860b1" containerName="cleanup"
Oct 30 08:04:11.362236 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.361860    6224 cpu_manager.go:414] "containerMap: removing container" podUID="1f058cca-8314-4ff2-aa1f-3169f7d770fa" containerName="payload-processing"
Oct 30 08:04:11.362236 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.361867    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f058cca-8314-4ff2-aa1f-3169f7d770fa" containerName="payload-processing"
Oct 30 08:04:11.362236 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.361891    6224 cpu_manager.go:414] "containerMap: removing container" podUID="e2a703c4-5066-4875-be2d-5fc882b38502" containerName="payload-pre-processing"
Oct 30 08:04:11.362236 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.361898    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2a703c4-5066-4875-be2d-5fc882b38502" containerName="payload-pre-processing"
Oct 30 08:04:11.362236 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.362038    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="1313ec66-e8a0-4845-9181-cf9fe5b2d8c9" containerName="payload-processing"
Oct 30 08:04:11.362236 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.362049    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff56f991-7e93-4d06-8d87-c42cb72465e3" containerName="payload-pre-processing"
Oct 30 08:04:11.362236 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.362057    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a639cc2-221b-4e62-86c7-64ae4f7c43f7" containerName="cleanup"
Oct 30 08:04:11.362236 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.362068    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb6ead4a-d324-450f-adb6-dbbb75afc174" containerName="cleanup"
Oct 30 08:04:11.362236 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.362081    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="89326bce-8281-4747-aa3f-c1d37c615234" containerName="istio-proxy"
Oct 30 08:04:11.362236 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.362089    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="246d2cbc-c719-46fd-a84d-0e47a21358fe" containerName="cleanup"
Oct 30 08:04:11.362236 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.362103    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="11c3cd4b-c4fc-4ef6-9eb7-85f9be0b8a00" containerName="main"
Oct 30 08:04:11.362236 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.362113    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8936cc2-40e5-4d2e-81b0-1ad1ca9e5a2a" containerName="cleanup"
Oct 30 08:04:11.362236 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.362121    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="4444b346-84b4-4ac2-b445-0b43ec4d6fda" containerName="maas-api"
Oct 30 08:04:11.362236 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.362130    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="38de25ac-6fa2-4650-9cfa-ec23daf2ec76" containerName="cleanup"
Oct 30 08:04:11.362236 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.362138    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="f68a7a87-6354-4456-ab2c-17cf8995865f" containerName="cleanup"
Oct 30 08:04:11.362236 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.362147    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7b09178-9ac7-492f-890b-5ff3da130981" containerName="main"
Oct 30 08:04:11.362236 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.362156    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cd532fb-2369-434f-9622-d6792f8ee7a0" containerName="cleanup"
Oct 30 08:04:11.362236 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.362164    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="da7c6736-ac92-44ec-a1cf-5f835eb860b1" containerName="cleanup"
Oct 30 08:04:11.362236 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.362180    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f058cca-8314-4ff2-aa1f-3169f7d770fa" containerName="payload-processing"
Oct 30 08:04:11.362236 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.362192    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2a703c4-5066-4875-be2d-5fc882b38502" containerName="payload-pre-processing"
Oct 30 08:04:11.362236 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.362202    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="30fa2cf6-c67b-4be5-ab71-df50207183c5" containerName="payload-processing"
Oct 30 08:04:11.362236 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.362212    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9463476-fcf5-4d66-bfd2-b85ce4ed651f" containerName="main"
Oct 30 08:04:11.362236 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.362221    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b8c015e-9234-4ecb-a908-9ce7eb600db5" containerName="payload-pre-processing"
Oct 30 08:04:11.362236 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.362232    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d82945b-c43f-413a-a2a4-941c9be966c4" containerName="payload-pre-processing"
Oct 30 08:04:11.362236 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.362241    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4d66eef-bccf-4b9f-8c10-d4ac990c531f" containerName="maas-api"
Oct 30 08:04:11.362236 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.362250    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="dda66230-cf6c-4f92-a50a-2edf07a1b0b8" containerName="maas-api"
Oct 30 08:04:11.362236 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.362259    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3d25e61-890d-45bd-957a-6690c5f26a08" containerName="istio-proxy"
Oct 30 08:04:11.371442 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.371417    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7427-eb12a0"
Oct 30 08:04:11.385580 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.385549    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7427-eb12a0"]
Oct 30 08:04:11.390725 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.390694    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5q8j4\""
Oct 30 08:04:11.412147 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.412109    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djzjx\" (UniqueName: \"kubernetes.io/projected/cb9ac64f-913e-496b-9ac3-eafec40bf2bd-kube-api-access-djzjx\") pod \"test-curl-7427-eb12a0\" (UID: \"cb9ac64f-913e-496b-9ac3-eafec40bf2bd\") " pod="opendatahub/test-curl-7427-eb12a0"
Oct 30 08:04:11.504317 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.504283    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4d66eef-bccf-4b9f-8c10-d4ac990c531f" path="/var/lib/kubelet/pods/a4d66eef-bccf-4b9f-8c10-d4ac990c531f/volumes"
Oct 30 08:04:11.513584 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.513557    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djzjx\" (UniqueName: \"kubernetes.io/projected/cb9ac64f-913e-496b-9ac3-eafec40bf2bd-kube-api-access-djzjx\") pod \"test-curl-7427-eb12a0\" (UID: \"cb9ac64f-913e-496b-9ac3-eafec40bf2bd\") " pod="opendatahub/test-curl-7427-eb12a0"
Oct 30 08:04:11.557367 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.557337    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djzjx\" (UniqueName: \"kubernetes.io/projected/cb9ac64f-913e-496b-9ac3-eafec40bf2bd-kube-api-access-djzjx\") pod \"test-curl-7427-eb12a0\" (UID: \"cb9ac64f-913e-496b-9ac3-eafec40bf2bd\") " pod="opendatahub/test-curl-7427-eb12a0"
Oct 30 08:04:11.689271 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.689194    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7427-eb12a0"
Oct 30 08:04:11.996101 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:11.996073    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7427-eb12a0"]
Oct 30 08:04:11.997259 ip-10-0-128-125 kubenswrapper[6224]: W1030 08:04:11.997232    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb9ac64f_913e_496b_9ac3_eafec40bf2bd.slice/crio-4dae8ede3c52d60749788023a55f6fea35429d0b30b91ddea6abccdcce8dd51b WatchSource:0}: Error finding container 4dae8ede3c52d60749788023a55f6fea35429d0b30b91ddea6abccdcce8dd51b: Status 404 returned error can't find the container with id 4dae8ede3c52d60749788023a55f6fea35429d0b30b91ddea6abccdcce8dd51b
Oct 30 08:04:12.732555 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:12.732508    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7427-eb12a0" event={"ID":"cb9ac64f-913e-496b-9ac3-eafec40bf2bd","Type":"ContainerStarted","Data":"9742b6249ec1eb7366d49a323b33ce5b119a678af6c3954d14dcd94c902fe7d4"}
Oct 30 08:04:12.732555 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:12.732555    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7427-eb12a0" event={"ID":"cb9ac64f-913e-496b-9ac3-eafec40bf2bd","Type":"ContainerStarted","Data":"4dae8ede3c52d60749788023a55f6fea35429d0b30b91ddea6abccdcce8dd51b"}
Oct 30 08:04:12.789676 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:12.789626    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7427-eb12a0" podStartSLOduration=1.7896095669999998 podStartE2EDuration="1.789609567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:04:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:04:12.788495521 +0000 UTC m=+4011.397902555" watchObservedRunningTime="2026-10-30 08:04:12.789609567 +0000 UTC m=+4011.399016538"
Oct 30 08:04:13.728284 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:13.728249    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7427-eb12a0"]
Oct 30 08:04:13.732579 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:13.732549    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7427-eb12a0"]
Oct 30 08:04:13.897539 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:13.897487    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7427-852b4b"]
Oct 30 08:04:13.898056 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:13.898034    6224 cpu_manager.go:414] "containerMap: removing container" podUID="cb9ac64f-913e-496b-9ac3-eafec40bf2bd" containerName="test-curl-7427-eb12a0"
Oct 30 08:04:13.898056 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:13.898056    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb9ac64f-913e-496b-9ac3-eafec40bf2bd" containerName="test-curl-7427-eb12a0"
Oct 30 08:04:13.898241 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:13.898227    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb9ac64f-913e-496b-9ac3-eafec40bf2bd" containerName="test-curl-7427-eb12a0"
Oct 30 08:04:13.898299 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:13.898254    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="89326bce-8281-4747-aa3f-c1d37c615234" containerName="istio-proxy"
Oct 30 08:04:13.901515 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:13.901495    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7427-852b4b"
Oct 30 08:04:13.918500 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:13.918467    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7427-852b4b"]
Oct 30 08:04:13.938534 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:13.938509    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chzmg\" (UniqueName: \"kubernetes.io/projected/9981a1cd-161f-437f-93f3-877f324f6472-kube-api-access-chzmg\") pod \"test-curl-7427-852b4b\" (UID: \"9981a1cd-161f-437f-93f3-877f324f6472\") " pod="opendatahub/test-curl-7427-852b4b"
Oct 30 08:04:14.039757 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:14.039734    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chzmg\" (UniqueName: \"kubernetes.io/projected/9981a1cd-161f-437f-93f3-877f324f6472-kube-api-access-chzmg\") pod \"test-curl-7427-852b4b\" (UID: \"9981a1cd-161f-437f-93f3-877f324f6472\") " pod="opendatahub/test-curl-7427-852b4b"
Oct 30 08:04:14.073253 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:14.073227    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chzmg\" (UniqueName: \"kubernetes.io/projected/9981a1cd-161f-437f-93f3-877f324f6472-kube-api-access-chzmg\") pod \"test-curl-7427-852b4b\" (UID: \"9981a1cd-161f-437f-93f3-877f324f6472\") " pod="opendatahub/test-curl-7427-852b4b"
Oct 30 08:04:14.213303 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:14.213271    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7427-852b4b"
Oct 30 08:04:14.508859 ip-10-0-128-125 kubenswrapper[6224]: W1030 08:04:14.508823    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9981a1cd_161f_437f_93f3_877f324f6472.slice/crio-b4a0f3cdd56cd828db5f91358480b5f0ff6a9bd29f8ff46d75e3478ad99f0289 WatchSource:0}: Error finding container b4a0f3cdd56cd828db5f91358480b5f0ff6a9bd29f8ff46d75e3478ad99f0289: Status 404 returned error can't find the container with id b4a0f3cdd56cd828db5f91358480b5f0ff6a9bd29f8ff46d75e3478ad99f0289
Oct 30 08:04:14.509147 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:14.509113    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7427-852b4b"]
Oct 30 08:04:14.745318 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:14.745276    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7427-852b4b" event={"ID":"9981a1cd-161f-437f-93f3-877f324f6472","Type":"ContainerStarted","Data":"ff5573c80fe7854591b1f3926d5025448a6a4e287e8b3ba88c4579ac22a171e8"}
Oct 30 08:04:14.745318 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:14.745324    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7427-852b4b" event={"ID":"9981a1cd-161f-437f-93f3-877f324f6472","Type":"ContainerStarted","Data":"b4a0f3cdd56cd828db5f91358480b5f0ff6a9bd29f8ff46d75e3478ad99f0289"}
Oct 30 08:04:14.745865 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:14.745444    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7427-eb12a0" podUID="cb9ac64f-913e-496b-9ac3-eafec40bf2bd" containerName="test-curl-7427-eb12a0" containerID="cri-o://9742b6249ec1eb7366d49a323b33ce5b119a678af6c3954d14dcd94c902fe7d4" gracePeriod=2
Oct 30 08:04:14.788169 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:14.788125    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7427-852b4b" podStartSLOduration=1.788110608 podStartE2EDuration="1.788110608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:04:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:04:14.787189217 +0000 UTC m=+4013.396596187" watchObservedRunningTime="2026-10-30 08:04:14.788110608 +0000 UTC m=+4013.397517578"
Oct 30 08:04:14.802796 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:04:14.802762    6224 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7427-eb12a0\" is forbidden: User \"system:node:ip-10-0-128-125.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-125.ec2.internal' and this object" podUID="cb9ac64f-913e-496b-9ac3-eafec40bf2bd" pod="opendatahub/test-curl-7427-eb12a0"
Oct 30 08:04:15.707368 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:15.707328    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7427-852b4b"]
Oct 30 08:04:15.712893 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:15.712863    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7427-852b4b"]
Oct 30 08:04:16.757881 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:16.757848    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7427-852b4b" podUID="9981a1cd-161f-437f-93f3-877f324f6472" containerName="test-curl-7427-852b4b" containerID="cri-o://ff5573c80fe7854591b1f3926d5025448a6a4e287e8b3ba88c4579ac22a171e8" gracePeriod=2
Oct 30 08:04:16.773497 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:04:16.773466    6224 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7427-852b4b\" is forbidden: User \"system:node:ip-10-0-128-125.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-125.ec2.internal' and this object" podUID="9981a1cd-161f-437f-93f3-877f324f6472" pod="opendatahub/test-curl-7427-852b4b"
Oct 30 08:04:16.952588 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:16.952564    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7427-eb12a0"
Oct 30 08:04:16.968286 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:04:16.968253    6224 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7427-852b4b\" is forbidden: User \"system:node:ip-10-0-128-125.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-125.ec2.internal' and this object" podUID="9981a1cd-161f-437f-93f3-877f324f6472" pod="opendatahub/test-curl-7427-852b4b"
Oct 30 08:04:17.025371 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.025303    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-bb6be4/premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t" podUID="7dd5133c-480f-4fa0-9dd1-27662004aae9" containerName="main" containerID="cri-o://a3887e8e179a28288c6197f63e0d5a3549dec18c36e3ff7675e25d2c6cc18830" gracePeriod=45
Oct 30 08:04:17.040298 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.040227    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-bb6be4/simulator-w2-bb6be4-kserve-8485bbf954-dnbbz" podUID="6370f7e2-d6c8-4fb5-8198-32b38837ca31" containerName="main" containerID="cri-o://811a8b06e535ff298712ff1b196869cd36756bd34d8241bafff1c8c07dc21ed9" gracePeriod=45
Oct 30 08:04:17.070543 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.070513    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb9ac64f-913e-496b-9ac3-eafec40bf2bd-kube-api-access-djzjx\" (UniqueName: \"kubernetes.io/projected/cb9ac64f-913e-496b-9ac3-eafec40bf2bd-kube-api-access-djzjx\") pod \"cb9ac64f-913e-496b-9ac3-eafec40bf2bd\" (UID: \"cb9ac64f-913e-496b-9ac3-eafec40bf2bd\") "
Oct 30 08:04:17.072382 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.072358    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb9ac64f-913e-496b-9ac3-eafec40bf2bd-kube-api-access-djzjx" pod "cb9ac64f-913e-496b-9ac3-eafec40bf2bd" (UID: "cb9ac64f-913e-496b-9ac3-eafec40bf2bd"). InnerVolumeSpecName "kube-api-access-djzjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:04:17.172220 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.172153    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djzjx\" (UniqueName: \"kubernetes.io/projected/cb9ac64f-913e-496b-9ac3-eafec40bf2bd-kube-api-access-djzjx\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:04:17.309727 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.309682    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-bb6be4/premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t"
Oct 30 08:04:17.322473 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.322440    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-bb6be4/simulator-w2-bb6be4-kserve-8485bbf954-dnbbz"
Oct 30 08:04:17.325736 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:04:17.325708    6224 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7427-852b4b\" is forbidden: User \"system:node:ip-10-0-128-125.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-125.ec2.internal' and this object" podUID="9981a1cd-161f-437f-93f3-877f324f6472" pod="opendatahub/test-curl-7427-852b4b"
Oct 30 08:04:17.374349 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.374316    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6370f7e2-d6c8-4fb5-8198-32b38837ca31-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6370f7e2-d6c8-4fb5-8198-32b38837ca31-model-cache\") pod \"6370f7e2-d6c8-4fb5-8198-32b38837ca31\" (UID: \"6370f7e2-d6c8-4fb5-8198-32b38837ca31\") "
Oct 30 08:04:17.374532 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.374404    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6370f7e2-d6c8-4fb5-8198-32b38837ca31-home\" (UniqueName: \"kubernetes.io/empty-dir/6370f7e2-d6c8-4fb5-8198-32b38837ca31-home\") pod \"6370f7e2-d6c8-4fb5-8198-32b38837ca31\" (UID: \"6370f7e2-d6c8-4fb5-8198-32b38837ca31\") "
Oct 30 08:04:17.374532 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.374446    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dd5133c-480f-4fa0-9dd1-27662004aae9-home\" (UniqueName: \"kubernetes.io/empty-dir/7dd5133c-480f-4fa0-9dd1-27662004aae9-home\") pod \"7dd5133c-480f-4fa0-9dd1-27662004aae9\" (UID: \"7dd5133c-480f-4fa0-9dd1-27662004aae9\") "
Oct 30 08:04:17.374532 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.374480    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7dd5133c-480f-4fa0-9dd1-27662004aae9-tls-certs\" (UniqueName: \"kubernetes.io/secret/7dd5133c-480f-4fa0-9dd1-27662004aae9-tls-certs\") pod \"7dd5133c-480f-4fa0-9dd1-27662004aae9\" (UID: \"7dd5133c-480f-4fa0-9dd1-27662004aae9\") "
Oct 30 08:04:17.374532 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.374518    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6370f7e2-d6c8-4fb5-8198-32b38837ca31-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6370f7e2-d6c8-4fb5-8198-32b38837ca31-dshm\") pod \"6370f7e2-d6c8-4fb5-8198-32b38837ca31\" (UID: \"6370f7e2-d6c8-4fb5-8198-32b38837ca31\") "
Oct 30 08:04:17.374742 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.374541    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6370f7e2-d6c8-4fb5-8198-32b38837ca31-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6370f7e2-d6c8-4fb5-8198-32b38837ca31-tmp-dir\") pod \"6370f7e2-d6c8-4fb5-8198-32b38837ca31\" (UID: \"6370f7e2-d6c8-4fb5-8198-32b38837ca31\") "
Oct 30 08:04:17.374742 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.374562    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6370f7e2-d6c8-4fb5-8198-32b38837ca31-kube-api-access-ntsfp\" (UniqueName: \"kubernetes.io/projected/6370f7e2-d6c8-4fb5-8198-32b38837ca31-kube-api-access-ntsfp\") pod \"6370f7e2-d6c8-4fb5-8198-32b38837ca31\" (UID: \"6370f7e2-d6c8-4fb5-8198-32b38837ca31\") "
Oct 30 08:04:17.374742 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.374639    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dd5133c-480f-4fa0-9dd1-27662004aae9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7dd5133c-480f-4fa0-9dd1-27662004aae9-tmp-dir\") pod \"7dd5133c-480f-4fa0-9dd1-27662004aae9\" (UID: \"7dd5133c-480f-4fa0-9dd1-27662004aae9\") "
Oct 30 08:04:17.374742 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.374663    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dd5133c-480f-4fa0-9dd1-27662004aae9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7dd5133c-480f-4fa0-9dd1-27662004aae9-model-cache\") pod \"7dd5133c-480f-4fa0-9dd1-27662004aae9\" (UID: \"7dd5133c-480f-4fa0-9dd1-27662004aae9\") "
Oct 30 08:04:17.374742 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.374688    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dd5133c-480f-4fa0-9dd1-27662004aae9-kube-api-access-m8mtk\" (UniqueName: \"kubernetes.io/projected/7dd5133c-480f-4fa0-9dd1-27662004aae9-kube-api-access-m8mtk\") pod \"7dd5133c-480f-4fa0-9dd1-27662004aae9\" (UID: \"7dd5133c-480f-4fa0-9dd1-27662004aae9\") "
Oct 30 08:04:17.374742 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.374713    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dd5133c-480f-4fa0-9dd1-27662004aae9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7dd5133c-480f-4fa0-9dd1-27662004aae9-dshm\") pod \"7dd5133c-480f-4fa0-9dd1-27662004aae9\" (UID: \"7dd5133c-480f-4fa0-9dd1-27662004aae9\") "
Oct 30 08:04:17.375704 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.374759    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6370f7e2-d6c8-4fb5-8198-32b38837ca31-tls-certs\" (UniqueName: \"kubernetes.io/secret/6370f7e2-d6c8-4fb5-8198-32b38837ca31-tls-certs\") pod \"6370f7e2-d6c8-4fb5-8198-32b38837ca31\" (UID: \"6370f7e2-d6c8-4fb5-8198-32b38837ca31\") "
Oct 30 08:04:17.375704 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.375169    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6370f7e2-d6c8-4fb5-8198-32b38837ca31-tmp-dir" pod "6370f7e2-d6c8-4fb5-8198-32b38837ca31" (UID: "6370f7e2-d6c8-4fb5-8198-32b38837ca31"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:04:17.375704 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.375174    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dd5133c-480f-4fa0-9dd1-27662004aae9-model-cache" pod "7dd5133c-480f-4fa0-9dd1-27662004aae9" (UID: "7dd5133c-480f-4fa0-9dd1-27662004aae9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:04:17.375704 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.375442    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6370f7e2-d6c8-4fb5-8198-32b38837ca31-model-cache" pod "6370f7e2-d6c8-4fb5-8198-32b38837ca31" (UID: "6370f7e2-d6c8-4fb5-8198-32b38837ca31"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:04:17.375704 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.375625    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6370f7e2-d6c8-4fb5-8198-32b38837ca31-home" pod "6370f7e2-d6c8-4fb5-8198-32b38837ca31" (UID: "6370f7e2-d6c8-4fb5-8198-32b38837ca31"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:04:17.376077 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.376048    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dd5133c-480f-4fa0-9dd1-27662004aae9-home" pod "7dd5133c-480f-4fa0-9dd1-27662004aae9" (UID: "7dd5133c-480f-4fa0-9dd1-27662004aae9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:04:17.376162 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.376065    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dd5133c-480f-4fa0-9dd1-27662004aae9-tmp-dir" pod "7dd5133c-480f-4fa0-9dd1-27662004aae9" (UID: "7dd5133c-480f-4fa0-9dd1-27662004aae9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:04:17.381284 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.381254    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6370f7e2-d6c8-4fb5-8198-32b38837ca31-tls-certs" pod "6370f7e2-d6c8-4fb5-8198-32b38837ca31" (UID: "6370f7e2-d6c8-4fb5-8198-32b38837ca31"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:04:17.381284 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.381263    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dd5133c-480f-4fa0-9dd1-27662004aae9-tls-certs" pod "7dd5133c-480f-4fa0-9dd1-27662004aae9" (UID: "7dd5133c-480f-4fa0-9dd1-27662004aae9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:04:17.381505 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.381367    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dd5133c-480f-4fa0-9dd1-27662004aae9-kube-api-access-m8mtk" pod "7dd5133c-480f-4fa0-9dd1-27662004aae9" (UID: "7dd5133c-480f-4fa0-9dd1-27662004aae9"). InnerVolumeSpecName "kube-api-access-m8mtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:04:17.381505 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.381465    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6370f7e2-d6c8-4fb5-8198-32b38837ca31-kube-api-access-ntsfp" pod "6370f7e2-d6c8-4fb5-8198-32b38837ca31" (UID: "6370f7e2-d6c8-4fb5-8198-32b38837ca31"). InnerVolumeSpecName "kube-api-access-ntsfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:04:17.381624 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.381532    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dd5133c-480f-4fa0-9dd1-27662004aae9-dshm" pod "7dd5133c-480f-4fa0-9dd1-27662004aae9" (UID: "7dd5133c-480f-4fa0-9dd1-27662004aae9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:04:17.381728 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.381696    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6370f7e2-d6c8-4fb5-8198-32b38837ca31-dshm" pod "6370f7e2-d6c8-4fb5-8198-32b38837ca31" (UID: "6370f7e2-d6c8-4fb5-8198-32b38837ca31"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:04:17.388964 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:04:17.388928    6224 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7427-852b4b\" is forbidden: User \"system:node:ip-10-0-128-125.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-125.ec2.internal' and this object" podUID="9981a1cd-161f-437f-93f3-877f324f6472" pod="opendatahub/test-curl-7427-852b4b"
Oct 30 08:04:17.476458 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.476421    6224 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7dd5133c-480f-4fa0-9dd1-27662004aae9-tmp-dir\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:04:17.476458 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.476456    6224 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7dd5133c-480f-4fa0-9dd1-27662004aae9-model-cache\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:04:17.476458 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.476466    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8mtk\" (UniqueName: \"kubernetes.io/projected/7dd5133c-480f-4fa0-9dd1-27662004aae9-kube-api-access-m8mtk\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:04:17.476709 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.476474    6224 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7dd5133c-480f-4fa0-9dd1-27662004aae9-dshm\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:04:17.476709 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.476483    6224 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6370f7e2-d6c8-4fb5-8198-32b38837ca31-tls-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:04:17.476709 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.476493    6224 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6370f7e2-d6c8-4fb5-8198-32b38837ca31-model-cache\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:04:17.476709 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.476508    6224 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6370f7e2-d6c8-4fb5-8198-32b38837ca31-home\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:04:17.476709 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.476524    6224 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7dd5133c-480f-4fa0-9dd1-27662004aae9-home\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:04:17.476709 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.476537    6224 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7dd5133c-480f-4fa0-9dd1-27662004aae9-tls-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:04:17.476709 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.476549    6224 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6370f7e2-d6c8-4fb5-8198-32b38837ca31-dshm\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:04:17.476709 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.476560    6224 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6370f7e2-d6c8-4fb5-8198-32b38837ca31-tmp-dir\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:04:17.476709 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.476572    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntsfp\" (UniqueName: \"kubernetes.io/projected/6370f7e2-d6c8-4fb5-8198-32b38837ca31-kube-api-access-ntsfp\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:04:17.504567 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.504535    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb9ac64f-913e-496b-9ac3-eafec40bf2bd" path="/var/lib/kubelet/pods/cb9ac64f-913e-496b-9ac3-eafec40bf2bd/volumes"
Oct 30 08:04:17.766309 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.766278    6224 generic.go:356] "Generic (PLEG): container finished" podID="7dd5133c-480f-4fa0-9dd1-27662004aae9" containerID="a3887e8e179a28288c6197f63e0d5a3549dec18c36e3ff7675e25d2c6cc18830" exitCode=0
Oct 30 08:04:17.766752 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.766324    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-bb6be4/premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t"
Oct 30 08:04:17.766752 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.766364    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-bb6be4/premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t" event={"ID":"7dd5133c-480f-4fa0-9dd1-27662004aae9","Type":"ContainerDied","Data":"a3887e8e179a28288c6197f63e0d5a3549dec18c36e3ff7675e25d2c6cc18830"}
Oct 30 08:04:17.766752 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.766424    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-bb6be4/premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t" event={"ID":"7dd5133c-480f-4fa0-9dd1-27662004aae9","Type":"ContainerDied","Data":"59a2aad79c02f433e063ec33611d880d51b04d87d06806c24f711e04265bd95c"}
Oct 30 08:04:17.766752 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.766446    6224 scope.go:122] "RemoveContainer" containerID="a3887e8e179a28288c6197f63e0d5a3549dec18c36e3ff7675e25d2c6cc18830"
Oct 30 08:04:17.767912 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.767822    6224 generic.go:356] "Generic (PLEG): container finished" podID="cb9ac64f-913e-496b-9ac3-eafec40bf2bd" containerID="9742b6249ec1eb7366d49a323b33ce5b119a678af6c3954d14dcd94c902fe7d4" exitCode=137
Oct 30 08:04:17.767912 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.767870    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7427-eb12a0"
Oct 30 08:04:17.769352 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.769327    6224 generic.go:356] "Generic (PLEG): container finished" podID="6370f7e2-d6c8-4fb5-8198-32b38837ca31" containerID="811a8b06e535ff298712ff1b196869cd36756bd34d8241bafff1c8c07dc21ed9" exitCode=0
Oct 30 08:04:17.769453 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.769366    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-bb6be4/simulator-w2-bb6be4-kserve-8485bbf954-dnbbz"
Oct 30 08:04:17.769518 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.769367    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-bb6be4/simulator-w2-bb6be4-kserve-8485bbf954-dnbbz" event={"ID":"6370f7e2-d6c8-4fb5-8198-32b38837ca31","Type":"ContainerDied","Data":"811a8b06e535ff298712ff1b196869cd36756bd34d8241bafff1c8c07dc21ed9"}
Oct 30 08:04:17.769518 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.769486    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-bb6be4/simulator-w2-bb6be4-kserve-8485bbf954-dnbbz" event={"ID":"6370f7e2-d6c8-4fb5-8198-32b38837ca31","Type":"ContainerDied","Data":"237cfe39a1e08c3eae36a1a820c12a064674552254f07947b4e3d329b1fafe37"}
Oct 30 08:04:17.777641 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.777625    6224 scope.go:122] "RemoveContainer" containerID="a3887e8e179a28288c6197f63e0d5a3549dec18c36e3ff7675e25d2c6cc18830"
Oct 30 08:04:17.777869 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:04:17.777847    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3887e8e179a28288c6197f63e0d5a3549dec18c36e3ff7675e25d2c6cc18830\": container with ID starting with a3887e8e179a28288c6197f63e0d5a3549dec18c36e3ff7675e25d2c6cc18830 not found: ID does not exist" containerID="a3887e8e179a28288c6197f63e0d5a3549dec18c36e3ff7675e25d2c6cc18830"
Oct 30 08:04:17.777963 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.777879    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3887e8e179a28288c6197f63e0d5a3549dec18c36e3ff7675e25d2c6cc18830"} err="failed to get container status \"a3887e8e179a28288c6197f63e0d5a3549dec18c36e3ff7675e25d2c6cc18830\": rpc error: code = NotFound desc = could not find container \"a3887e8e179a28288c6197f63e0d5a3549dec18c36e3ff7675e25d2c6cc18830\": container with ID starting with a3887e8e179a28288c6197f63e0d5a3549dec18c36e3ff7675e25d2c6cc18830 not found: ID does not exist"
Oct 30 08:04:17.777963 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.777895    6224 scope.go:122] "RemoveContainer" containerID="9742b6249ec1eb7366d49a323b33ce5b119a678af6c3954d14dcd94c902fe7d4"
Oct 30 08:04:17.786072 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.786054    6224 scope.go:122] "RemoveContainer" containerID="9742b6249ec1eb7366d49a323b33ce5b119a678af6c3954d14dcd94c902fe7d4"
Oct 30 08:04:17.786333 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:04:17.786315    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9742b6249ec1eb7366d49a323b33ce5b119a678af6c3954d14dcd94c902fe7d4\": container with ID starting with 9742b6249ec1eb7366d49a323b33ce5b119a678af6c3954d14dcd94c902fe7d4 not found: ID does not exist" containerID="9742b6249ec1eb7366d49a323b33ce5b119a678af6c3954d14dcd94c902fe7d4"
Oct 30 08:04:17.786379 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.786338    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9742b6249ec1eb7366d49a323b33ce5b119a678af6c3954d14dcd94c902fe7d4"} err="failed to get container status \"9742b6249ec1eb7366d49a323b33ce5b119a678af6c3954d14dcd94c902fe7d4\": rpc error: code = NotFound desc = could not find container \"9742b6249ec1eb7366d49a323b33ce5b119a678af6c3954d14dcd94c902fe7d4\": container with ID starting with 9742b6249ec1eb7366d49a323b33ce5b119a678af6c3954d14dcd94c902fe7d4 not found: ID does not exist"
Oct 30 08:04:17.786379 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.786354    6224 scope.go:122] "RemoveContainer" containerID="811a8b06e535ff298712ff1b196869cd36756bd34d8241bafff1c8c07dc21ed9"
Oct 30 08:04:17.795168 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.795151    6224 scope.go:122] "RemoveContainer" containerID="811a8b06e535ff298712ff1b196869cd36756bd34d8241bafff1c8c07dc21ed9"
Oct 30 08:04:17.795442 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:04:17.795421    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"811a8b06e535ff298712ff1b196869cd36756bd34d8241bafff1c8c07dc21ed9\": container with ID starting with 811a8b06e535ff298712ff1b196869cd36756bd34d8241bafff1c8c07dc21ed9 not found: ID does not exist" containerID="811a8b06e535ff298712ff1b196869cd36756bd34d8241bafff1c8c07dc21ed9"
Oct 30 08:04:17.795490 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.795452    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"811a8b06e535ff298712ff1b196869cd36756bd34d8241bafff1c8c07dc21ed9"} err="failed to get container status \"811a8b06e535ff298712ff1b196869cd36756bd34d8241bafff1c8c07dc21ed9\": rpc error: code = NotFound desc = could not find container \"811a8b06e535ff298712ff1b196869cd36756bd34d8241bafff1c8c07dc21ed9\": container with ID starting with 811a8b06e535ff298712ff1b196869cd36756bd34d8241bafff1c8c07dc21ed9 not found: ID does not exist"
Oct 30 08:04:17.827490 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.827465    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-bb6be4/simulator-w2-bb6be4-kserve-8485bbf954-dnbbz"]
Oct 30 08:04:17.842203 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.842179    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-bb6be4/simulator-w2-bb6be4-kserve-8485bbf954-dnbbz"]
Oct 30 08:04:17.870634 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.870610    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-bb6be4/premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t"]
Oct 30 08:04:17.874153 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.874133    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-bb6be4/premium-w2-bb6be4-kserve-5b8f89cbc9-zfr7t"]
Oct 30 08:04:17.961698 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.961669    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bgnvh"]
Oct 30 08:04:17.961898 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.961876    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bgnvh" podUID="14ba5fea-7709-42cb-8fe3-b63093fdcda7" containerName="istio-proxy" containerID="cri-o://5e8f88e9e3c524f1b361251c3f0b6e8ac09d68ed2d5bf237a767580b1076d8fe" gracePeriod=30
Oct 30 08:04:17.965044 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.965020    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mn4cr"]
Oct 30 08:04:17.965248 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.965223    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mn4cr" podUID="5747b4f9-c446-4172-9e98-cea9f7489f2f" containerName="istio-proxy" containerID="cri-o://eb6baacd206507e426943b8df3d65b1fdf6f43cdc9bbced3cd49f433751dfb7a" gracePeriod=30
Oct 30 08:04:17.967498 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.967471    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hpbmz"]
Oct 30 08:04:17.968837 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.967654    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hpbmz" podUID="522040eb-5fce-4b49-9856-410c83cffd8f" containerName="istio-proxy" containerID="cri-o://6acbf2763ed95634cb7c9b74684e73272afdd2fd7735ee2e1cb32e493d6392c9" gracePeriod=30
Oct 30 08:04:17.969840 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.969818    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-54xv2"]
Oct 30 08:04:17.970013 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:17.969991    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-54xv2" podUID="bb3586a2-80e5-44b7-8c23-bcf26c3a8147" containerName="istio-proxy" containerID="cri-o://479382ec07779630bc6b07691d6c931ab608ab847fdf9b998e282c6c9c9f5617" gracePeriod=30
Oct 30 08:04:18.779906 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:18.779876    6224 generic.go:356] "Generic (PLEG): container finished" podID="9981a1cd-161f-437f-93f3-877f324f6472" containerID="ff5573c80fe7854591b1f3926d5025448a6a4e287e8b3ba88c4579ac22a171e8" exitCode=137
Oct 30 08:04:18.925819 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:18.925787    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7427-852b4b"
Oct 30 08:04:18.989761 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:18.989717    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9981a1cd-161f-437f-93f3-877f324f6472-kube-api-access-chzmg\" (UniqueName: \"kubernetes.io/projected/9981a1cd-161f-437f-93f3-877f324f6472-kube-api-access-chzmg\") pod \"9981a1cd-161f-437f-93f3-877f324f6472\" (UID: \"9981a1cd-161f-437f-93f3-877f324f6472\") "
Oct 30 08:04:18.991744 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:18.991719    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9981a1cd-161f-437f-93f3-877f324f6472-kube-api-access-chzmg" pod "9981a1cd-161f-437f-93f3-877f324f6472" (UID: "9981a1cd-161f-437f-93f3-877f324f6472"). InnerVolumeSpecName "kube-api-access-chzmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:04:19.090765 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:19.090737    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chzmg\" (UniqueName: \"kubernetes.io/projected/9981a1cd-161f-437f-93f3-877f324f6472-kube-api-access-chzmg\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:04:19.502182 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:19.502104    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6370f7e2-d6c8-4fb5-8198-32b38837ca31" path="/var/lib/kubelet/pods/6370f7e2-d6c8-4fb5-8198-32b38837ca31/volumes"
Oct 30 08:04:19.502547 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:19.502534    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dd5133c-480f-4fa0-9dd1-27662004aae9" path="/var/lib/kubelet/pods/7dd5133c-480f-4fa0-9dd1-27662004aae9/volumes"
Oct 30 08:04:19.502873 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:19.502860    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9981a1cd-161f-437f-93f3-877f324f6472" path="/var/lib/kubelet/pods/9981a1cd-161f-437f-93f3-877f324f6472/volumes"
Oct 30 08:04:19.787522 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:19.787487    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7427-852b4b"
Oct 30 08:04:19.787522 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:19.787513    6224 scope.go:122] "RemoveContainer" containerID="ff5573c80fe7854591b1f3926d5025448a6a4e287e8b3ba88c4579ac22a171e8"
Oct 30 08:04:23.250302 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.250266    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bgnvh"
Oct 30 08:04:23.291679 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.291645    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hpbmz"
Oct 30 08:04:23.296234 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.296209    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-54xv2"
Oct 30 08:04:23.298663 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.298647    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mn4cr"
Oct 30 08:04:23.336401 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.336364    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14ba5fea-7709-42cb-8fe3-b63093fdcda7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14ba5fea-7709-42cb-8fe3-b63093fdcda7-credential-socket\") pod \"14ba5fea-7709-42cb-8fe3-b63093fdcda7\" (UID: \"14ba5fea-7709-42cb-8fe3-b63093fdcda7\") "
Oct 30 08:04:23.336610 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.336450    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14ba5fea-7709-42cb-8fe3-b63093fdcda7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14ba5fea-7709-42cb-8fe3-b63093fdcda7-istio-envoy\") pod \"14ba5fea-7709-42cb-8fe3-b63093fdcda7\" (UID: \"14ba5fea-7709-42cb-8fe3-b63093fdcda7\") "
Oct 30 08:04:23.336610 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.336484    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14ba5fea-7709-42cb-8fe3-b63093fdcda7-kube-api-access-p8l6t\" (UniqueName: \"kubernetes.io/projected/14ba5fea-7709-42cb-8fe3-b63093fdcda7-kube-api-access-p8l6t\") pod \"14ba5fea-7709-42cb-8fe3-b63093fdcda7\" (UID: \"14ba5fea-7709-42cb-8fe3-b63093fdcda7\") "
Oct 30 08:04:23.336610 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.336549    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14ba5fea-7709-42cb-8fe3-b63093fdcda7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14ba5fea-7709-42cb-8fe3-b63093fdcda7-istiod-ca-cert\") pod \"14ba5fea-7709-42cb-8fe3-b63093fdcda7\" (UID: \"14ba5fea-7709-42cb-8fe3-b63093fdcda7\") "
Oct 30 08:04:23.336610 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.336580    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14ba5fea-7709-42cb-8fe3-b63093fdcda7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14ba5fea-7709-42cb-8fe3-b63093fdcda7-istio-data\") pod \"14ba5fea-7709-42cb-8fe3-b63093fdcda7\" (UID: \"14ba5fea-7709-42cb-8fe3-b63093fdcda7\") "
Oct 30 08:04:23.336940 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.336634    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14ba5fea-7709-42cb-8fe3-b63093fdcda7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14ba5fea-7709-42cb-8fe3-b63093fdcda7-workload-certs\") pod \"14ba5fea-7709-42cb-8fe3-b63093fdcda7\" (UID: \"14ba5fea-7709-42cb-8fe3-b63093fdcda7\") "
Oct 30 08:04:23.336940 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.336699    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14ba5fea-7709-42cb-8fe3-b63093fdcda7-credential-socket" pod "14ba5fea-7709-42cb-8fe3-b63093fdcda7" (UID: "14ba5fea-7709-42cb-8fe3-b63093fdcda7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:04:23.336940 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.336741    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/14ba5fea-7709-42cb-8fe3-b63093fdcda7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14ba5fea-7709-42cb-8fe3-b63093fdcda7-istio-podinfo\") pod \"14ba5fea-7709-42cb-8fe3-b63093fdcda7\" (UID: \"14ba5fea-7709-42cb-8fe3-b63093fdcda7\") "
Oct 30 08:04:23.336940 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.336789    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14ba5fea-7709-42cb-8fe3-b63093fdcda7-istio-token\" (UniqueName: \"kubernetes.io/projected/14ba5fea-7709-42cb-8fe3-b63093fdcda7-istio-token\") pod \"14ba5fea-7709-42cb-8fe3-b63093fdcda7\" (UID: \"14ba5fea-7709-42cb-8fe3-b63093fdcda7\") "
Oct 30 08:04:23.337171 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.336964    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14ba5fea-7709-42cb-8fe3-b63093fdcda7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14ba5fea-7709-42cb-8fe3-b63093fdcda7-workload-socket\") pod \"14ba5fea-7709-42cb-8fe3-b63093fdcda7\" (UID: \"14ba5fea-7709-42cb-8fe3-b63093fdcda7\") "
Oct 30 08:04:23.337171 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.337033    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14ba5fea-7709-42cb-8fe3-b63093fdcda7-istiod-ca-cert" pod "14ba5fea-7709-42cb-8fe3-b63093fdcda7" (UID: "14ba5fea-7709-42cb-8fe3-b63093fdcda7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:04:23.337386 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.337363    6224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14ba5fea-7709-42cb-8fe3-b63093fdcda7-credential-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:04:23.337480 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.337393    6224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14ba5fea-7709-42cb-8fe3-b63093fdcda7-istiod-ca-cert\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:04:23.337480 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.337391    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14ba5fea-7709-42cb-8fe3-b63093fdcda7-workload-socket" pod "14ba5fea-7709-42cb-8fe3-b63093fdcda7" (UID: "14ba5fea-7709-42cb-8fe3-b63093fdcda7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:04:23.337673 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.337648    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14ba5fea-7709-42cb-8fe3-b63093fdcda7-workload-certs" pod "14ba5fea-7709-42cb-8fe3-b63093fdcda7" (UID: "14ba5fea-7709-42cb-8fe3-b63093fdcda7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:04:23.337831 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.337795    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14ba5fea-7709-42cb-8fe3-b63093fdcda7-istio-data" pod "14ba5fea-7709-42cb-8fe3-b63093fdcda7" (UID: "14ba5fea-7709-42cb-8fe3-b63093fdcda7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:04:23.339348 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.339318    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14ba5fea-7709-42cb-8fe3-b63093fdcda7-istio-token" pod "14ba5fea-7709-42cb-8fe3-b63093fdcda7" (UID: "14ba5fea-7709-42cb-8fe3-b63093fdcda7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:04:23.341832 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.339753    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14ba5fea-7709-42cb-8fe3-b63093fdcda7-istio-envoy" pod "14ba5fea-7709-42cb-8fe3-b63093fdcda7" (UID: "14ba5fea-7709-42cb-8fe3-b63093fdcda7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:04:23.341832 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.339797    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/14ba5fea-7709-42cb-8fe3-b63093fdcda7-istio-podinfo" pod "14ba5fea-7709-42cb-8fe3-b63093fdcda7" (UID: "14ba5fea-7709-42cb-8fe3-b63093fdcda7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 08:04:23.341832 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.340043    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14ba5fea-7709-42cb-8fe3-b63093fdcda7-kube-api-access-p8l6t" pod "14ba5fea-7709-42cb-8fe3-b63093fdcda7" (UID: "14ba5fea-7709-42cb-8fe3-b63093fdcda7"). InnerVolumeSpecName "kube-api-access-p8l6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:04:23.438725 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.438691    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5747b4f9-c446-4172-9e98-cea9f7489f2f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5747b4f9-c446-4172-9e98-cea9f7489f2f-workload-socket\") pod \"5747b4f9-c446-4172-9e98-cea9f7489f2f\" (UID: \"5747b4f9-c446-4172-9e98-cea9f7489f2f\") "
Oct 30 08:04:23.438979 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.438738    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb3586a2-80e5-44b7-8c23-bcf26c3a8147-istio-token\" (UniqueName: \"kubernetes.io/projected/bb3586a2-80e5-44b7-8c23-bcf26c3a8147-istio-token\") pod \"bb3586a2-80e5-44b7-8c23-bcf26c3a8147\" (UID: \"bb3586a2-80e5-44b7-8c23-bcf26c3a8147\") "
Oct 30 08:04:23.438979 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.438769    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb3586a2-80e5-44b7-8c23-bcf26c3a8147-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb3586a2-80e5-44b7-8c23-bcf26c3a8147-istio-podinfo\") pod \"bb3586a2-80e5-44b7-8c23-bcf26c3a8147\" (UID: \"bb3586a2-80e5-44b7-8c23-bcf26c3a8147\") "
Oct 30 08:04:23.438979 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.438833    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5747b4f9-c446-4172-9e98-cea9f7489f2f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5747b4f9-c446-4172-9e98-cea9f7489f2f-istio-podinfo\") pod \"5747b4f9-c446-4172-9e98-cea9f7489f2f\" (UID: \"5747b4f9-c446-4172-9e98-cea9f7489f2f\") "
Oct 30 08:04:23.439234 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.439006    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5747b4f9-c446-4172-9e98-cea9f7489f2f-workload-socket" pod "5747b4f9-c446-4172-9e98-cea9f7489f2f" (UID: "5747b4f9-c446-4172-9e98-cea9f7489f2f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:04:23.439234 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.439037    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/522040eb-5fce-4b49-9856-410c83cffd8f-kube-api-access-tkspn\" (UniqueName: \"kubernetes.io/projected/522040eb-5fce-4b49-9856-410c83cffd8f-kube-api-access-tkspn\") pod \"522040eb-5fce-4b49-9856-410c83cffd8f\" (UID: \"522040eb-5fce-4b49-9856-410c83cffd8f\") "
Oct 30 08:04:23.439234 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.439115    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/522040eb-5fce-4b49-9856-410c83cffd8f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/522040eb-5fce-4b49-9856-410c83cffd8f-workload-socket\") pod \"522040eb-5fce-4b49-9856-410c83cffd8f\" (UID: \"522040eb-5fce-4b49-9856-410c83cffd8f\") "
Oct 30 08:04:23.439234 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.439168    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/522040eb-5fce-4b49-9856-410c83cffd8f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/522040eb-5fce-4b49-9856-410c83cffd8f-workload-certs\") pod \"522040eb-5fce-4b49-9856-410c83cffd8f\" (UID: \"522040eb-5fce-4b49-9856-410c83cffd8f\") "
Oct 30 08:04:23.439234 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.439202    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb3586a2-80e5-44b7-8c23-bcf26c3a8147-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3586a2-80e5-44b7-8c23-bcf26c3a8147-credential-socket\") pod \"bb3586a2-80e5-44b7-8c23-bcf26c3a8147\" (UID: \"bb3586a2-80e5-44b7-8c23-bcf26c3a8147\") "
Oct 30 08:04:23.439504 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.439259    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb3586a2-80e5-44b7-8c23-bcf26c3a8147-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3586a2-80e5-44b7-8c23-bcf26c3a8147-workload-socket\") pod \"bb3586a2-80e5-44b7-8c23-bcf26c3a8147\" (UID: \"bb3586a2-80e5-44b7-8c23-bcf26c3a8147\") "
Oct 30 08:04:23.439504 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.439319    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5747b4f9-c446-4172-9e98-cea9f7489f2f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5747b4f9-c446-4172-9e98-cea9f7489f2f-istio-data\") pod \"5747b4f9-c446-4172-9e98-cea9f7489f2f\" (UID: \"5747b4f9-c446-4172-9e98-cea9f7489f2f\") "
Oct 30 08:04:23.439504 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.439354    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb3586a2-80e5-44b7-8c23-bcf26c3a8147-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb3586a2-80e5-44b7-8c23-bcf26c3a8147-istio-data\") pod \"bb3586a2-80e5-44b7-8c23-bcf26c3a8147\" (UID: \"bb3586a2-80e5-44b7-8c23-bcf26c3a8147\") "
Oct 30 08:04:23.439504 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.439431    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/522040eb-5fce-4b49-9856-410c83cffd8f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/522040eb-5fce-4b49-9856-410c83cffd8f-istio-data\") pod \"522040eb-5fce-4b49-9856-410c83cffd8f\" (UID: \"522040eb-5fce-4b49-9856-410c83cffd8f\") "
Oct 30 08:04:23.439504 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.439435    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/522040eb-5fce-4b49-9856-410c83cffd8f-workload-socket" pod "522040eb-5fce-4b49-9856-410c83cffd8f" (UID: "522040eb-5fce-4b49-9856-410c83cffd8f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:04:23.439504 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.439486    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5747b4f9-c446-4172-9e98-cea9f7489f2f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5747b4f9-c446-4172-9e98-cea9f7489f2f-istiod-ca-cert\") pod \"5747b4f9-c446-4172-9e98-cea9f7489f2f\" (UID: \"5747b4f9-c446-4172-9e98-cea9f7489f2f\") "
Oct 30 08:04:23.439861 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.439543    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/522040eb-5fce-4b49-9856-410c83cffd8f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/522040eb-5fce-4b49-9856-410c83cffd8f-istio-podinfo\") pod \"522040eb-5fce-4b49-9856-410c83cffd8f\" (UID: \"522040eb-5fce-4b49-9856-410c83cffd8f\") "
Oct 30 08:04:23.439861 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.439619    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/522040eb-5fce-4b49-9856-410c83cffd8f-workload-certs" pod "522040eb-5fce-4b49-9856-410c83cffd8f" (UID: "522040eb-5fce-4b49-9856-410c83cffd8f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:04:23.439861 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.439637    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5747b4f9-c446-4172-9e98-cea9f7489f2f-istio-data" pod "5747b4f9-c446-4172-9e98-cea9f7489f2f" (UID: "5747b4f9-c446-4172-9e98-cea9f7489f2f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:04:23.439861 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.439674    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb3586a2-80e5-44b7-8c23-bcf26c3a8147-istio-data" pod "bb3586a2-80e5-44b7-8c23-bcf26c3a8147" (UID: "bb3586a2-80e5-44b7-8c23-bcf26c3a8147"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:04:23.440107 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.439847    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb3586a2-80e5-44b7-8c23-bcf26c3a8147-workload-socket" pod "bb3586a2-80e5-44b7-8c23-bcf26c3a8147" (UID: "bb3586a2-80e5-44b7-8c23-bcf26c3a8147"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:04:23.440107 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.439883    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/522040eb-5fce-4b49-9856-410c83cffd8f-istio-data" pod "522040eb-5fce-4b49-9856-410c83cffd8f" (UID: "522040eb-5fce-4b49-9856-410c83cffd8f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:04:23.440107 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.440051    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5747b4f9-c446-4172-9e98-cea9f7489f2f-istiod-ca-cert" pod "5747b4f9-c446-4172-9e98-cea9f7489f2f" (UID: "5747b4f9-c446-4172-9e98-cea9f7489f2f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:04:23.440262 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.440219    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb3586a2-80e5-44b7-8c23-bcf26c3a8147-credential-socket" pod "bb3586a2-80e5-44b7-8c23-bcf26c3a8147" (UID: "bb3586a2-80e5-44b7-8c23-bcf26c3a8147"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:04:23.441385 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.441347    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5747b4f9-c446-4172-9e98-cea9f7489f2f-istio-podinfo" pod "5747b4f9-c446-4172-9e98-cea9f7489f2f" (UID: "5747b4f9-c446-4172-9e98-cea9f7489f2f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 08:04:23.443300 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.441858    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb3586a2-80e5-44b7-8c23-bcf26c3a8147-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb3586a2-80e5-44b7-8c23-bcf26c3a8147-workload-certs\") pod \"bb3586a2-80e5-44b7-8c23-bcf26c3a8147\" (UID: \"bb3586a2-80e5-44b7-8c23-bcf26c3a8147\") "
Oct 30 08:04:23.443300 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.441920    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5747b4f9-c446-4172-9e98-cea9f7489f2f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5747b4f9-c446-4172-9e98-cea9f7489f2f-workload-certs\") pod \"5747b4f9-c446-4172-9e98-cea9f7489f2f\" (UID: \"5747b4f9-c446-4172-9e98-cea9f7489f2f\") "
Oct 30 08:04:23.443300 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.441979    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/522040eb-5fce-4b49-9856-410c83cffd8f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/522040eb-5fce-4b49-9856-410c83cffd8f-istiod-ca-cert\") pod \"522040eb-5fce-4b49-9856-410c83cffd8f\" (UID: \"522040eb-5fce-4b49-9856-410c83cffd8f\") "
Oct 30 08:04:23.443300 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.442010    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/522040eb-5fce-4b49-9856-410c83cffd8f-istio-token\" (UniqueName: \"kubernetes.io/projected/522040eb-5fce-4b49-9856-410c83cffd8f-istio-token\") pod \"522040eb-5fce-4b49-9856-410c83cffd8f\" (UID: \"522040eb-5fce-4b49-9856-410c83cffd8f\") "
Oct 30 08:04:23.443300 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.442041    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb3586a2-80e5-44b7-8c23-bcf26c3a8147-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb3586a2-80e5-44b7-8c23-bcf26c3a8147-istio-envoy\") pod \"bb3586a2-80e5-44b7-8c23-bcf26c3a8147\" (UID: \"bb3586a2-80e5-44b7-8c23-bcf26c3a8147\") "
Oct 30 08:04:23.443300 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.442082    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5747b4f9-c446-4172-9e98-cea9f7489f2f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5747b4f9-c446-4172-9e98-cea9f7489f2f-istio-envoy\") pod \"5747b4f9-c446-4172-9e98-cea9f7489f2f\" (UID: \"5747b4f9-c446-4172-9e98-cea9f7489f2f\") "
Oct 30 08:04:23.443300 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.442112    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb3586a2-80e5-44b7-8c23-bcf26c3a8147-istio-podinfo" pod "bb3586a2-80e5-44b7-8c23-bcf26c3a8147" (UID: "bb3586a2-80e5-44b7-8c23-bcf26c3a8147"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 08:04:23.443300 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.442127    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5747b4f9-c446-4172-9e98-cea9f7489f2f-istio-token\" (UniqueName: \"kubernetes.io/projected/5747b4f9-c446-4172-9e98-cea9f7489f2f-istio-token\") pod \"5747b4f9-c446-4172-9e98-cea9f7489f2f\" (UID: \"5747b4f9-c446-4172-9e98-cea9f7489f2f\") "
Oct 30 08:04:23.443300 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.442157    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb3586a2-80e5-44b7-8c23-bcf26c3a8147-kube-api-access-mdrb5\" (UniqueName: \"kubernetes.io/projected/bb3586a2-80e5-44b7-8c23-bcf26c3a8147-kube-api-access-mdrb5\") pod \"bb3586a2-80e5-44b7-8c23-bcf26c3a8147\" (UID: \"bb3586a2-80e5-44b7-8c23-bcf26c3a8147\") "
Oct 30 08:04:23.443300 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.442206    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5747b4f9-c446-4172-9e98-cea9f7489f2f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5747b4f9-c446-4172-9e98-cea9f7489f2f-credential-socket\") pod \"5747b4f9-c446-4172-9e98-cea9f7489f2f\" (UID: \"5747b4f9-c446-4172-9e98-cea9f7489f2f\") "
Oct 30 08:04:23.443300 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.442207    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5747b4f9-c446-4172-9e98-cea9f7489f2f-workload-certs" pod "5747b4f9-c446-4172-9e98-cea9f7489f2f" (UID: "5747b4f9-c446-4172-9e98-cea9f7489f2f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:04:23.443300 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.442241    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5747b4f9-c446-4172-9e98-cea9f7489f2f-kube-api-access-fkwpj\" (UniqueName: \"kubernetes.io/projected/5747b4f9-c446-4172-9e98-cea9f7489f2f-kube-api-access-fkwpj\") pod \"5747b4f9-c446-4172-9e98-cea9f7489f2f\" (UID: \"5747b4f9-c446-4172-9e98-cea9f7489f2f\") "
Oct 30 08:04:23.443300 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.442274    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb3586a2-80e5-44b7-8c23-bcf26c3a8147-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb3586a2-80e5-44b7-8c23-bcf26c3a8147-istiod-ca-cert\") pod \"bb3586a2-80e5-44b7-8c23-bcf26c3a8147\" (UID: \"bb3586a2-80e5-44b7-8c23-bcf26c3a8147\") "
Oct 30 08:04:23.443300 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.442308    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/522040eb-5fce-4b49-9856-410c83cffd8f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/522040eb-5fce-4b49-9856-410c83cffd8f-istio-envoy\") pod \"522040eb-5fce-4b49-9856-410c83cffd8f\" (UID: \"522040eb-5fce-4b49-9856-410c83cffd8f\") "
Oct 30 08:04:23.443300 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.442349    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/522040eb-5fce-4b49-9856-410c83cffd8f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/522040eb-5fce-4b49-9856-410c83cffd8f-credential-socket\") pod \"522040eb-5fce-4b49-9856-410c83cffd8f\" (UID: \"522040eb-5fce-4b49-9856-410c83cffd8f\") "
Oct 30 08:04:23.443300 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.442492    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb3586a2-80e5-44b7-8c23-bcf26c3a8147-istio-token" pod "bb3586a2-80e5-44b7-8c23-bcf26c3a8147" (UID: "bb3586a2-80e5-44b7-8c23-bcf26c3a8147"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:04:23.443300 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.442504    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/522040eb-5fce-4b49-9856-410c83cffd8f-istio-podinfo" pod "522040eb-5fce-4b49-9856-410c83cffd8f" (UID: "522040eb-5fce-4b49-9856-410c83cffd8f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 08:04:23.443300 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.442694    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/522040eb-5fce-4b49-9856-410c83cffd8f-credential-socket" pod "522040eb-5fce-4b49-9856-410c83cffd8f" (UID: "522040eb-5fce-4b49-9856-410c83cffd8f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:04:23.443300 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.442750    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/522040eb-5fce-4b49-9856-410c83cffd8f-kube-api-access-tkspn" pod "522040eb-5fce-4b49-9856-410c83cffd8f" (UID: "522040eb-5fce-4b49-9856-410c83cffd8f"). InnerVolumeSpecName "kube-api-access-tkspn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:04:23.443300 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.442869    6224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5747b4f9-c446-4172-9e98-cea9f7489f2f-workload-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:04:23.443300 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.442890    6224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14ba5fea-7709-42cb-8fe3-b63093fdcda7-istio-envoy\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:04:23.443300 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.442905    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8l6t\" (UniqueName: \"kubernetes.io/projected/14ba5fea-7709-42cb-8fe3-b63093fdcda7-kube-api-access-p8l6t\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:04:23.443300 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.442920    6224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14ba5fea-7709-42cb-8fe3-b63093fdcda7-istio-data\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:04:23.443300 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.442935    6224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/522040eb-5fce-4b49-9856-410c83cffd8f-credential-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:04:23.443300 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.442950    6224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14ba5fea-7709-42cb-8fe3-b63093fdcda7-workload-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:04:23.443300 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.442963    6224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5747b4f9-c446-4172-9e98-cea9f7489f2f-workload-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:04:23.443300 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.442979    6224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb3586a2-80e5-44b7-8c23-bcf26c3a8147-istio-token\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:04:23.443300 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.442993    6224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb3586a2-80e5-44b7-8c23-bcf26c3a8147-istio-podinfo\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:04:23.443300 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.443009    6224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5747b4f9-c446-4172-9e98-cea9f7489f2f-istio-podinfo\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:04:23.443300 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.443022    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkspn\" (UniqueName: \"kubernetes.io/projected/522040eb-5fce-4b49-9856-410c83cffd8f-kube-api-access-tkspn\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:04:23.443300 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.443036    6224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/522040eb-5fce-4b49-9856-410c83cffd8f-workload-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:04:23.443300 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.443049    6224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/522040eb-5fce-4b49-9856-410c83cffd8f-workload-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:04:23.443300 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.443062    6224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3586a2-80e5-44b7-8c23-bcf26c3a8147-credential-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:04:23.443300 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.443075    6224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3586a2-80e5-44b7-8c23-bcf26c3a8147-workload-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:04:23.443300 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.443095    6224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5747b4f9-c446-4172-9e98-cea9f7489f2f-istio-data\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:04:23.443300 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.443114    6224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14ba5fea-7709-42cb-8fe3-b63093fdcda7-istio-podinfo\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:04:23.443300 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.443130    6224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb3586a2-80e5-44b7-8c23-bcf26c3a8147-istio-data\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:04:23.443300 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.443154    6224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14ba5fea-7709-42cb-8fe3-b63093fdcda7-istio-token\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:04:23.443300 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.443167    6224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/522040eb-5fce-4b49-9856-410c83cffd8f-istio-data\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:04:23.443300 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.443180    6224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5747b4f9-c446-4172-9e98-cea9f7489f2f-istiod-ca-cert\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:04:23.443300 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.443193    6224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14ba5fea-7709-42cb-8fe3-b63093fdcda7-workload-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:04:23.443300 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.443206    6224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/522040eb-5fce-4b49-9856-410c83cffd8f-istio-podinfo\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:04:23.445797 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.443516    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5747b4f9-c446-4172-9e98-cea9f7489f2f-credential-socket" pod "5747b4f9-c446-4172-9e98-cea9f7489f2f" (UID: "5747b4f9-c446-4172-9e98-cea9f7489f2f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:04:23.445797 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.444110    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/522040eb-5fce-4b49-9856-410c83cffd8f-istiod-ca-cert" pod "522040eb-5fce-4b49-9856-410c83cffd8f" (UID: "522040eb-5fce-4b49-9856-410c83cffd8f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:04:23.445797 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.444312    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb3586a2-80e5-44b7-8c23-bcf26c3a8147-istiod-ca-cert" pod "bb3586a2-80e5-44b7-8c23-bcf26c3a8147" (UID: "bb3586a2-80e5-44b7-8c23-bcf26c3a8147"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:04:23.445797 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.444409    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5747b4f9-c446-4172-9e98-cea9f7489f2f-istio-token" pod "5747b4f9-c446-4172-9e98-cea9f7489f2f" (UID: "5747b4f9-c446-4172-9e98-cea9f7489f2f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:04:23.445797 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.444478    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb3586a2-80e5-44b7-8c23-bcf26c3a8147-workload-certs" pod "bb3586a2-80e5-44b7-8c23-bcf26c3a8147" (UID: "bb3586a2-80e5-44b7-8c23-bcf26c3a8147"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:04:23.445797 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.445028    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5747b4f9-c446-4172-9e98-cea9f7489f2f-kube-api-access-fkwpj" pod "5747b4f9-c446-4172-9e98-cea9f7489f2f" (UID: "5747b4f9-c446-4172-9e98-cea9f7489f2f"). InnerVolumeSpecName "kube-api-access-fkwpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:04:23.446166 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.446116    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5747b4f9-c446-4172-9e98-cea9f7489f2f-istio-envoy" pod "5747b4f9-c446-4172-9e98-cea9f7489f2f" (UID: "5747b4f9-c446-4172-9e98-cea9f7489f2f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:04:23.446166 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.446139    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/522040eb-5fce-4b49-9856-410c83cffd8f-istio-token" pod "522040eb-5fce-4b49-9856-410c83cffd8f" (UID: "522040eb-5fce-4b49-9856-410c83cffd8f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:04:23.446316 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.446291    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb3586a2-80e5-44b7-8c23-bcf26c3a8147-istio-envoy" pod "bb3586a2-80e5-44b7-8c23-bcf26c3a8147" (UID: "bb3586a2-80e5-44b7-8c23-bcf26c3a8147"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:04:23.446793 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.446768    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb3586a2-80e5-44b7-8c23-bcf26c3a8147-kube-api-access-mdrb5" pod "bb3586a2-80e5-44b7-8c23-bcf26c3a8147" (UID: "bb3586a2-80e5-44b7-8c23-bcf26c3a8147"). InnerVolumeSpecName "kube-api-access-mdrb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:04:23.447414 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.447388    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/522040eb-5fce-4b49-9856-410c83cffd8f-istio-envoy" pod "522040eb-5fce-4b49-9856-410c83cffd8f" (UID: "522040eb-5fce-4b49-9856-410c83cffd8f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:04:23.544746 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.544706    6224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb3586a2-80e5-44b7-8c23-bcf26c3a8147-workload-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:04:23.544746 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.544739    6224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/522040eb-5fce-4b49-9856-410c83cffd8f-istiod-ca-cert\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:04:23.544746 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.544752    6224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/522040eb-5fce-4b49-9856-410c83cffd8f-istio-token\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:04:23.545004 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.544763    6224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb3586a2-80e5-44b7-8c23-bcf26c3a8147-istio-envoy\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:04:23.545004 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.544777    6224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5747b4f9-c446-4172-9e98-cea9f7489f2f-istio-envoy\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:04:23.545004 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.544789    6224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5747b4f9-c446-4172-9e98-cea9f7489f2f-istio-token\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:04:23.545004 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.544800    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdrb5\" (UniqueName: \"kubernetes.io/projected/bb3586a2-80e5-44b7-8c23-bcf26c3a8147-kube-api-access-mdrb5\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:04:23.545004 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.544826    6224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5747b4f9-c446-4172-9e98-cea9f7489f2f-credential-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:04:23.545004 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.544839    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkwpj\" (UniqueName: \"kubernetes.io/projected/5747b4f9-c446-4172-9e98-cea9f7489f2f-kube-api-access-fkwpj\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:04:23.545004 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.544850    6224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb3586a2-80e5-44b7-8c23-bcf26c3a8147-istiod-ca-cert\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:04:23.545004 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.544863    6224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/522040eb-5fce-4b49-9856-410c83cffd8f-istio-envoy\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:04:23.814671 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.814636    6224 generic.go:356] "Generic (PLEG): container finished" podID="5747b4f9-c446-4172-9e98-cea9f7489f2f" containerID="eb6baacd206507e426943b8df3d65b1fdf6f43cdc9bbced3cd49f433751dfb7a" exitCode=0
Oct 30 08:04:23.814899 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.814697    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mn4cr"
Oct 30 08:04:23.814899 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.814734    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mn4cr" event={"ID":"5747b4f9-c446-4172-9e98-cea9f7489f2f","Type":"ContainerDied","Data":"eb6baacd206507e426943b8df3d65b1fdf6f43cdc9bbced3cd49f433751dfb7a"}
Oct 30 08:04:23.814899 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.814792    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mn4cr" event={"ID":"5747b4f9-c446-4172-9e98-cea9f7489f2f","Type":"ContainerDied","Data":"23f29004bcaf5b58880392789496652f4090fef76dfd941af701537258760c08"}
Oct 30 08:04:23.814899 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.814842    6224 scope.go:122] "RemoveContainer" containerID="eb6baacd206507e426943b8df3d65b1fdf6f43cdc9bbced3cd49f433751dfb7a"
Oct 30 08:04:23.816254 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.816230    6224 generic.go:356] "Generic (PLEG): container finished" podID="14ba5fea-7709-42cb-8fe3-b63093fdcda7" containerID="5e8f88e9e3c524f1b361251c3f0b6e8ac09d68ed2d5bf237a767580b1076d8fe" exitCode=0
Oct 30 08:04:23.816355 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.816271    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bgnvh"
Oct 30 08:04:23.816355 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.816324    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bgnvh" event={"ID":"14ba5fea-7709-42cb-8fe3-b63093fdcda7","Type":"ContainerDied","Data":"5e8f88e9e3c524f1b361251c3f0b6e8ac09d68ed2d5bf237a767580b1076d8fe"}
Oct 30 08:04:23.816503 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.816367    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bgnvh" event={"ID":"14ba5fea-7709-42cb-8fe3-b63093fdcda7","Type":"ContainerDied","Data":"17f9db5ae43882b3d346317d79989841e9b4f07b9e1acbba7997553f6f5aaea2"}
Oct 30 08:04:23.817972 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.817947    6224 generic.go:356] "Generic (PLEG): container finished" podID="522040eb-5fce-4b49-9856-410c83cffd8f" containerID="6acbf2763ed95634cb7c9b74684e73272afdd2fd7735ee2e1cb32e493d6392c9" exitCode=0
Oct 30 08:04:23.818127 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.817991    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hpbmz"
Oct 30 08:04:23.818127 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.817994    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hpbmz" event={"ID":"522040eb-5fce-4b49-9856-410c83cffd8f","Type":"ContainerDied","Data":"6acbf2763ed95634cb7c9b74684e73272afdd2fd7735ee2e1cb32e493d6392c9"}
Oct 30 08:04:23.818127 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.818038    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hpbmz" event={"ID":"522040eb-5fce-4b49-9856-410c83cffd8f","Type":"ContainerDied","Data":"884862cf8c20ec3e042285f005223b7b71c2343f50e5b1f5c843eacfda93c571"}
Oct 30 08:04:23.819551 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.819525    6224 generic.go:356] "Generic (PLEG): container finished" podID="bb3586a2-80e5-44b7-8c23-bcf26c3a8147" containerID="479382ec07779630bc6b07691d6c931ab608ab847fdf9b998e282c6c9c9f5617" exitCode=0
Oct 30 08:04:23.819677 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.819554    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-54xv2" event={"ID":"bb3586a2-80e5-44b7-8c23-bcf26c3a8147","Type":"ContainerDied","Data":"479382ec07779630bc6b07691d6c931ab608ab847fdf9b998e282c6c9c9f5617"}
Oct 30 08:04:23.819677 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.819572    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-54xv2"
Oct 30 08:04:23.819677 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.819592    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-54xv2" event={"ID":"bb3586a2-80e5-44b7-8c23-bcf26c3a8147","Type":"ContainerDied","Data":"84e2629232b6d8285e9ca27c5fc21c6687011f36738799c5e37194d5a39074dd"}
Oct 30 08:04:23.826281 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.826262    6224 scope.go:122] "RemoveContainer" containerID="eb6baacd206507e426943b8df3d65b1fdf6f43cdc9bbced3cd49f433751dfb7a"
Oct 30 08:04:23.826540 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:04:23.826518    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb6baacd206507e426943b8df3d65b1fdf6f43cdc9bbced3cd49f433751dfb7a\": container with ID starting with eb6baacd206507e426943b8df3d65b1fdf6f43cdc9bbced3cd49f433751dfb7a not found: ID does not exist" containerID="eb6baacd206507e426943b8df3d65b1fdf6f43cdc9bbced3cd49f433751dfb7a"
Oct 30 08:04:23.826597 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.826552    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb6baacd206507e426943b8df3d65b1fdf6f43cdc9bbced3cd49f433751dfb7a"} err="failed to get container status \"eb6baacd206507e426943b8df3d65b1fdf6f43cdc9bbced3cd49f433751dfb7a\": rpc error: code = NotFound desc = could not find container \"eb6baacd206507e426943b8df3d65b1fdf6f43cdc9bbced3cd49f433751dfb7a\": container with ID starting with eb6baacd206507e426943b8df3d65b1fdf6f43cdc9bbced3cd49f433751dfb7a not found: ID does not exist"
Oct 30 08:04:23.826597 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.826571    6224 scope.go:122] "RemoveContainer" containerID="5e8f88e9e3c524f1b361251c3f0b6e8ac09d68ed2d5bf237a767580b1076d8fe"
Oct 30 08:04:23.838005 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.837986    6224 scope.go:122] "RemoveContainer" containerID="5e8f88e9e3c524f1b361251c3f0b6e8ac09d68ed2d5bf237a767580b1076d8fe"
Oct 30 08:04:23.838303 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:04:23.838279    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e8f88e9e3c524f1b361251c3f0b6e8ac09d68ed2d5bf237a767580b1076d8fe\": container with ID starting with 5e8f88e9e3c524f1b361251c3f0b6e8ac09d68ed2d5bf237a767580b1076d8fe not found: ID does not exist" containerID="5e8f88e9e3c524f1b361251c3f0b6e8ac09d68ed2d5bf237a767580b1076d8fe"
Oct 30 08:04:23.838351 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.838311    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e8f88e9e3c524f1b361251c3f0b6e8ac09d68ed2d5bf237a767580b1076d8fe"} err="failed to get container status \"5e8f88e9e3c524f1b361251c3f0b6e8ac09d68ed2d5bf237a767580b1076d8fe\": rpc error: code = NotFound desc = could not find container \"5e8f88e9e3c524f1b361251c3f0b6e8ac09d68ed2d5bf237a767580b1076d8fe\": container with ID starting with 5e8f88e9e3c524f1b361251c3f0b6e8ac09d68ed2d5bf237a767580b1076d8fe not found: ID does not exist"
Oct 30 08:04:23.838351 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.838329    6224 scope.go:122] "RemoveContainer" containerID="6acbf2763ed95634cb7c9b74684e73272afdd2fd7735ee2e1cb32e493d6392c9"
Oct 30 08:04:23.846981 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.846964    6224 scope.go:122] "RemoveContainer" containerID="6acbf2763ed95634cb7c9b74684e73272afdd2fd7735ee2e1cb32e493d6392c9"
Oct 30 08:04:23.847219 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:04:23.847203    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6acbf2763ed95634cb7c9b74684e73272afdd2fd7735ee2e1cb32e493d6392c9\": container with ID starting with 6acbf2763ed95634cb7c9b74684e73272afdd2fd7735ee2e1cb32e493d6392c9 not found: ID does not exist" containerID="6acbf2763ed95634cb7c9b74684e73272afdd2fd7735ee2e1cb32e493d6392c9"
Oct 30 08:04:23.847272 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.847223    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6acbf2763ed95634cb7c9b74684e73272afdd2fd7735ee2e1cb32e493d6392c9"} err="failed to get container status \"6acbf2763ed95634cb7c9b74684e73272afdd2fd7735ee2e1cb32e493d6392c9\": rpc error: code = NotFound desc = could not find container \"6acbf2763ed95634cb7c9b74684e73272afdd2fd7735ee2e1cb32e493d6392c9\": container with ID starting with 6acbf2763ed95634cb7c9b74684e73272afdd2fd7735ee2e1cb32e493d6392c9 not found: ID does not exist"
Oct 30 08:04:23.847272 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.847240    6224 scope.go:122] "RemoveContainer" containerID="479382ec07779630bc6b07691d6c931ab608ab847fdf9b998e282c6c9c9f5617"
Oct 30 08:04:23.855998 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.855981    6224 scope.go:122] "RemoveContainer" containerID="479382ec07779630bc6b07691d6c931ab608ab847fdf9b998e282c6c9c9f5617"
Oct 30 08:04:23.856273 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:04:23.856252    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"479382ec07779630bc6b07691d6c931ab608ab847fdf9b998e282c6c9c9f5617\": container with ID starting with 479382ec07779630bc6b07691d6c931ab608ab847fdf9b998e282c6c9c9f5617 not found: ID does not exist" containerID="479382ec07779630bc6b07691d6c931ab608ab847fdf9b998e282c6c9c9f5617"
Oct 30 08:04:23.856347 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.856284    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"479382ec07779630bc6b07691d6c931ab608ab847fdf9b998e282c6c9c9f5617"} err="failed to get container status \"479382ec07779630bc6b07691d6c931ab608ab847fdf9b998e282c6c9c9f5617\": rpc error: code = NotFound desc = could not find container \"479382ec07779630bc6b07691d6c931ab608ab847fdf9b998e282c6c9c9f5617\": container with ID starting with 479382ec07779630bc6b07691d6c931ab608ab847fdf9b998e282c6c9c9f5617 not found: ID does not exist"
Oct 30 08:04:23.874241 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.874219    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-54xv2"]
Oct 30 08:04:23.888531 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.888501    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-54xv2"]
Oct 30 08:04:23.918141 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.918113    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hpbmz"]
Oct 30 08:04:23.921357 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.921335    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hpbmz"]
Oct 30 08:04:23.962479 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.962455    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bgnvh"]
Oct 30 08:04:23.964384 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:23.964361    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bgnvh"]
Oct 30 08:04:24.002728 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:24.002702    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mn4cr"]
Oct 30 08:04:24.008009 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:24.007989    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mn4cr"]
Oct 30 08:04:25.502851 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:25.502790    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14ba5fea-7709-42cb-8fe3-b63093fdcda7" path="/var/lib/kubelet/pods/14ba5fea-7709-42cb-8fe3-b63093fdcda7/volumes"
Oct 30 08:04:25.503429 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:25.503412    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="522040eb-5fce-4b49-9856-410c83cffd8f" path="/var/lib/kubelet/pods/522040eb-5fce-4b49-9856-410c83cffd8f/volumes"
Oct 30 08:04:25.503889 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:25.503875    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5747b4f9-c446-4172-9e98-cea9f7489f2f" path="/var/lib/kubelet/pods/5747b4f9-c446-4172-9e98-cea9f7489f2f/volumes"
Oct 30 08:04:25.504283 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:25.504270    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb3586a2-80e5-44b7-8c23-bcf26c3a8147" path="/var/lib/kubelet/pods/bb3586a2-80e5-44b7-8c23-bcf26c3a8147/volumes"
Oct 30 08:04:29.357570 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:29.357532    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 08:04:31.857570 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:04:31.857531    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 08:04:35.667462 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:35.667428    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq"]
Oct 30 08:04:35.667889 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:35.667849    6224 cpu_manager.go:414] "containerMap: removing container" podUID="9981a1cd-161f-437f-93f3-877f324f6472" containerName="test-curl-7427-852b4b"
Oct 30 08:04:35.667889 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:35.667861    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9981a1cd-161f-437f-93f3-877f324f6472" containerName="test-curl-7427-852b4b"
Oct 30 08:04:35.667889 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:35.667883    6224 cpu_manager.go:414] "containerMap: removing container" podUID="522040eb-5fce-4b49-9856-410c83cffd8f" containerName="istio-proxy"
Oct 30 08:04:35.667889 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:35.667888    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="522040eb-5fce-4b49-9856-410c83cffd8f" containerName="istio-proxy"
Oct 30 08:04:35.668036 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:35.667903    6224 cpu_manager.go:414] "containerMap: removing container" podUID="7dd5133c-480f-4fa0-9dd1-27662004aae9" containerName="main"
Oct 30 08:04:35.668036 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:35.667909    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dd5133c-480f-4fa0-9dd1-27662004aae9" containerName="main"
Oct 30 08:04:35.668036 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:35.667919    6224 cpu_manager.go:414] "containerMap: removing container" podUID="14ba5fea-7709-42cb-8fe3-b63093fdcda7" containerName="istio-proxy"
Oct 30 08:04:35.668036 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:35.667924    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14ba5fea-7709-42cb-8fe3-b63093fdcda7" containerName="istio-proxy"
Oct 30 08:04:35.668036 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:35.667931    6224 cpu_manager.go:414] "containerMap: removing container" podUID="5747b4f9-c446-4172-9e98-cea9f7489f2f" containerName="istio-proxy"
Oct 30 08:04:35.668036 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:35.667935    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5747b4f9-c446-4172-9e98-cea9f7489f2f" containerName="istio-proxy"
Oct 30 08:04:35.668036 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:35.667943    6224 cpu_manager.go:414] "containerMap: removing container" podUID="6370f7e2-d6c8-4fb5-8198-32b38837ca31" containerName="main"
Oct 30 08:04:35.668036 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:35.667948    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6370f7e2-d6c8-4fb5-8198-32b38837ca31" containerName="main"
Oct 30 08:04:35.668036 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:35.667957    6224 cpu_manager.go:414] "containerMap: removing container" podUID="bb3586a2-80e5-44b7-8c23-bcf26c3a8147" containerName="istio-proxy"
Oct 30 08:04:35.668036 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:35.667962    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb3586a2-80e5-44b7-8c23-bcf26c3a8147" containerName="istio-proxy"
Oct 30 08:04:35.668036 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:35.668019    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="9981a1cd-161f-437f-93f3-877f324f6472" containerName="test-curl-7427-852b4b"
Oct 30 08:04:35.668036 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:35.668028    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="522040eb-5fce-4b49-9856-410c83cffd8f" containerName="istio-proxy"
Oct 30 08:04:35.668036 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:35.668037    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dd5133c-480f-4fa0-9dd1-27662004aae9" containerName="main"
Oct 30 08:04:35.668441 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:35.668045    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="14ba5fea-7709-42cb-8fe3-b63093fdcda7" containerName="istio-proxy"
Oct 30 08:04:35.668441 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:35.668052    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="5747b4f9-c446-4172-9e98-cea9f7489f2f" containerName="istio-proxy"
Oct 30 08:04:35.668441 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:35.668057    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="6370f7e2-d6c8-4fb5-8198-32b38837ca31" containerName="main"
Oct 30 08:04:35.668441 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:35.668064    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb3586a2-80e5-44b7-8c23-bcf26c3a8147" containerName="istio-proxy"
Oct 30 08:04:35.671433 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:35.671402    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq"
Oct 30 08:04:35.688871 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:35.688844    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-bd8589-openshift-default-dockercfg-72qhh\""
Oct 30 08:04:35.705182 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:35.705155    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq"]
Oct 30 08:04:35.865164 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:35.865120    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvx62\" (UniqueName: \"kubernetes.io/projected/61643ced-f0bd-472a-b42b-abb02994320c-kube-api-access-gvx62\") pod \"e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq\" (UID: \"61643ced-f0bd-472a-b42b-abb02994320c\") " pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq"
Oct 30 08:04:35.865352 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:35.865172    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61643ced-f0bd-472a-b42b-abb02994320c-istio-data\") pod \"e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq\" (UID: \"61643ced-f0bd-472a-b42b-abb02994320c\") " pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq"
Oct 30 08:04:35.865352 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:35.865267    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61643ced-f0bd-472a-b42b-abb02994320c-istiod-ca-cert\") pod \"e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq\" (UID: \"61643ced-f0bd-472a-b42b-abb02994320c\") " pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq"
Oct 30 08:04:35.865352 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:35.865298    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61643ced-f0bd-472a-b42b-abb02994320c-istio-podinfo\") pod \"e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq\" (UID: \"61643ced-f0bd-472a-b42b-abb02994320c\") " pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq"
Oct 30 08:04:35.865352 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:35.865330    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61643ced-f0bd-472a-b42b-abb02994320c-istio-envoy\") pod \"e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq\" (UID: \"61643ced-f0bd-472a-b42b-abb02994320c\") " pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq"
Oct 30 08:04:35.865621 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:35.865458    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61643ced-f0bd-472a-b42b-abb02994320c-workload-socket\") pod \"e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq\" (UID: \"61643ced-f0bd-472a-b42b-abb02994320c\") " pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq"
Oct 30 08:04:35.865621 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:35.865513    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61643ced-f0bd-472a-b42b-abb02994320c-credential-socket\") pod \"e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq\" (UID: \"61643ced-f0bd-472a-b42b-abb02994320c\") " pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq"
Oct 30 08:04:35.865621 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:35.865560    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61643ced-f0bd-472a-b42b-abb02994320c-workload-certs\") pod \"e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq\" (UID: \"61643ced-f0bd-472a-b42b-abb02994320c\") " pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq"
Oct 30 08:04:35.865621 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:35.865593    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61643ced-f0bd-472a-b42b-abb02994320c-istio-token\") pod \"e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq\" (UID: \"61643ced-f0bd-472a-b42b-abb02994320c\") " pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq"
Oct 30 08:04:35.967170 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:35.967092    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61643ced-f0bd-472a-b42b-abb02994320c-istio-podinfo\") pod \"e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq\" (UID: \"61643ced-f0bd-472a-b42b-abb02994320c\") " pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq"
Oct 30 08:04:35.967170 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:35.967137    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61643ced-f0bd-472a-b42b-abb02994320c-istio-envoy\") pod \"e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq\" (UID: \"61643ced-f0bd-472a-b42b-abb02994320c\") " pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq"
Oct 30 08:04:35.967386 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:35.967229    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61643ced-f0bd-472a-b42b-abb02994320c-workload-socket\") pod \"e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq\" (UID: \"61643ced-f0bd-472a-b42b-abb02994320c\") " pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq"
Oct 30 08:04:35.967386 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:35.967256    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61643ced-f0bd-472a-b42b-abb02994320c-credential-socket\") pod \"e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq\" (UID: \"61643ced-f0bd-472a-b42b-abb02994320c\") " pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq"
Oct 30 08:04:35.967386 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:35.967278    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61643ced-f0bd-472a-b42b-abb02994320c-workload-certs\") pod \"e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq\" (UID: \"61643ced-f0bd-472a-b42b-abb02994320c\") " pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq"
Oct 30 08:04:35.967386 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:35.967296    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61643ced-f0bd-472a-b42b-abb02994320c-istio-token\") pod \"e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq\" (UID: \"61643ced-f0bd-472a-b42b-abb02994320c\") " pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq"
Oct 30 08:04:35.967386 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:35.967322    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvx62\" (UniqueName: \"kubernetes.io/projected/61643ced-f0bd-472a-b42b-abb02994320c-kube-api-access-gvx62\") pod \"e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq\" (UID: \"61643ced-f0bd-472a-b42b-abb02994320c\") " pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq"
Oct 30 08:04:35.967386 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:35.967345    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61643ced-f0bd-472a-b42b-abb02994320c-istio-data\") pod \"e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq\" (UID: \"61643ced-f0bd-472a-b42b-abb02994320c\") " pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq"
Oct 30 08:04:35.967714 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:35.967403    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61643ced-f0bd-472a-b42b-abb02994320c-istiod-ca-cert\") pod \"e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq\" (UID: \"61643ced-f0bd-472a-b42b-abb02994320c\") " pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq"
Oct 30 08:04:35.967771 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:35.967716    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61643ced-f0bd-472a-b42b-abb02994320c-credential-socket\") pod \"e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq\" (UID: \"61643ced-f0bd-472a-b42b-abb02994320c\") " pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq"
Oct 30 08:04:35.967771 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:35.967748    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61643ced-f0bd-472a-b42b-abb02994320c-workload-socket\") pod \"e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq\" (UID: \"61643ced-f0bd-472a-b42b-abb02994320c\") " pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq"
Oct 30 08:04:35.967927 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:35.967869    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61643ced-f0bd-472a-b42b-abb02994320c-workload-certs\") pod \"e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq\" (UID: \"61643ced-f0bd-472a-b42b-abb02994320c\") " pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq"
Oct 30 08:04:35.967927 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:35.967909    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61643ced-f0bd-472a-b42b-abb02994320c-istio-data\") pod \"e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq\" (UID: \"61643ced-f0bd-472a-b42b-abb02994320c\") " pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq"
Oct 30 08:04:35.968155 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:35.968130    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61643ced-f0bd-472a-b42b-abb02994320c-istiod-ca-cert\") pod \"e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq\" (UID: \"61643ced-f0bd-472a-b42b-abb02994320c\") " pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq"
Oct 30 08:04:35.970017 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:35.969992    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61643ced-f0bd-472a-b42b-abb02994320c-istio-envoy\") pod \"e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq\" (UID: \"61643ced-f0bd-472a-b42b-abb02994320c\") " pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq"
Oct 30 08:04:35.970230 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:35.970211    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61643ced-f0bd-472a-b42b-abb02994320c-istio-podinfo\") pod \"e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq\" (UID: \"61643ced-f0bd-472a-b42b-abb02994320c\") " pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq"
Oct 30 08:04:35.993106 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:35.993075    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61643ced-f0bd-472a-b42b-abb02994320c-istio-token\") pod \"e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq\" (UID: \"61643ced-f0bd-472a-b42b-abb02994320c\") " pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq"
Oct 30 08:04:35.993338 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:35.993312    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvx62\" (UniqueName: \"kubernetes.io/projected/61643ced-f0bd-472a-b42b-abb02994320c-kube-api-access-gvx62\") pod \"e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq\" (UID: \"61643ced-f0bd-472a-b42b-abb02994320c\") " pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq"
Oct 30 08:04:36.285373 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:36.285340    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq"
Oct 30 08:04:36.617152 ip-10-0-128-125 kubenswrapper[6224]: W1030 08:04:36.617113    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61643ced_f0bd_472a_b42b_abb02994320c.slice/crio-a9b4e2d1fc2148c7ed2c50c77b120e6efb9b00a943e970832a2b921f3eb1cc0d WatchSource:0}: Error finding container a9b4e2d1fc2148c7ed2c50c77b120e6efb9b00a943e970832a2b921f3eb1cc0d: Status 404 returned error can't find the container with id a9b4e2d1fc2148c7ed2c50c77b120e6efb9b00a943e970832a2b921f3eb1cc0d
Oct 30 08:04:36.617334 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:36.617311    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq"]
Oct 30 08:04:36.619410 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:36.619380    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 08:04:36.619563 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:36.619493    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 08:04:36.619639 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:36.619594    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 08:04:36.900888 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:36.900785    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq" event={"ID":"61643ced-f0bd-472a-b42b-abb02994320c","Type":"ContainerStarted","Data":"403af861fff15770b5656b90f6f649ab7a2207b6e236d93c3365c90fc00f7f17"}
Oct 30 08:04:36.900888 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:36.900858    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq" event={"ID":"61643ced-f0bd-472a-b42b-abb02994320c","Type":"ContainerStarted","Data":"a9b4e2d1fc2148c7ed2c50c77b120e6efb9b00a943e970832a2b921f3eb1cc0d"}
Oct 30 08:04:36.955868 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:36.955821    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq" podStartSLOduration=1.955795582 podStartE2EDuration="1.955795582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:04:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:04:36.953879181 +0000 UTC m=+4035.563286152" watchObservedRunningTime="2026-10-30 08:04:36.955795582 +0000 UTC m=+4035.565202551"
Oct 30 08:04:37.074396 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:37.074355    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz"]
Oct 30 08:04:37.106832 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:37.103341    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz"]
Oct 30 08:04:37.106832 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:37.103459    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz"
Oct 30 08:04:37.123417 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:37.123388    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-b5998f-openshift-default-dockercfg-h44g7\""
Oct 30 08:04:37.188883 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:37.188746    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f15b9c95-390e-4258-a66e-0a03806392d9-istio-envoy\") pod \"e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz\" (UID: \"f15b9c95-390e-4258-a66e-0a03806392d9\") " pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz"
Oct 30 08:04:37.188883 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:37.188834    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q67lh\" (UniqueName: \"kubernetes.io/projected/f15b9c95-390e-4258-a66e-0a03806392d9-kube-api-access-q67lh\") pod \"e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz\" (UID: \"f15b9c95-390e-4258-a66e-0a03806392d9\") " pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz"
Oct 30 08:04:37.189157 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:37.188938    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f15b9c95-390e-4258-a66e-0a03806392d9-istio-data\") pod \"e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz\" (UID: \"f15b9c95-390e-4258-a66e-0a03806392d9\") " pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz"
Oct 30 08:04:37.189157 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:37.189006    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f15b9c95-390e-4258-a66e-0a03806392d9-credential-socket\") pod \"e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz\" (UID: \"f15b9c95-390e-4258-a66e-0a03806392d9\") " pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz"
Oct 30 08:04:37.189157 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:37.189039    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f15b9c95-390e-4258-a66e-0a03806392d9-istiod-ca-cert\") pod \"e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz\" (UID: \"f15b9c95-390e-4258-a66e-0a03806392d9\") " pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz"
Oct 30 08:04:37.189157 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:37.189133    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f15b9c95-390e-4258-a66e-0a03806392d9-istio-token\") pod \"e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz\" (UID: \"f15b9c95-390e-4258-a66e-0a03806392d9\") " pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz"
Oct 30 08:04:37.189382 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:37.189172    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f15b9c95-390e-4258-a66e-0a03806392d9-workload-certs\") pod \"e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz\" (UID: \"f15b9c95-390e-4258-a66e-0a03806392d9\") " pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz"
Oct 30 08:04:37.189382 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:37.189200    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f15b9c95-390e-4258-a66e-0a03806392d9-istio-podinfo\") pod \"e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz\" (UID: \"f15b9c95-390e-4258-a66e-0a03806392d9\") " pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz"
Oct 30 08:04:37.189382 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:37.189279    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f15b9c95-390e-4258-a66e-0a03806392d9-workload-socket\") pod \"e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz\" (UID: \"f15b9c95-390e-4258-a66e-0a03806392d9\") " pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz"
Oct 30 08:04:37.288046 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:37.288000    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq"
Oct 30 08:04:37.290056 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:37.290026    6224 patch_prober.go:29] interesting pod/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body=
Oct 30 08:04:37.290208 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:37.290078    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq" podUID="61643ced-f0bd-472a-b42b-abb02994320c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused"
Oct 30 08:04:37.290440 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:37.290411    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f15b9c95-390e-4258-a66e-0a03806392d9-istio-envoy\") pod \"e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz\" (UID: \"f15b9c95-390e-4258-a66e-0a03806392d9\") " pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz"
Oct 30 08:04:37.290529 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:37.290458    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q67lh\" (UniqueName: \"kubernetes.io/projected/f15b9c95-390e-4258-a66e-0a03806392d9-kube-api-access-q67lh\") pod \"e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz\" (UID: \"f15b9c95-390e-4258-a66e-0a03806392d9\") " pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz"
Oct 30 08:04:37.290529 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:37.290496    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f15b9c95-390e-4258-a66e-0a03806392d9-istio-data\") pod \"e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz\" (UID: \"f15b9c95-390e-4258-a66e-0a03806392d9\") " pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz"
Oct 30 08:04:37.290725 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:37.290577    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f15b9c95-390e-4258-a66e-0a03806392d9-credential-socket\") pod \"e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz\" (UID: \"f15b9c95-390e-4258-a66e-0a03806392d9\") " pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz"
Oct 30 08:04:37.290725 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:37.290622    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f15b9c95-390e-4258-a66e-0a03806392d9-istiod-ca-cert\") pod \"e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz\" (UID: \"f15b9c95-390e-4258-a66e-0a03806392d9\") " pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz"
Oct 30 08:04:37.290725 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:37.290667    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f15b9c95-390e-4258-a66e-0a03806392d9-istio-token\") pod \"e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz\" (UID: \"f15b9c95-390e-4258-a66e-0a03806392d9\") " pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz"
Oct 30 08:04:37.290725 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:37.290696    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f15b9c95-390e-4258-a66e-0a03806392d9-workload-certs\") pod \"e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz\" (UID: \"f15b9c95-390e-4258-a66e-0a03806392d9\") " pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz"
Oct 30 08:04:37.290990 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:37.290734    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f15b9c95-390e-4258-a66e-0a03806392d9-istio-podinfo\") pod \"e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz\" (UID: \"f15b9c95-390e-4258-a66e-0a03806392d9\") " pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz"
Oct 30 08:04:37.290990 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:37.290759    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f15b9c95-390e-4258-a66e-0a03806392d9-workload-socket\") pod \"e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz\" (UID: \"f15b9c95-390e-4258-a66e-0a03806392d9\") " pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz"
Oct 30 08:04:37.291102 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:37.291063    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f15b9c95-390e-4258-a66e-0a03806392d9-istio-data\") pod \"e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz\" (UID: \"f15b9c95-390e-4258-a66e-0a03806392d9\") " pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz"
Oct 30 08:04:37.291102 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:37.291087    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f15b9c95-390e-4258-a66e-0a03806392d9-credential-socket\") pod \"e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz\" (UID: \"f15b9c95-390e-4258-a66e-0a03806392d9\") " pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz"
Oct 30 08:04:37.291235 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:37.291129    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f15b9c95-390e-4258-a66e-0a03806392d9-workload-socket\") pod \"e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz\" (UID: \"f15b9c95-390e-4258-a66e-0a03806392d9\") " pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz"
Oct 30 08:04:37.291440 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:37.291413    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f15b9c95-390e-4258-a66e-0a03806392d9-workload-certs\") pod \"e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz\" (UID: \"f15b9c95-390e-4258-a66e-0a03806392d9\") " pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz"
Oct 30 08:04:37.291694 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:37.291674    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f15b9c95-390e-4258-a66e-0a03806392d9-istiod-ca-cert\") pod \"e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz\" (UID: \"f15b9c95-390e-4258-a66e-0a03806392d9\") " pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz"
Oct 30 08:04:37.292900 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:37.292879    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f15b9c95-390e-4258-a66e-0a03806392d9-istio-envoy\") pod \"e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz\" (UID: \"f15b9c95-390e-4258-a66e-0a03806392d9\") " pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz"
Oct 30 08:04:37.293492 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:37.293471    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f15b9c95-390e-4258-a66e-0a03806392d9-istio-podinfo\") pod \"e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz\" (UID: \"f15b9c95-390e-4258-a66e-0a03806392d9\") " pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz"
Oct 30 08:04:37.316505 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:37.316478    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f15b9c95-390e-4258-a66e-0a03806392d9-istio-token\") pod \"e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz\" (UID: \"f15b9c95-390e-4258-a66e-0a03806392d9\") " pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz"
Oct 30 08:04:37.316593 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:37.316545    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q67lh\" (UniqueName: \"kubernetes.io/projected/f15b9c95-390e-4258-a66e-0a03806392d9-kube-api-access-q67lh\") pod \"e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz\" (UID: \"f15b9c95-390e-4258-a66e-0a03806392d9\") " pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz"
Oct 30 08:04:37.419162 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:37.419122    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz"
Oct 30 08:04:37.754898 ip-10-0-128-125 kubenswrapper[6224]: W1030 08:04:37.754861    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf15b9c95_390e_4258_a66e_0a03806392d9.slice/crio-837b2ec2c6615aa7bbda9aec1f57b1112844aa1945adf45896469be6eff55a53 WatchSource:0}: Error finding container 837b2ec2c6615aa7bbda9aec1f57b1112844aa1945adf45896469be6eff55a53: Status 404 returned error can't find the container with id 837b2ec2c6615aa7bbda9aec1f57b1112844aa1945adf45896469be6eff55a53
Oct 30 08:04:37.755738 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:37.755708    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz"]
Oct 30 08:04:37.757984 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:37.757939    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 08:04:37.758076 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:37.758032    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 08:04:37.758115 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:37.758075    6224 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 08:04:37.909906 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:37.909868    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz" event={"ID":"f15b9c95-390e-4258-a66e-0a03806392d9","Type":"ContainerStarted","Data":"bf2ba91f603ed61edac39a7b8fd589f07276ef239a4ae2a5340a0d663e4edccf"}
Oct 30 08:04:37.910372 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:37.909933    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz" event={"ID":"f15b9c95-390e-4258-a66e-0a03806392d9","Type":"ContainerStarted","Data":"837b2ec2c6615aa7bbda9aec1f57b1112844aa1945adf45896469be6eff55a53"}
Oct 30 08:04:37.961158 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:37.961096    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz" podStartSLOduration=0.961081749 podStartE2EDuration="961.081749ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:04:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:04:37.95880249 +0000 UTC m=+4036.568209462" watchObservedRunningTime="2026-10-30 08:04:37.961081749 +0000 UTC m=+4036.570488719"
Oct 30 08:04:38.291238 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:38.291196    6224 patch_prober.go:29] interesting pod/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body=
Oct 30 08:04:38.291453 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:38.291261    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq" podUID="61643ced-f0bd-472a-b42b-abb02994320c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused"
Oct 30 08:04:38.419474 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:38.419435    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz"
Oct 30 08:04:38.421184 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:38.421155    6224 patch_prober.go:29] interesting pod/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body=
Oct 30 08:04:38.421353 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:38.421211    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz" podUID="f15b9c95-390e-4258-a66e-0a03806392d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused"
Oct 30 08:04:39.286268 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:39.286225    6224 patch_prober.go:29] interesting pod/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body=
Oct 30 08:04:39.286887 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:39.286853    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq" podUID="61643ced-f0bd-472a-b42b-abb02994320c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused"
Oct 30 08:04:39.420560 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:39.420522    6224 patch_prober.go:29] interesting pod/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body=
Oct 30 08:04:39.420914 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:39.420883    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz" podUID="f15b9c95-390e-4258-a66e-0a03806392d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused"
Oct 30 08:04:40.289235 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:40.289193    6224 patch_prober.go:29] interesting pod/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body=
Oct 30 08:04:40.289746 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:40.289259    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq" podUID="61643ced-f0bd-472a-b42b-abb02994320c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused"
Oct 30 08:04:40.420830 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:40.420167    6224 patch_prober.go:29] interesting pod/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body=
Oct 30 08:04:40.420830 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:40.420228    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz" podUID="f15b9c95-390e-4258-a66e-0a03806392d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused"
Oct 30 08:04:41.290158 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:41.290122    6224 patch_prober.go:29] interesting pod/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body=
Oct 30 08:04:41.290748 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:41.290716    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq" podUID="61643ced-f0bd-472a-b42b-abb02994320c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused"
Oct 30 08:04:41.419583 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:41.419544    6224 patch_prober.go:29] interesting pod/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body=
Oct 30 08:04:41.419736 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:41.419592    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz" podUID="f15b9c95-390e-4258-a66e-0a03806392d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused"
Oct 30 08:04:42.287280 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:42.287238    6224 patch_prober.go:29] interesting pod/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body=
Oct 30 08:04:42.287574 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:42.287542    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq" podUID="61643ced-f0bd-472a-b42b-abb02994320c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused"
Oct 30 08:04:42.420274 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:42.420235    6224 patch_prober.go:29] interesting pod/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body=
Oct 30 08:04:42.420688 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:42.420310    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz" podUID="f15b9c95-390e-4258-a66e-0a03806392d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused"
Oct 30 08:04:43.349140 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:43.349097    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq"
Oct 30 08:04:43.349483 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:43.349458    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq"
Oct 30 08:04:43.350440 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:43.350418    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq"
Oct 30 08:04:43.420569 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:43.420523    6224 patch_prober.go:29] interesting pod/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body=
Oct 30 08:04:43.420976 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:43.420628    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz" podUID="f15b9c95-390e-4258-a66e-0a03806392d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused"
Oct 30 08:04:44.419615 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:44.419575    6224 patch_prober.go:29] interesting pod/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body=
Oct 30 08:04:44.419840 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:44.419636    6224 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz" podUID="f15b9c95-390e-4258-a66e-0a03806392d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused"
Oct 30 08:04:45.423976 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:45.423935    6224 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz"
Oct 30 08:04:45.424410 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:45.424314    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz"
Oct 30 08:04:45.425075 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:45.425051    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz"
Oct 30 08:04:53.439671 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:04:53.439636    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 08:05:01.860433 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:05:01.860390    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 08:05:03.932674 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:03.932591    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-b5998f/sub-test-model-b5998f-kserve-6bc79cd46c-qbgzz"]
Oct 30 08:05:14.666422 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:14.666385    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-8d0c2f-df895bbbb-gczsx"]
Oct 30 08:05:16.048998 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:16.048963    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 08:05:18.952358 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:18.952303    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-b5998f/sub-test-model-b5998f-kserve-6bc79cd46c-qbgzz" podUID="58a0459f-c497-4c48-a55a-f848e61dcbda" containerName="main" containerID="cri-o://2f02c807ccddd4437b3ada3a3e48a516e520ef58b9ed2f8529896b647efc0d57" gracePeriod=45
Oct 30 08:05:19.143284 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:19.143248    6224 generic.go:356] "Generic (PLEG): container finished" podID="58a0459f-c497-4c48-a55a-f848e61dcbda" containerID="2f02c807ccddd4437b3ada3a3e48a516e520ef58b9ed2f8529896b647efc0d57" exitCode=0
Oct 30 08:05:19.143463 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:19.143322    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-b5998f/sub-test-model-b5998f-kserve-6bc79cd46c-qbgzz" event={"ID":"58a0459f-c497-4c48-a55a-f848e61dcbda","Type":"ContainerDied","Data":"2f02c807ccddd4437b3ada3a3e48a516e520ef58b9ed2f8529896b647efc0d57"}
Oct 30 08:05:19.211460 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:19.211398    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-b5998f/sub-test-model-b5998f-kserve-6bc79cd46c-qbgzz"
Oct 30 08:05:19.245571 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:19.245543    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58a0459f-c497-4c48-a55a-f848e61dcbda-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58a0459f-c497-4c48-a55a-f848e61dcbda-tmp-dir\") pod \"58a0459f-c497-4c48-a55a-f848e61dcbda\" (UID: \"58a0459f-c497-4c48-a55a-f848e61dcbda\") "
Oct 30 08:05:19.245710 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:19.245609    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58a0459f-c497-4c48-a55a-f848e61dcbda-dshm\" (UniqueName: \"kubernetes.io/empty-dir/58a0459f-c497-4c48-a55a-f848e61dcbda-dshm\") pod \"58a0459f-c497-4c48-a55a-f848e61dcbda\" (UID: \"58a0459f-c497-4c48-a55a-f848e61dcbda\") "
Oct 30 08:05:19.245710 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:19.245628    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58a0459f-c497-4c48-a55a-f848e61dcbda-home\" (UniqueName: \"kubernetes.io/empty-dir/58a0459f-c497-4c48-a55a-f848e61dcbda-home\") pod \"58a0459f-c497-4c48-a55a-f848e61dcbda\" (UID: \"58a0459f-c497-4c48-a55a-f848e61dcbda\") "
Oct 30 08:05:19.245710 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:19.245672    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58a0459f-c497-4c48-a55a-f848e61dcbda-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58a0459f-c497-4c48-a55a-f848e61dcbda-model-cache\") pod \"58a0459f-c497-4c48-a55a-f848e61dcbda\" (UID: \"58a0459f-c497-4c48-a55a-f848e61dcbda\") "
Oct 30 08:05:19.245710 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:19.245700    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58a0459f-c497-4c48-a55a-f848e61dcbda-tls-certs\" (UniqueName: \"kubernetes.io/secret/58a0459f-c497-4c48-a55a-f848e61dcbda-tls-certs\") pod \"58a0459f-c497-4c48-a55a-f848e61dcbda\" (UID: \"58a0459f-c497-4c48-a55a-f848e61dcbda\") "
Oct 30 08:05:19.245988 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:19.245773    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58a0459f-c497-4c48-a55a-f848e61dcbda-kube-api-access-xxsft\" (UniqueName: \"kubernetes.io/projected/58a0459f-c497-4c48-a55a-f848e61dcbda-kube-api-access-xxsft\") pod \"58a0459f-c497-4c48-a55a-f848e61dcbda\" (UID: \"58a0459f-c497-4c48-a55a-f848e61dcbda\") "
Oct 30 08:05:19.245988 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:19.245864    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58a0459f-c497-4c48-a55a-f848e61dcbda-tmp-dir" pod "58a0459f-c497-4c48-a55a-f848e61dcbda" (UID: "58a0459f-c497-4c48-a55a-f848e61dcbda"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:05:19.245988 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:19.245951    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58a0459f-c497-4c48-a55a-f848e61dcbda-model-cache" pod "58a0459f-c497-4c48-a55a-f848e61dcbda" (UID: "58a0459f-c497-4c48-a55a-f848e61dcbda"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:05:19.245988 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:19.245966    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58a0459f-c497-4c48-a55a-f848e61dcbda-home" pod "58a0459f-c497-4c48-a55a-f848e61dcbda" (UID: "58a0459f-c497-4c48-a55a-f848e61dcbda"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:05:19.246220 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:19.246199    6224 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58a0459f-c497-4c48-a55a-f848e61dcbda-tmp-dir\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:05:19.246273 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:19.246225    6224 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58a0459f-c497-4c48-a55a-f848e61dcbda-home\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:05:19.246273 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:19.246238    6224 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58a0459f-c497-4c48-a55a-f848e61dcbda-model-cache\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:05:19.247676 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:19.247657    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58a0459f-c497-4c48-a55a-f848e61dcbda-dshm" pod "58a0459f-c497-4c48-a55a-f848e61dcbda" (UID: "58a0459f-c497-4c48-a55a-f848e61dcbda"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:05:19.247771 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:19.247751    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58a0459f-c497-4c48-a55a-f848e61dcbda-tls-certs" pod "58a0459f-c497-4c48-a55a-f848e61dcbda" (UID: "58a0459f-c497-4c48-a55a-f848e61dcbda"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:05:19.247856 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:19.247834    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58a0459f-c497-4c48-a55a-f848e61dcbda-kube-api-access-xxsft" pod "58a0459f-c497-4c48-a55a-f848e61dcbda" (UID: "58a0459f-c497-4c48-a55a-f848e61dcbda"). InnerVolumeSpecName "kube-api-access-xxsft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:05:19.346988 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:19.346956    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxsft\" (UniqueName: \"kubernetes.io/projected/58a0459f-c497-4c48-a55a-f848e61dcbda-kube-api-access-xxsft\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:05:19.346988 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:19.346984    6224 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58a0459f-c497-4c48-a55a-f848e61dcbda-dshm\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:05:19.346988 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:19.346993    6224 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58a0459f-c497-4c48-a55a-f848e61dcbda-tls-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:05:19.667251 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:19.667203    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-8d0c2f-df895bbbb-gczsx" podUID="845c4712-065d-4f17-87c6-514dc1b5c1a8" containerName="payload-processing" containerID="cri-o://b4d33d94102ed4318b4c7faef30925178a454b65e17ec002086a10d2e5d4ad20" gracePeriod=25
Oct 30 08:05:20.149845 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:20.149800    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-b5998f/sub-test-model-b5998f-kserve-6bc79cd46c-qbgzz"
Oct 30 08:05:20.150283 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:20.149844    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-b5998f/sub-test-model-b5998f-kserve-6bc79cd46c-qbgzz" event={"ID":"58a0459f-c497-4c48-a55a-f848e61dcbda","Type":"ContainerDied","Data":"9e3eeddbc1f27238209a24eb117c9abcf2e76bc7cfdc7412c388c4006390b04b"}
Oct 30 08:05:20.150283 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:20.149902    6224 scope.go:122] "RemoveContainer" containerID="2f02c807ccddd4437b3ada3a3e48a516e520ef58b9ed2f8529896b647efc0d57"
Oct 30 08:05:20.209541 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:20.209511    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-b5998f/sub-test-model-b5998f-kserve-6bc79cd46c-qbgzz"]
Oct 30 08:05:20.223267 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:20.223237    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-b5998f/sub-test-model-b5998f-kserve-6bc79cd46c-qbgzz"]
Oct 30 08:05:21.417228 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:21.417190    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d0c2f-54d45b7497-c4bvr"]
Oct 30 08:05:21.418088 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:21.418052    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d0c2f-54d45b7497-c4bvr" podUID="543279e1-d9a2-413b-a6ad-f76814756067" containerName="maas-api" containerID="cri-o://9659922acad6ef26ae72b22432c038f00364a386d9381e4a4b19d54ccc41abd1" gracePeriod=30
Oct 30 08:05:21.504160 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:21.503563    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58a0459f-c497-4c48-a55a-f848e61dcbda" path="/var/lib/kubelet/pods/58a0459f-c497-4c48-a55a-f848e61dcbda/volumes"
Oct 30 08:05:21.715870 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:21.715839    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d0c2f-54d45b7497-c4bvr"
Oct 30 08:05:21.771161 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:21.771126    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/543279e1-d9a2-413b-a6ad-f76814756067-metrics-certs\" (UniqueName: \"kubernetes.io/secret/543279e1-d9a2-413b-a6ad-f76814756067-metrics-certs\") pod \"543279e1-d9a2-413b-a6ad-f76814756067\" (UID: \"543279e1-d9a2-413b-a6ad-f76814756067\") "
Oct 30 08:05:21.771161 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:21.771165    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/543279e1-d9a2-413b-a6ad-f76814756067-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/543279e1-d9a2-413b-a6ad-f76814756067-maas-api-tls\") pod \"543279e1-d9a2-413b-a6ad-f76814756067\" (UID: \"543279e1-d9a2-413b-a6ad-f76814756067\") "
Oct 30 08:05:21.771425 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:21.771191    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/543279e1-d9a2-413b-a6ad-f76814756067-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/543279e1-d9a2-413b-a6ad-f76814756067-openshift-service-ca\") pod \"543279e1-d9a2-413b-a6ad-f76814756067\" (UID: \"543279e1-d9a2-413b-a6ad-f76814756067\") "
Oct 30 08:05:21.771425 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:21.771254    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/543279e1-d9a2-413b-a6ad-f76814756067-kube-api-access-mkshr\" (UniqueName: \"kubernetes.io/projected/543279e1-d9a2-413b-a6ad-f76814756067-kube-api-access-mkshr\") pod \"543279e1-d9a2-413b-a6ad-f76814756067\" (UID: \"543279e1-d9a2-413b-a6ad-f76814756067\") "
Oct 30 08:05:21.771995 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:21.771948    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/543279e1-d9a2-413b-a6ad-f76814756067-openshift-service-ca" pod "543279e1-d9a2-413b-a6ad-f76814756067" (UID: "543279e1-d9a2-413b-a6ad-f76814756067"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:05:21.772371 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:21.772345    6224 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/543279e1-d9a2-413b-a6ad-f76814756067-openshift-service-ca\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:05:21.773988 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:21.773959    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/543279e1-d9a2-413b-a6ad-f76814756067-metrics-certs" pod "543279e1-d9a2-413b-a6ad-f76814756067" (UID: "543279e1-d9a2-413b-a6ad-f76814756067"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:05:21.774153 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:21.774132    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/543279e1-d9a2-413b-a6ad-f76814756067-maas-api-tls" pod "543279e1-d9a2-413b-a6ad-f76814756067" (UID: "543279e1-d9a2-413b-a6ad-f76814756067"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:05:21.774284 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:21.774264    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/543279e1-d9a2-413b-a6ad-f76814756067-kube-api-access-mkshr" pod "543279e1-d9a2-413b-a6ad-f76814756067" (UID: "543279e1-d9a2-413b-a6ad-f76814756067"). InnerVolumeSpecName "kube-api-access-mkshr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:05:21.873579 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:21.873538    6224 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/543279e1-d9a2-413b-a6ad-f76814756067-metrics-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:05:21.873579 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:21.873579    6224 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/543279e1-d9a2-413b-a6ad-f76814756067-maas-api-tls\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:05:21.873790 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:21.873593    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkshr\" (UniqueName: \"kubernetes.io/projected/543279e1-d9a2-413b-a6ad-f76814756067-kube-api-access-mkshr\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:05:22.165240 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:22.165198    6224 generic.go:356] "Generic (PLEG): container finished" podID="543279e1-d9a2-413b-a6ad-f76814756067" containerID="9659922acad6ef26ae72b22432c038f00364a386d9381e4a4b19d54ccc41abd1" exitCode=0
Oct 30 08:05:22.165240 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:22.165235    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d0c2f-54d45b7497-c4bvr" event={"ID":"543279e1-d9a2-413b-a6ad-f76814756067","Type":"ContainerDied","Data":"9659922acad6ef26ae72b22432c038f00364a386d9381e4a4b19d54ccc41abd1"}
Oct 30 08:05:22.165506 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:22.165266    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d0c2f-54d45b7497-c4bvr"
Oct 30 08:05:22.165506 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:22.165297    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d0c2f-54d45b7497-c4bvr" event={"ID":"543279e1-d9a2-413b-a6ad-f76814756067","Type":"ContainerDied","Data":"e903ed86356ad415ff98a14662b8a7033df2d69071fe68d9c6d1b5e865fa325f"}
Oct 30 08:05:22.165506 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:22.165320    6224 scope.go:122] "RemoveContainer" containerID="9659922acad6ef26ae72b22432c038f00364a386d9381e4a4b19d54ccc41abd1"
Oct 30 08:05:22.175183 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:22.175167    6224 scope.go:122] "RemoveContainer" containerID="9659922acad6ef26ae72b22432c038f00364a386d9381e4a4b19d54ccc41abd1"
Oct 30 08:05:22.175448 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:05:22.175430    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9659922acad6ef26ae72b22432c038f00364a386d9381e4a4b19d54ccc41abd1\": container with ID starting with 9659922acad6ef26ae72b22432c038f00364a386d9381e4a4b19d54ccc41abd1 not found: ID does not exist" containerID="9659922acad6ef26ae72b22432c038f00364a386d9381e4a4b19d54ccc41abd1"
Oct 30 08:05:22.175496 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:22.175455    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9659922acad6ef26ae72b22432c038f00364a386d9381e4a4b19d54ccc41abd1"} err="failed to get container status \"9659922acad6ef26ae72b22432c038f00364a386d9381e4a4b19d54ccc41abd1\": rpc error: code = NotFound desc = could not find container \"9659922acad6ef26ae72b22432c038f00364a386d9381e4a4b19d54ccc41abd1\": container with ID starting with 9659922acad6ef26ae72b22432c038f00364a386d9381e4a4b19d54ccc41abd1 not found: ID does not exist"
Oct 30 08:05:22.224022 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:22.223993    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d0c2f-54d45b7497-c4bvr"]
Oct 30 08:05:22.230346 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:22.230321    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d0c2f-54d45b7497-c4bvr"]
Oct 30 08:05:23.503358 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:23.503322    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="543279e1-d9a2-413b-a6ad-f76814756067" path="/var/lib/kubelet/pods/543279e1-d9a2-413b-a6ad-f76814756067/volumes"
Oct 30 08:05:28.527765 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:28.527726    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-8d0c2f-df895bbbb-gczsx" podUID="845c4712-065d-4f17-87c6-514dc1b5c1a8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:05:29.754674 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:29.754635    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f5ceca/rate-test-model-f5ceca-kserve-845bf4bfdf-ck76g"]
Oct 30 08:05:30.055523 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:30.055486    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0f5081-hgmc7"]
Oct 30 08:05:30.056014 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:30.055993    6224 cpu_manager.go:414] "containerMap: removing container" podUID="58a0459f-c497-4c48-a55a-f848e61dcbda" containerName="main"
Oct 30 08:05:30.056014 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:30.056013    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58a0459f-c497-4c48-a55a-f848e61dcbda" containerName="main"
Oct 30 08:05:30.056143 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:30.056030    6224 cpu_manager.go:414] "containerMap: removing container" podUID="543279e1-d9a2-413b-a6ad-f76814756067" containerName="maas-api"
Oct 30 08:05:30.056143 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:30.056036    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="543279e1-d9a2-413b-a6ad-f76814756067" containerName="maas-api"
Oct 30 08:05:30.056143 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:30.056130    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="58a0459f-c497-4c48-a55a-f848e61dcbda" containerName="main"
Oct 30 08:05:30.056143 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:30.056139    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="543279e1-d9a2-413b-a6ad-f76814756067" containerName="maas-api"
Oct 30 08:05:30.059151 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:30.059135    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0f5081-hgmc7"
Oct 30 08:05:30.076687 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:30.076301    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 08:05:30.076687 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:30.076424    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 08:05:30.078450 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:30.078422    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0f5081-hgmc7"]
Oct 30 08:05:30.163624 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:30.163587    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d9a2b3b-ed1e-46a9-9e4e-2bc5a21d7aa8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-0f5081-hgmc7\" (UID: \"9d9a2b3b-ed1e-46a9-9e4e-2bc5a21d7aa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0f5081-hgmc7"
Oct 30 08:05:30.163852 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:30.163633    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx8j2\" (UniqueName: \"kubernetes.io/projected/9d9a2b3b-ed1e-46a9-9e4e-2bc5a21d7aa8-kube-api-access-mx8j2\") pod \"maas-api-revoke-keys-e2e-extmod-a-0f5081-hgmc7\" (UID: \"9d9a2b3b-ed1e-46a9-9e4e-2bc5a21d7aa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0f5081-hgmc7"
Oct 30 08:05:30.163852 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:30.163717    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d9a2b3b-ed1e-46a9-9e4e-2bc5a21d7aa8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-0f5081-hgmc7\" (UID: \"9d9a2b3b-ed1e-46a9-9e4e-2bc5a21d7aa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0f5081-hgmc7"
Oct 30 08:05:30.265851 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:30.265777    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d9a2b3b-ed1e-46a9-9e4e-2bc5a21d7aa8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-0f5081-hgmc7\" (UID: \"9d9a2b3b-ed1e-46a9-9e4e-2bc5a21d7aa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0f5081-hgmc7"
Oct 30 08:05:30.266010 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:30.265870    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx8j2\" (UniqueName: \"kubernetes.io/projected/9d9a2b3b-ed1e-46a9-9e4e-2bc5a21d7aa8-kube-api-access-mx8j2\") pod \"maas-api-revoke-keys-e2e-extmod-a-0f5081-hgmc7\" (UID: \"9d9a2b3b-ed1e-46a9-9e4e-2bc5a21d7aa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0f5081-hgmc7"
Oct 30 08:05:30.266010 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:30.265914    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d9a2b3b-ed1e-46a9-9e4e-2bc5a21d7aa8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-0f5081-hgmc7\" (UID: \"9d9a2b3b-ed1e-46a9-9e4e-2bc5a21d7aa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0f5081-hgmc7"
Oct 30 08:05:30.266687 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:30.266665    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d9a2b3b-ed1e-46a9-9e4e-2bc5a21d7aa8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-0f5081-hgmc7\" (UID: \"9d9a2b3b-ed1e-46a9-9e4e-2bc5a21d7aa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0f5081-hgmc7"
Oct 30 08:05:30.268245 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:30.268212    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d9a2b3b-ed1e-46a9-9e4e-2bc5a21d7aa8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-0f5081-hgmc7\" (UID: \"9d9a2b3b-ed1e-46a9-9e4e-2bc5a21d7aa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0f5081-hgmc7"
Oct 30 08:05:30.303067 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:30.303035    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx8j2\" (UniqueName: \"kubernetes.io/projected/9d9a2b3b-ed1e-46a9-9e4e-2bc5a21d7aa8-kube-api-access-mx8j2\") pod \"maas-api-revoke-keys-e2e-extmod-a-0f5081-hgmc7\" (UID: \"9d9a2b3b-ed1e-46a9-9e4e-2bc5a21d7aa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0f5081-hgmc7"
Oct 30 08:05:30.373618 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:30.373535    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0f5081-hgmc7"
Oct 30 08:05:30.672418 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:30.672392    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0f5081-hgmc7"]
Oct 30 08:05:30.674866 ip-10-0-128-125 kubenswrapper[6224]: W1030 08:05:30.674836    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d9a2b3b_ed1e_46a9_9e4e_2bc5a21d7aa8.slice/crio-a02ee73bf051934d393799fb57b992745d5549ff0d7b361db63ba8df20172032 WatchSource:0}: Error finding container a02ee73bf051934d393799fb57b992745d5549ff0d7b361db63ba8df20172032: Status 404 returned error can't find the container with id a02ee73bf051934d393799fb57b992745d5549ff0d7b361db63ba8df20172032
Oct 30 08:05:31.218845 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:31.218792    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0f5081-hgmc7" event={"ID":"9d9a2b3b-ed1e-46a9-9e4e-2bc5a21d7aa8","Type":"ContainerStarted","Data":"a052c4585f619ae6769e5a105ae17631f70b63f25d71a7ff32237e9a2484c964"}
Oct 30 08:05:31.219224 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:31.218874    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0f5081-hgmc7" event={"ID":"9d9a2b3b-ed1e-46a9-9e4e-2bc5a21d7aa8","Type":"ContainerStarted","Data":"a02ee73bf051934d393799fb57b992745d5549ff0d7b361db63ba8df20172032"}
Oct 30 08:05:31.265528 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:31.265433    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0f5081-hgmc7" podStartSLOduration=1.2654189 podStartE2EDuration="1.2654189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:05:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:05:31.263793023 +0000 UTC m=+4089.873199994" watchObservedRunningTime="2026-10-30 08:05:31.2654189 +0000 UTC m=+4089.874825869"
Oct 30 08:05:31.861440 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:05:31.861344    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 08:05:32.226210 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:32.226117    6224 generic.go:356] "Generic (PLEG): container finished" podID="9d9a2b3b-ed1e-46a9-9e4e-2bc5a21d7aa8" containerID="a052c4585f619ae6769e5a105ae17631f70b63f25d71a7ff32237e9a2484c964" exitCode=0
Oct 30 08:05:32.226210 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:32.226173    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0f5081-hgmc7" event={"ID":"9d9a2b3b-ed1e-46a9-9e4e-2bc5a21d7aa8","Type":"ContainerDied","Data":"a052c4585f619ae6769e5a105ae17631f70b63f25d71a7ff32237e9a2484c964"}
Oct 30 08:05:33.370483 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:33.370448    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0f5081-hgmc7"
Oct 30 08:05:33.498100 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:33.498005    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d9a2b3b-ed1e-46a9-9e4e-2bc5a21d7aa8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d9a2b3b-ed1e-46a9-9e4e-2bc5a21d7aa8-maas-api-xks-ca\") pod \"9d9a2b3b-ed1e-46a9-9e4e-2bc5a21d7aa8\" (UID: \"9d9a2b3b-ed1e-46a9-9e4e-2bc5a21d7aa8\") "
Oct 30 08:05:33.498100 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:33.498053    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d9a2b3b-ed1e-46a9-9e4e-2bc5a21d7aa8-kube-api-access-mx8j2\" (UniqueName: \"kubernetes.io/projected/9d9a2b3b-ed1e-46a9-9e4e-2bc5a21d7aa8-kube-api-access-mx8j2\") pod \"9d9a2b3b-ed1e-46a9-9e4e-2bc5a21d7aa8\" (UID: \"9d9a2b3b-ed1e-46a9-9e4e-2bc5a21d7aa8\") "
Oct 30 08:05:33.498352 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:33.498214    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d9a2b3b-ed1e-46a9-9e4e-2bc5a21d7aa8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d9a2b3b-ed1e-46a9-9e4e-2bc5a21d7aa8-maas-api-openshift-ca\") pod \"9d9a2b3b-ed1e-46a9-9e4e-2bc5a21d7aa8\" (UID: \"9d9a2b3b-ed1e-46a9-9e4e-2bc5a21d7aa8\") "
Oct 30 08:05:33.498583 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:33.498556    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d9a2b3b-ed1e-46a9-9e4e-2bc5a21d7aa8-maas-api-openshift-ca" pod "9d9a2b3b-ed1e-46a9-9e4e-2bc5a21d7aa8" (UID: "9d9a2b3b-ed1e-46a9-9e4e-2bc5a21d7aa8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:05:33.500190 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:33.500163    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d9a2b3b-ed1e-46a9-9e4e-2bc5a21d7aa8-maas-api-xks-ca" pod "9d9a2b3b-ed1e-46a9-9e4e-2bc5a21d7aa8" (UID: "9d9a2b3b-ed1e-46a9-9e4e-2bc5a21d7aa8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:05:33.500336 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:33.500317    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d9a2b3b-ed1e-46a9-9e4e-2bc5a21d7aa8-kube-api-access-mx8j2" pod "9d9a2b3b-ed1e-46a9-9e4e-2bc5a21d7aa8" (UID: "9d9a2b3b-ed1e-46a9-9e4e-2bc5a21d7aa8"). InnerVolumeSpecName "kube-api-access-mx8j2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:05:33.599384 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:33.599348    6224 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d9a2b3b-ed1e-46a9-9e4e-2bc5a21d7aa8-maas-api-xks-ca\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:05:33.599384 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:33.599379    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx8j2\" (UniqueName: \"kubernetes.io/projected/9d9a2b3b-ed1e-46a9-9e4e-2bc5a21d7aa8-kube-api-access-mx8j2\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:05:33.599384 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:33.599389    6224 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d9a2b3b-ed1e-46a9-9e4e-2bc5a21d7aa8-maas-api-openshift-ca\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:05:34.242927 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:34.242884    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0f5081-hgmc7" event={"ID":"9d9a2b3b-ed1e-46a9-9e4e-2bc5a21d7aa8","Type":"ContainerDied","Data":"a02ee73bf051934d393799fb57b992745d5549ff0d7b361db63ba8df20172032"}
Oct 30 08:05:34.242927 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:34.242924    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0f5081-hgmc7"
Oct 30 08:05:34.243152 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:34.242949    6224 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a02ee73bf051934d393799fb57b992745d5549ff0d7b361db63ba8df20172032"
Oct 30 08:05:36.687580 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:36.687547    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 08:05:38.527356 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:38.527310    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-8d0c2f-df895bbbb-gczsx" podUID="845c4712-065d-4f17-87c6-514dc1b5c1a8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:05:39.925306 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:39.925253    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-8d0c2f-df895bbbb-gczsx"
Oct 30 08:05:40.068254 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:40.068222    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/845c4712-065d-4f17-87c6-514dc1b5c1a8-kube-api-access-wn8ml\" (UniqueName: \"kubernetes.io/projected/845c4712-065d-4f17-87c6-514dc1b5c1a8-kube-api-access-wn8ml\") pod \"845c4712-065d-4f17-87c6-514dc1b5c1a8\" (UID: \"845c4712-065d-4f17-87c6-514dc1b5c1a8\") "
Oct 30 08:05:40.068435 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:40.068329    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/845c4712-065d-4f17-87c6-514dc1b5c1a8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/845c4712-065d-4f17-87c6-514dc1b5c1a8-plugins-config-volume\") pod \"845c4712-065d-4f17-87c6-514dc1b5c1a8\" (UID: \"845c4712-065d-4f17-87c6-514dc1b5c1a8\") "
Oct 30 08:05:40.068734 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:40.068710    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/845c4712-065d-4f17-87c6-514dc1b5c1a8-plugins-config-volume" pod "845c4712-065d-4f17-87c6-514dc1b5c1a8" (UID: "845c4712-065d-4f17-87c6-514dc1b5c1a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:05:40.070413 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:40.070390    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/845c4712-065d-4f17-87c6-514dc1b5c1a8-kube-api-access-wn8ml" pod "845c4712-065d-4f17-87c6-514dc1b5c1a8" (UID: "845c4712-065d-4f17-87c6-514dc1b5c1a8"). InnerVolumeSpecName "kube-api-access-wn8ml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:05:40.169650 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:40.169607    6224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/845c4712-065d-4f17-87c6-514dc1b5c1a8-plugins-config-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:05:40.169650 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:40.169654    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wn8ml\" (UniqueName: \"kubernetes.io/projected/845c4712-065d-4f17-87c6-514dc1b5c1a8-kube-api-access-wn8ml\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:05:40.276893 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:40.276785    6224 generic.go:356] "Generic (PLEG): container finished" podID="845c4712-065d-4f17-87c6-514dc1b5c1a8" containerID="b4d33d94102ed4318b4c7faef30925178a454b65e17ec002086a10d2e5d4ad20" exitCode=0
Oct 30 08:05:40.276893 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:40.276846    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-8d0c2f-df895bbbb-gczsx"
Oct 30 08:05:40.276893 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:40.276857    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-8d0c2f-df895bbbb-gczsx" event={"ID":"845c4712-065d-4f17-87c6-514dc1b5c1a8","Type":"ContainerDied","Data":"b4d33d94102ed4318b4c7faef30925178a454b65e17ec002086a10d2e5d4ad20"}
Oct 30 08:05:40.277222 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:40.276903    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-8d0c2f-df895bbbb-gczsx" event={"ID":"845c4712-065d-4f17-87c6-514dc1b5c1a8","Type":"ContainerDied","Data":"b603b070c4de2d9da68f76afadc1375f5fa0f8a6162abfbe291ac0e4f048d6e4"}
Oct 30 08:05:40.277222 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:40.276927    6224 scope.go:122] "RemoveContainer" containerID="b4d33d94102ed4318b4c7faef30925178a454b65e17ec002086a10d2e5d4ad20"
Oct 30 08:05:40.287778 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:40.287760    6224 scope.go:122] "RemoveContainer" containerID="b4d33d94102ed4318b4c7faef30925178a454b65e17ec002086a10d2e5d4ad20"
Oct 30 08:05:40.288160 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:05:40.288139    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4d33d94102ed4318b4c7faef30925178a454b65e17ec002086a10d2e5d4ad20\": container with ID starting with b4d33d94102ed4318b4c7faef30925178a454b65e17ec002086a10d2e5d4ad20 not found: ID does not exist" containerID="b4d33d94102ed4318b4c7faef30925178a454b65e17ec002086a10d2e5d4ad20"
Oct 30 08:05:40.288220 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:40.288168    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4d33d94102ed4318b4c7faef30925178a454b65e17ec002086a10d2e5d4ad20"} err="failed to get container status \"b4d33d94102ed4318b4c7faef30925178a454b65e17ec002086a10d2e5d4ad20\": rpc error: code = NotFound desc = could not find container \"b4d33d94102ed4318b4c7faef30925178a454b65e17ec002086a10d2e5d4ad20\": container with ID starting with b4d33d94102ed4318b4c7faef30925178a454b65e17ec002086a10d2e5d4ad20 not found: ID does not exist"
Oct 30 08:05:40.336831 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:40.336777    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-8d0c2f-df895bbbb-gczsx"]
Oct 30 08:05:40.339503 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:40.339477    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-8d0c2f-df895bbbb-gczsx"]
Oct 30 08:05:40.548230 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:40.548194    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e27e0c-6996bc9b75-kmm89"]
Oct 30 08:05:41.502455 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:41.502414    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="845c4712-065d-4f17-87c6-514dc1b5c1a8" path="/var/lib/kubelet/pods/845c4712-065d-4f17-87c6-514dc1b5c1a8/volumes"
Oct 30 08:05:44.775252 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:44.775205    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-f5ceca/rate-test-model-f5ceca-kserve-845bf4bfdf-ck76g" podUID="f9d4bb88-868c-4767-be0a-08f21d7a1f8c" containerName="main" containerID="cri-o://60801af8224f24604a725d97b39820ec79c9a67252a5f5d62c8fb11c80043998" gracePeriod=45
Oct 30 08:05:45.035035 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:45.035010    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f5ceca/rate-test-model-f5ceca-kserve-845bf4bfdf-ck76g"
Oct 30 08:05:45.115349 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:45.115316    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9d4bb88-868c-4767-be0a-08f21d7a1f8c-kube-api-access-jflkt\" (UniqueName: \"kubernetes.io/projected/f9d4bb88-868c-4767-be0a-08f21d7a1f8c-kube-api-access-jflkt\") pod \"f9d4bb88-868c-4767-be0a-08f21d7a1f8c\" (UID: \"f9d4bb88-868c-4767-be0a-08f21d7a1f8c\") "
Oct 30 08:05:45.115538 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:45.115360    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9d4bb88-868c-4767-be0a-08f21d7a1f8c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9d4bb88-868c-4767-be0a-08f21d7a1f8c-tmp-dir\") pod \"f9d4bb88-868c-4767-be0a-08f21d7a1f8c\" (UID: \"f9d4bb88-868c-4767-be0a-08f21d7a1f8c\") "
Oct 30 08:05:45.115538 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:45.115400    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9d4bb88-868c-4767-be0a-08f21d7a1f8c-tls-certs\" (UniqueName: \"kubernetes.io/secret/f9d4bb88-868c-4767-be0a-08f21d7a1f8c-tls-certs\") pod \"f9d4bb88-868c-4767-be0a-08f21d7a1f8c\" (UID: \"f9d4bb88-868c-4767-be0a-08f21d7a1f8c\") "
Oct 30 08:05:45.115538 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:45.115449    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9d4bb88-868c-4767-be0a-08f21d7a1f8c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9d4bb88-868c-4767-be0a-08f21d7a1f8c-dshm\") pod \"f9d4bb88-868c-4767-be0a-08f21d7a1f8c\" (UID: \"f9d4bb88-868c-4767-be0a-08f21d7a1f8c\") "
Oct 30 08:05:45.115538 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:45.115469    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9d4bb88-868c-4767-be0a-08f21d7a1f8c-home\" (UniqueName: \"kubernetes.io/empty-dir/f9d4bb88-868c-4767-be0a-08f21d7a1f8c-home\") pod \"f9d4bb88-868c-4767-be0a-08f21d7a1f8c\" (UID: \"f9d4bb88-868c-4767-be0a-08f21d7a1f8c\") "
Oct 30 08:05:45.115538 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:45.115498    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9d4bb88-868c-4767-be0a-08f21d7a1f8c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9d4bb88-868c-4767-be0a-08f21d7a1f8c-model-cache\") pod \"f9d4bb88-868c-4767-be0a-08f21d7a1f8c\" (UID: \"f9d4bb88-868c-4767-be0a-08f21d7a1f8c\") "
Oct 30 08:05:45.115887 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:45.115688    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9d4bb88-868c-4767-be0a-08f21d7a1f8c-tmp-dir" pod "f9d4bb88-868c-4767-be0a-08f21d7a1f8c" (UID: "f9d4bb88-868c-4767-be0a-08f21d7a1f8c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:05:45.115887 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:45.115874    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9d4bb88-868c-4767-be0a-08f21d7a1f8c-home" pod "f9d4bb88-868c-4767-be0a-08f21d7a1f8c" (UID: "f9d4bb88-868c-4767-be0a-08f21d7a1f8c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:05:45.115983 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:45.115882    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9d4bb88-868c-4767-be0a-08f21d7a1f8c-model-cache" pod "f9d4bb88-868c-4767-be0a-08f21d7a1f8c" (UID: "f9d4bb88-868c-4767-be0a-08f21d7a1f8c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:05:45.115983 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:45.115974    6224 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9d4bb88-868c-4767-be0a-08f21d7a1f8c-tmp-dir\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:05:45.116074 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:45.115994    6224 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9d4bb88-868c-4767-be0a-08f21d7a1f8c-home\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:05:45.116074 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:45.116008    6224 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9d4bb88-868c-4767-be0a-08f21d7a1f8c-model-cache\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:05:45.117697 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:45.117666    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9d4bb88-868c-4767-be0a-08f21d7a1f8c-dshm" pod "f9d4bb88-868c-4767-be0a-08f21d7a1f8c" (UID: "f9d4bb88-868c-4767-be0a-08f21d7a1f8c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:05:45.117836 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:45.117725    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9d4bb88-868c-4767-be0a-08f21d7a1f8c-kube-api-access-jflkt" pod "f9d4bb88-868c-4767-be0a-08f21d7a1f8c" (UID: "f9d4bb88-868c-4767-be0a-08f21d7a1f8c"). InnerVolumeSpecName "kube-api-access-jflkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:05:45.117836 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:45.117751    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9d4bb88-868c-4767-be0a-08f21d7a1f8c-tls-certs" pod "f9d4bb88-868c-4767-be0a-08f21d7a1f8c" (UID: "f9d4bb88-868c-4767-be0a-08f21d7a1f8c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:05:45.217279 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:45.217234    6224 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9d4bb88-868c-4767-be0a-08f21d7a1f8c-tls-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:05:45.217279 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:45.217275    6224 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9d4bb88-868c-4767-be0a-08f21d7a1f8c-dshm\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:05:45.217279 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:45.217286    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jflkt\" (UniqueName: \"kubernetes.io/projected/f9d4bb88-868c-4767-be0a-08f21d7a1f8c-kube-api-access-jflkt\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:05:45.315272 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:45.315176    6224 generic.go:356] "Generic (PLEG): container finished" podID="f9d4bb88-868c-4767-be0a-08f21d7a1f8c" containerID="60801af8224f24604a725d97b39820ec79c9a67252a5f5d62c8fb11c80043998" exitCode=0
Oct 30 08:05:45.315272 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:45.315245    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f5ceca/rate-test-model-f5ceca-kserve-845bf4bfdf-ck76g"
Oct 30 08:05:45.315272 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:45.315264    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f5ceca/rate-test-model-f5ceca-kserve-845bf4bfdf-ck76g" event={"ID":"f9d4bb88-868c-4767-be0a-08f21d7a1f8c","Type":"ContainerDied","Data":"60801af8224f24604a725d97b39820ec79c9a67252a5f5d62c8fb11c80043998"}
Oct 30 08:05:45.315521 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:45.315305    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f5ceca/rate-test-model-f5ceca-kserve-845bf4bfdf-ck76g" event={"ID":"f9d4bb88-868c-4767-be0a-08f21d7a1f8c","Type":"ContainerDied","Data":"95cbdc209e22a3cdc087dbffb99c7c10a41f0a9d1fae23b028efc31b5828865d"}
Oct 30 08:05:45.315521 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:45.315324    6224 scope.go:122] "RemoveContainer" containerID="60801af8224f24604a725d97b39820ec79c9a67252a5f5d62c8fb11c80043998"
Oct 30 08:05:45.335765 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:45.335732    6224 scope.go:122] "RemoveContainer" containerID="60801af8224f24604a725d97b39820ec79c9a67252a5f5d62c8fb11c80043998"
Oct 30 08:05:45.336143 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:05:45.336113    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60801af8224f24604a725d97b39820ec79c9a67252a5f5d62c8fb11c80043998\": container with ID starting with 60801af8224f24604a725d97b39820ec79c9a67252a5f5d62c8fb11c80043998 not found: ID does not exist" containerID="60801af8224f24604a725d97b39820ec79c9a67252a5f5d62c8fb11c80043998"
Oct 30 08:05:45.336250 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:45.336153    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60801af8224f24604a725d97b39820ec79c9a67252a5f5d62c8fb11c80043998"} err="failed to get container status \"60801af8224f24604a725d97b39820ec79c9a67252a5f5d62c8fb11c80043998\": rpc error: code = NotFound desc = could not find container \"60801af8224f24604a725d97b39820ec79c9a67252a5f5d62c8fb11c80043998\": container with ID starting with 60801af8224f24604a725d97b39820ec79c9a67252a5f5d62c8fb11c80043998 not found: ID does not exist"
Oct 30 08:05:45.373827 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:45.373769    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f5ceca/rate-test-model-f5ceca-kserve-845bf4bfdf-ck76g"]
Oct 30 08:05:45.379468 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:45.379441    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f5ceca/rate-test-model-f5ceca-kserve-845bf4bfdf-ck76g"]
Oct 30 08:05:45.502778 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:45.502721    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9d4bb88-868c-4767-be0a-08f21d7a1f8c" path="/var/lib/kubelet/pods/f9d4bb88-868c-4767-be0a-08f21d7a1f8c/volumes"
Oct 30 08:05:45.549075 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:45.549008    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e27e0c-6996bc9b75-kmm89" podUID="b78cab03-456c-4ea9-93bb-e5c84f7d72a6" containerName="payload-pre-processing" containerID="cri-o://e2903d2883e92218c1ec2b118815e042e99bbf154d027a7e13bea76583c1b221" gracePeriod=25
Oct 30 08:05:46.011995 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:46.011947    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-f5ceca/rate-test-model-f5ceca-kserve-845bf4bfdf-ck76g" podUID="f9d4bb88-868c-4767-be0a-08f21d7a1f8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.229:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 08:05:46.448894 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:46.448852    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e27e0c-6996bc9b75-kmm89" podUID="b78cab03-456c-4ea9-93bb-e5c84f7d72a6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:05:51.471358 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:51.471318    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 08:05:54.077890 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:54.077846    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-bd8589-wprkl"]
Oct 30 08:05:54.078322 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:54.078306    6224 cpu_manager.go:414] "containerMap: removing container" podUID="9d9a2b3b-ed1e-46a9-9e4e-2bc5a21d7aa8" containerName="revoke-keys"
Oct 30 08:05:54.078373 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:54.078324    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d9a2b3b-ed1e-46a9-9e4e-2bc5a21d7aa8" containerName="revoke-keys"
Oct 30 08:05:54.078373 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:54.078348    6224 cpu_manager.go:414] "containerMap: removing container" podUID="845c4712-065d-4f17-87c6-514dc1b5c1a8" containerName="payload-processing"
Oct 30 08:05:54.078373 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:54.078353    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="845c4712-065d-4f17-87c6-514dc1b5c1a8" containerName="payload-processing"
Oct 30 08:05:54.078373 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:54.078368    6224 cpu_manager.go:414] "containerMap: removing container" podUID="f9d4bb88-868c-4767-be0a-08f21d7a1f8c" containerName="main"
Oct 30 08:05:54.078503 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:54.078374    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9d4bb88-868c-4767-be0a-08f21d7a1f8c" containerName="main"
Oct 30 08:05:54.078503 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:54.078437    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d9a2b3b-ed1e-46a9-9e4e-2bc5a21d7aa8" containerName="revoke-keys"
Oct 30 08:05:54.078503 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:54.078448    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="845c4712-065d-4f17-87c6-514dc1b5c1a8" containerName="payload-processing"
Oct 30 08:05:54.078503 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:54.078454    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9d4bb88-868c-4767-be0a-08f21d7a1f8c" containerName="main"
Oct 30 08:05:54.081480 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:54.081461    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-bd8589-wprkl"
Oct 30 08:05:54.100877 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:54.100850    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-bd8589-wprkl"]
Oct 30 08:05:54.153685 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:54.153652    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 08:05:54.153896 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:54.153697    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 08:05:54.209362 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:54.209324    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e6a7f63-a247-424b-a491-4d451f994062-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-bd8589-wprkl\" (UID: \"9e6a7f63-a247-424b-a491-4d451f994062\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-bd8589-wprkl"
Oct 30 08:05:54.209550 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:54.209487    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvrrf\" (UniqueName: \"kubernetes.io/projected/9e6a7f63-a247-424b-a491-4d451f994062-kube-api-access-jvrrf\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-bd8589-wprkl\" (UID: \"9e6a7f63-a247-424b-a491-4d451f994062\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-bd8589-wprkl"
Oct 30 08:05:54.209550 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:54.209524    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e6a7f63-a247-424b-a491-4d451f994062-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-bd8589-wprkl\" (UID: \"9e6a7f63-a247-424b-a491-4d451f994062\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-bd8589-wprkl"
Oct 30 08:05:54.310466 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:54.310434    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvrrf\" (UniqueName: \"kubernetes.io/projected/9e6a7f63-a247-424b-a491-4d451f994062-kube-api-access-jvrrf\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-bd8589-wprkl\" (UID: \"9e6a7f63-a247-424b-a491-4d451f994062\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-bd8589-wprkl"
Oct 30 08:05:54.310466 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:54.310471    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e6a7f63-a247-424b-a491-4d451f994062-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-bd8589-wprkl\" (UID: \"9e6a7f63-a247-424b-a491-4d451f994062\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-bd8589-wprkl"
Oct 30 08:05:54.310715 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:54.310528    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e6a7f63-a247-424b-a491-4d451f994062-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-bd8589-wprkl\" (UID: \"9e6a7f63-a247-424b-a491-4d451f994062\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-bd8589-wprkl"
Oct 30 08:05:54.311339 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:54.311313    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e6a7f63-a247-424b-a491-4d451f994062-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-bd8589-wprkl\" (UID: \"9e6a7f63-a247-424b-a491-4d451f994062\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-bd8589-wprkl"
Oct 30 08:05:54.312874 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:54.312854    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e6a7f63-a247-424b-a491-4d451f994062-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-bd8589-wprkl\" (UID: \"9e6a7f63-a247-424b-a491-4d451f994062\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-bd8589-wprkl"
Oct 30 08:05:54.336563 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:54.336506    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvrrf\" (UniqueName: \"kubernetes.io/projected/9e6a7f63-a247-424b-a491-4d451f994062-kube-api-access-jvrrf\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-bd8589-wprkl\" (UID: \"9e6a7f63-a247-424b-a491-4d451f994062\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-bd8589-wprkl"
Oct 30 08:05:54.392967 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:54.392933    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-bd8589-wprkl"
Oct 30 08:05:54.693438 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:54.693407    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-bd8589-wprkl"]
Oct 30 08:05:54.694093 ip-10-0-128-125 kubenswrapper[6224]: W1030 08:05:54.694059    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e6a7f63_a247_424b_a491_4d451f994062.slice/crio-bb288e7a7648e493079924bbf5210b21582ec53695d90043f215303fb6c90d06 WatchSource:0}: Error finding container bb288e7a7648e493079924bbf5210b21582ec53695d90043f215303fb6c90d06: Status 404 returned error can't find the container with id bb288e7a7648e493079924bbf5210b21582ec53695d90043f215303fb6c90d06
Oct 30 08:05:55.373634 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:55.373600    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-bd8589-wprkl" event={"ID":"9e6a7f63-a247-424b-a491-4d451f994062","Type":"ContainerStarted","Data":"3fb3f96e1b125fac12ed6e335e09e41724901e36cf9c595b75ddf5d78e96cc9e"}
Oct 30 08:05:55.374071 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:55.373651    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-bd8589-wprkl" event={"ID":"9e6a7f63-a247-424b-a491-4d451f994062","Type":"ContainerStarted","Data":"bb288e7a7648e493079924bbf5210b21582ec53695d90043f215303fb6c90d06"}
Oct 30 08:05:55.418956 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:55.418900    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-bd8589-wprkl" podStartSLOduration=1.418886 podStartE2EDuration="1.418886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:05:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:05:55.41738949 +0000 UTC m=+4114.026796464" watchObservedRunningTime="2026-10-30 08:05:55.418886 +0000 UTC m=+4114.028292969"
Oct 30 08:05:56.381129 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:56.381085    6224 generic.go:356] "Generic (PLEG): container finished" podID="9e6a7f63-a247-424b-a491-4d451f994062" containerID="3fb3f96e1b125fac12ed6e335e09e41724901e36cf9c595b75ddf5d78e96cc9e" exitCode=0
Oct 30 08:05:56.381549 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:56.381171    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-bd8589-wprkl" event={"ID":"9e6a7f63-a247-424b-a491-4d451f994062","Type":"ContainerDied","Data":"3fb3f96e1b125fac12ed6e335e09e41724901e36cf9c595b75ddf5d78e96cc9e"}
Oct 30 08:05:56.448996 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:56.448950    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e27e0c-6996bc9b75-kmm89" podUID="b78cab03-456c-4ea9-93bb-e5c84f7d72a6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:05:57.530956 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:57.530926    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-bd8589-wprkl"
Oct 30 08:05:57.644556 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:57.644467    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e6a7f63-a247-424b-a491-4d451f994062-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e6a7f63-a247-424b-a491-4d451f994062-maas-api-openshift-ca\") pod \"9e6a7f63-a247-424b-a491-4d451f994062\" (UID: \"9e6a7f63-a247-424b-a491-4d451f994062\") "
Oct 30 08:05:57.644766 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:57.644734    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e6a7f63-a247-424b-a491-4d451f994062-kube-api-access-jvrrf\" (UniqueName: \"kubernetes.io/projected/9e6a7f63-a247-424b-a491-4d451f994062-kube-api-access-jvrrf\") pod \"9e6a7f63-a247-424b-a491-4d451f994062\" (UID: \"9e6a7f63-a247-424b-a491-4d451f994062\") "
Oct 30 08:05:57.644918 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:57.644879    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e6a7f63-a247-424b-a491-4d451f994062-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e6a7f63-a247-424b-a491-4d451f994062-maas-api-xks-ca\") pod \"9e6a7f63-a247-424b-a491-4d451f994062\" (UID: \"9e6a7f63-a247-424b-a491-4d451f994062\") "
Oct 30 08:05:57.644918 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:57.644893    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e6a7f63-a247-424b-a491-4d451f994062-maas-api-openshift-ca" pod "9e6a7f63-a247-424b-a491-4d451f994062" (UID: "9e6a7f63-a247-424b-a491-4d451f994062"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:05:57.645302 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:57.645277    6224 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e6a7f63-a247-424b-a491-4d451f994062-maas-api-openshift-ca\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:05:57.648274 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:57.648244    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e6a7f63-a247-424b-a491-4d451f994062-kube-api-access-jvrrf" pod "9e6a7f63-a247-424b-a491-4d451f994062" (UID: "9e6a7f63-a247-424b-a491-4d451f994062"). InnerVolumeSpecName "kube-api-access-jvrrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:05:57.657193 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:57.657162    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e6a7f63-a247-424b-a491-4d451f994062-maas-api-xks-ca" pod "9e6a7f63-a247-424b-a491-4d451f994062" (UID: "9e6a7f63-a247-424b-a491-4d451f994062"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:05:57.747080 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:57.747029    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvrrf\" (UniqueName: \"kubernetes.io/projected/9e6a7f63-a247-424b-a491-4d451f994062-kube-api-access-jvrrf\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:05:57.747080 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:57.747072    6224 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e6a7f63-a247-424b-a491-4d451f994062-maas-api-xks-ca\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:05:58.395878 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:58.395839    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-bd8589-wprkl" event={"ID":"9e6a7f63-a247-424b-a491-4d451f994062","Type":"ContainerDied","Data":"bb288e7a7648e493079924bbf5210b21582ec53695d90043f215303fb6c90d06"}
Oct 30 08:05:58.395878 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:58.395860    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-bd8589-wprkl"
Oct 30 08:05:58.396085 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:58.395898    6224 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb288e7a7648e493079924bbf5210b21582ec53695d90043f215303fb6c90d06"
Oct 30 08:05:58.504134 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:58.504102    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0f5081-hgmc7"]
Oct 30 08:05:58.506746 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:58.506724    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0f5081-hgmc7"]
Oct 30 08:05:59.501531 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:05:59.501497    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d9a2b3b-ed1e-46a9-9e4e-2bc5a21d7aa8" path="/var/lib/kubelet/pods/9d9a2b3b-ed1e-46a9-9e4e-2bc5a21d7aa8/volumes"
Oct 30 08:06:01.857863 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:06:01.857821    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 08:06:04.576172 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:04.576126    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-bd8589-7c4b96684-wqlnp"]
Oct 30 08:06:05.792091 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:05.792066    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e27e0c-6996bc9b75-kmm89"
Oct 30 08:06:05.933358 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:05.933265    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b78cab03-456c-4ea9-93bb-e5c84f7d72a6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b78cab03-456c-4ea9-93bb-e5c84f7d72a6-plugins-config-volume\") pod \"b78cab03-456c-4ea9-93bb-e5c84f7d72a6\" (UID: \"b78cab03-456c-4ea9-93bb-e5c84f7d72a6\") "
Oct 30 08:06:05.933522 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:05.933383    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b78cab03-456c-4ea9-93bb-e5c84f7d72a6-kube-api-access-5w4hn\" (UniqueName: \"kubernetes.io/projected/b78cab03-456c-4ea9-93bb-e5c84f7d72a6-kube-api-access-5w4hn\") pod \"b78cab03-456c-4ea9-93bb-e5c84f7d72a6\" (UID: \"b78cab03-456c-4ea9-93bb-e5c84f7d72a6\") "
Oct 30 08:06:05.933662 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:05.933638    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b78cab03-456c-4ea9-93bb-e5c84f7d72a6-plugins-config-volume" pod "b78cab03-456c-4ea9-93bb-e5c84f7d72a6" (UID: "b78cab03-456c-4ea9-93bb-e5c84f7d72a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:06:05.933748 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:05.933734    6224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b78cab03-456c-4ea9-93bb-e5c84f7d72a6-plugins-config-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:06:05.935588 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:05.935567    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b78cab03-456c-4ea9-93bb-e5c84f7d72a6-kube-api-access-5w4hn" pod "b78cab03-456c-4ea9-93bb-e5c84f7d72a6" (UID: "b78cab03-456c-4ea9-93bb-e5c84f7d72a6"). InnerVolumeSpecName "kube-api-access-5w4hn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:06:06.034620 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:06.034582    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w4hn\" (UniqueName: \"kubernetes.io/projected/b78cab03-456c-4ea9-93bb-e5c84f7d72a6-kube-api-access-5w4hn\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:06:06.441471 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:06.441431    6224 generic.go:356] "Generic (PLEG): container finished" podID="b78cab03-456c-4ea9-93bb-e5c84f7d72a6" containerID="e2903d2883e92218c1ec2b118815e042e99bbf154d027a7e13bea76583c1b221" exitCode=0
Oct 30 08:06:06.441631 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:06.441485    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e27e0c-6996bc9b75-kmm89"
Oct 30 08:06:06.441631 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:06.441512    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e27e0c-6996bc9b75-kmm89" event={"ID":"b78cab03-456c-4ea9-93bb-e5c84f7d72a6","Type":"ContainerDied","Data":"e2903d2883e92218c1ec2b118815e042e99bbf154d027a7e13bea76583c1b221"}
Oct 30 08:06:06.441631 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:06.441575    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e27e0c-6996bc9b75-kmm89" event={"ID":"b78cab03-456c-4ea9-93bb-e5c84f7d72a6","Type":"ContainerDied","Data":"07466f76df80f1050880a43313579b63f92e0273da872f6be67c8130e8a31551"}
Oct 30 08:06:06.441631 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:06.441597    6224 scope.go:122] "RemoveContainer" containerID="e2903d2883e92218c1ec2b118815e042e99bbf154d027a7e13bea76583c1b221"
Oct 30 08:06:06.451997 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:06.451972    6224 scope.go:122] "RemoveContainer" containerID="e2903d2883e92218c1ec2b118815e042e99bbf154d027a7e13bea76583c1b221"
Oct 30 08:06:06.452263 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:06:06.452242    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2903d2883e92218c1ec2b118815e042e99bbf154d027a7e13bea76583c1b221\": container with ID starting with e2903d2883e92218c1ec2b118815e042e99bbf154d027a7e13bea76583c1b221 not found: ID does not exist" containerID="e2903d2883e92218c1ec2b118815e042e99bbf154d027a7e13bea76583c1b221"
Oct 30 08:06:06.452314 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:06.452272    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2903d2883e92218c1ec2b118815e042e99bbf154d027a7e13bea76583c1b221"} err="failed to get container status \"e2903d2883e92218c1ec2b118815e042e99bbf154d027a7e13bea76583c1b221\": rpc error: code = NotFound desc = could not find container \"e2903d2883e92218c1ec2b118815e042e99bbf154d027a7e13bea76583c1b221\": container with ID starting with e2903d2883e92218c1ec2b118815e042e99bbf154d027a7e13bea76583c1b221 not found: ID does not exist"
Oct 30 08:06:06.499187 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:06.499151    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e27e0c-6996bc9b75-kmm89"]
Oct 30 08:06:06.501296 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:06.501267    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e27e0c-6996bc9b75-kmm89"]
Oct 30 08:06:07.502276 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:07.502240    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b78cab03-456c-4ea9-93bb-e5c84f7d72a6" path="/var/lib/kubelet/pods/b78cab03-456c-4ea9-93bb-e5c84f7d72a6/volumes"
Oct 30 08:06:09.576839 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:09.576759    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-bd8589-7c4b96684-wqlnp" podUID="e0988047-1b97-49f4-96c4-a5da4a25d114" containerName="payload-processing" containerID="cri-o://48436f50ba4dc1d50634248e25191e7dbcc333a65c182d498eded4710caf6bba" gracePeriod=25
Oct 30 08:06:12.804172 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:12.804127    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-bd8589-wprkl"]
Oct 30 08:06:12.818184 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:12.818151    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-bd8589-wprkl"]
Oct 30 08:06:12.959241 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:12.959204    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 08:06:13.502279 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:13.502245    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e6a7f63-a247-424b-a491-4d451f994062" path="/var/lib/kubelet/pods/9e6a7f63-a247-424b-a491-4d451f994062/volumes"
Oct 30 08:06:17.257598 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:17.257559    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-ace8fb-5bc7f544dc-nbdvs"]
Oct 30 08:06:18.134113 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:18.134069    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-bd8589-7c4b96684-wqlnp" podUID="e0988047-1b97-49f4-96c4-a5da4a25d114" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:06:22.258368 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:22.258315    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ace8fb-5bc7f544dc-nbdvs" podUID="cce353ad-244f-40c3-95e8-9dc84e054f89" containerName="payload-pre-processing" containerID="cri-o://0b6aafcc3e765018c05c237267e31cea7d47c5014aa6789cde4a90abc38844df" gracePeriod=25
Oct 30 08:06:23.630202 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:23.630160    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq"]
Oct 30 08:06:23.630687 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:23.630412    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq" podUID="61643ced-f0bd-472a-b42b-abb02994320c" containerName="istio-proxy" containerID="cri-o://403af861fff15770b5656b90f6f649ab7a2207b6e236d93c3365c90fc00f7f17" gracePeriod=30
Oct 30 08:06:23.764042 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:23.763998    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ace8fb-5bc7f544dc-nbdvs" podUID="cce353ad-244f-40c3-95e8-9dc84e054f89" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:06:27.789973 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:27.789939    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 08:06:28.134602 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:28.134511    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-bd8589-7c4b96684-wqlnp" podUID="e0988047-1b97-49f4-96c4-a5da4a25d114" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:06:28.352844 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:28.352789    6224 patch_prober.go:29] interesting pod/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 08:06:28.353030 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:28.352868    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq" podUID="61643ced-f0bd-472a-b42b-abb02994320c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 08:06:28.888309 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:28.888282    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq"
Oct 30 08:06:28.957928 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:28.957525    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61643ced-f0bd-472a-b42b-abb02994320c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61643ced-f0bd-472a-b42b-abb02994320c-credential-socket\") pod \"61643ced-f0bd-472a-b42b-abb02994320c\" (UID: \"61643ced-f0bd-472a-b42b-abb02994320c\") "
Oct 30 08:06:28.957928 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:28.957584    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61643ced-f0bd-472a-b42b-abb02994320c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61643ced-f0bd-472a-b42b-abb02994320c-istio-envoy\") pod \"61643ced-f0bd-472a-b42b-abb02994320c\" (UID: \"61643ced-f0bd-472a-b42b-abb02994320c\") "
Oct 30 08:06:28.957928 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:28.957627    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61643ced-f0bd-472a-b42b-abb02994320c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61643ced-f0bd-472a-b42b-abb02994320c-workload-certs\") pod \"61643ced-f0bd-472a-b42b-abb02994320c\" (UID: \"61643ced-f0bd-472a-b42b-abb02994320c\") "
Oct 30 08:06:28.957928 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:28.957677    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61643ced-f0bd-472a-b42b-abb02994320c-kube-api-access-gvx62\" (UniqueName: \"kubernetes.io/projected/61643ced-f0bd-472a-b42b-abb02994320c-kube-api-access-gvx62\") pod \"61643ced-f0bd-472a-b42b-abb02994320c\" (UID: \"61643ced-f0bd-472a-b42b-abb02994320c\") "
Oct 30 08:06:28.957928 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:28.957735    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61643ced-f0bd-472a-b42b-abb02994320c-istio-token\" (UniqueName: \"kubernetes.io/projected/61643ced-f0bd-472a-b42b-abb02994320c-istio-token\") pod \"61643ced-f0bd-472a-b42b-abb02994320c\" (UID: \"61643ced-f0bd-472a-b42b-abb02994320c\") "
Oct 30 08:06:28.957928 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:28.957775    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61643ced-f0bd-472a-b42b-abb02994320c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61643ced-f0bd-472a-b42b-abb02994320c-workload-socket\") pod \"61643ced-f0bd-472a-b42b-abb02994320c\" (UID: \"61643ced-f0bd-472a-b42b-abb02994320c\") "
Oct 30 08:06:28.958379 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:28.958331    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61643ced-f0bd-472a-b42b-abb02994320c-credential-socket" pod "61643ced-f0bd-472a-b42b-abb02994320c" (UID: "61643ced-f0bd-472a-b42b-abb02994320c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:06:28.959253 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:28.958542    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61643ced-f0bd-472a-b42b-abb02994320c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61643ced-f0bd-472a-b42b-abb02994320c-istio-podinfo\") pod \"61643ced-f0bd-472a-b42b-abb02994320c\" (UID: \"61643ced-f0bd-472a-b42b-abb02994320c\") "
Oct 30 08:06:28.959253 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:28.958643    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61643ced-f0bd-472a-b42b-abb02994320c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61643ced-f0bd-472a-b42b-abb02994320c-istio-data\") pod \"61643ced-f0bd-472a-b42b-abb02994320c\" (UID: \"61643ced-f0bd-472a-b42b-abb02994320c\") "
Oct 30 08:06:28.959253 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:28.958672    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61643ced-f0bd-472a-b42b-abb02994320c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61643ced-f0bd-472a-b42b-abb02994320c-istiod-ca-cert\") pod \"61643ced-f0bd-472a-b42b-abb02994320c\" (UID: \"61643ced-f0bd-472a-b42b-abb02994320c\") "
Oct 30 08:06:28.959253 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:28.958783    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61643ced-f0bd-472a-b42b-abb02994320c-workload-socket" pod "61643ced-f0bd-472a-b42b-abb02994320c" (UID: "61643ced-f0bd-472a-b42b-abb02994320c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:06:28.959253 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:28.958794    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61643ced-f0bd-472a-b42b-abb02994320c-workload-certs" pod "61643ced-f0bd-472a-b42b-abb02994320c" (UID: "61643ced-f0bd-472a-b42b-abb02994320c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:06:28.959253 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:28.959089    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61643ced-f0bd-472a-b42b-abb02994320c-istio-data" pod "61643ced-f0bd-472a-b42b-abb02994320c" (UID: "61643ced-f0bd-472a-b42b-abb02994320c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:06:28.959687 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:28.959275    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61643ced-f0bd-472a-b42b-abb02994320c-istiod-ca-cert" pod "61643ced-f0bd-472a-b42b-abb02994320c" (UID: "61643ced-f0bd-472a-b42b-abb02994320c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:06:28.960140 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:28.960110    6224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61643ced-f0bd-472a-b42b-abb02994320c-workload-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:06:28.960257 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:28.960141    6224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61643ced-f0bd-472a-b42b-abb02994320c-istio-data\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:06:28.960257 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:28.960157    6224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61643ced-f0bd-472a-b42b-abb02994320c-istiod-ca-cert\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:06:28.960257 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:28.960170    6224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61643ced-f0bd-472a-b42b-abb02994320c-credential-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:06:28.960257 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:28.960184    6224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61643ced-f0bd-472a-b42b-abb02994320c-workload-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:06:28.961058 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:28.961034    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61643ced-f0bd-472a-b42b-abb02994320c-istio-token" pod "61643ced-f0bd-472a-b42b-abb02994320c" (UID: "61643ced-f0bd-472a-b42b-abb02994320c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:06:28.961375 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:28.961346    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61643ced-f0bd-472a-b42b-abb02994320c-istio-podinfo" pod "61643ced-f0bd-472a-b42b-abb02994320c" (UID: "61643ced-f0bd-472a-b42b-abb02994320c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 08:06:28.961488 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:28.961434    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61643ced-f0bd-472a-b42b-abb02994320c-kube-api-access-gvx62" pod "61643ced-f0bd-472a-b42b-abb02994320c" (UID: "61643ced-f0bd-472a-b42b-abb02994320c"). InnerVolumeSpecName "kube-api-access-gvx62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:06:28.961851 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:28.961833    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61643ced-f0bd-472a-b42b-abb02994320c-istio-envoy" pod "61643ced-f0bd-472a-b42b-abb02994320c" (UID: "61643ced-f0bd-472a-b42b-abb02994320c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:06:29.061330 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:29.061294    6224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61643ced-f0bd-472a-b42b-abb02994320c-istio-envoy\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:06:29.061330 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:29.061322    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvx62\" (UniqueName: \"kubernetes.io/projected/61643ced-f0bd-472a-b42b-abb02994320c-kube-api-access-gvx62\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:06:29.061330 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:29.061333    6224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61643ced-f0bd-472a-b42b-abb02994320c-istio-token\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:06:29.061567 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:29.061341    6224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61643ced-f0bd-472a-b42b-abb02994320c-istio-podinfo\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:06:29.572964 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:29.572925    6224 generic.go:356] "Generic (PLEG): container finished" podID="61643ced-f0bd-472a-b42b-abb02994320c" containerID="403af861fff15770b5656b90f6f649ab7a2207b6e236d93c3365c90fc00f7f17" exitCode=0
Oct 30 08:06:29.573112 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:29.572977    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq"
Oct 30 08:06:29.573112 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:29.572979    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq" event={"ID":"61643ced-f0bd-472a-b42b-abb02994320c","Type":"ContainerDied","Data":"403af861fff15770b5656b90f6f649ab7a2207b6e236d93c3365c90fc00f7f17"}
Oct 30 08:06:29.573112 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:29.573099    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq" event={"ID":"61643ced-f0bd-472a-b42b-abb02994320c","Type":"ContainerDied","Data":"a9b4e2d1fc2148c7ed2c50c77b120e6efb9b00a943e970832a2b921f3eb1cc0d"}
Oct 30 08:06:29.573258 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:29.573130    6224 scope.go:122] "RemoveContainer" containerID="403af861fff15770b5656b90f6f649ab7a2207b6e236d93c3365c90fc00f7f17"
Oct 30 08:06:29.582875 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:29.582855    6224 scope.go:122] "RemoveContainer" containerID="403af861fff15770b5656b90f6f649ab7a2207b6e236d93c3365c90fc00f7f17"
Oct 30 08:06:29.583157 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:06:29.583136    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"403af861fff15770b5656b90f6f649ab7a2207b6e236d93c3365c90fc00f7f17\": container with ID starting with 403af861fff15770b5656b90f6f649ab7a2207b6e236d93c3365c90fc00f7f17 not found: ID does not exist" containerID="403af861fff15770b5656b90f6f649ab7a2207b6e236d93c3365c90fc00f7f17"
Oct 30 08:06:29.583266 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:29.583168    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"403af861fff15770b5656b90f6f649ab7a2207b6e236d93c3365c90fc00f7f17"} err="failed to get container status \"403af861fff15770b5656b90f6f649ab7a2207b6e236d93c3365c90fc00f7f17\": rpc error: code = NotFound desc = could not find container \"403af861fff15770b5656b90f6f649ab7a2207b6e236d93c3365c90fc00f7f17\": container with ID starting with 403af861fff15770b5656b90f6f649ab7a2207b6e236d93c3365c90fc00f7f17 not found: ID does not exist"
Oct 30 08:06:29.656588 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:29.656549    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq"]
Oct 30 08:06:29.667942 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:29.667909    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-bd8589-openshift-default-9c4bb9f47-scnhq"]
Oct 30 08:06:29.827114 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:29.827087    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-bd8589-7c4b96684-wqlnp"
Oct 30 08:06:29.871179 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:29.871147    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0988047-1b97-49f4-96c4-a5da4a25d114-kube-api-access-f2l7w\" (UniqueName: \"kubernetes.io/projected/e0988047-1b97-49f4-96c4-a5da4a25d114-kube-api-access-f2l7w\") pod \"e0988047-1b97-49f4-96c4-a5da4a25d114\" (UID: \"e0988047-1b97-49f4-96c4-a5da4a25d114\") "
Oct 30 08:06:29.871343 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:29.871185    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0988047-1b97-49f4-96c4-a5da4a25d114-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0988047-1b97-49f4-96c4-a5da4a25d114-plugins-config-volume\") pod \"e0988047-1b97-49f4-96c4-a5da4a25d114\" (UID: \"e0988047-1b97-49f4-96c4-a5da4a25d114\") "
Oct 30 08:06:29.871605 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:29.871580    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0988047-1b97-49f4-96c4-a5da4a25d114-plugins-config-volume" pod "e0988047-1b97-49f4-96c4-a5da4a25d114" (UID: "e0988047-1b97-49f4-96c4-a5da4a25d114"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:06:29.873389 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:29.873368    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0988047-1b97-49f4-96c4-a5da4a25d114-kube-api-access-f2l7w" pod "e0988047-1b97-49f4-96c4-a5da4a25d114" (UID: "e0988047-1b97-49f4-96c4-a5da4a25d114"). InnerVolumeSpecName "kube-api-access-f2l7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:06:29.972481 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:29.972443    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2l7w\" (UniqueName: \"kubernetes.io/projected/e0988047-1b97-49f4-96c4-a5da4a25d114-kube-api-access-f2l7w\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:06:29.972481 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:29.972477    6224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0988047-1b97-49f4-96c4-a5da4a25d114-plugins-config-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:06:30.580216 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:30.580181    6224 generic.go:356] "Generic (PLEG): container finished" podID="e0988047-1b97-49f4-96c4-a5da4a25d114" containerID="48436f50ba4dc1d50634248e25191e7dbcc333a65c182d498eded4710caf6bba" exitCode=0
Oct 30 08:06:30.580410 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:30.580226    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-bd8589-7c4b96684-wqlnp"
Oct 30 08:06:30.580410 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:30.580266    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-bd8589-7c4b96684-wqlnp" event={"ID":"e0988047-1b97-49f4-96c4-a5da4a25d114","Type":"ContainerDied","Data":"48436f50ba4dc1d50634248e25191e7dbcc333a65c182d498eded4710caf6bba"}
Oct 30 08:06:30.580410 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:30.580318    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-bd8589-7c4b96684-wqlnp" event={"ID":"e0988047-1b97-49f4-96c4-a5da4a25d114","Type":"ContainerDied","Data":"5bd5341ef76174103b4d969c167700fc03ec9c8aa823299e7ea119144515b9ca"}
Oct 30 08:06:30.580410 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:30.580341    6224 scope.go:122] "RemoveContainer" containerID="48436f50ba4dc1d50634248e25191e7dbcc333a65c182d498eded4710caf6bba"
Oct 30 08:06:30.592840 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:30.592791    6224 scope.go:122] "RemoveContainer" containerID="48436f50ba4dc1d50634248e25191e7dbcc333a65c182d498eded4710caf6bba"
Oct 30 08:06:30.593312 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:06:30.593287    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48436f50ba4dc1d50634248e25191e7dbcc333a65c182d498eded4710caf6bba\": container with ID starting with 48436f50ba4dc1d50634248e25191e7dbcc333a65c182d498eded4710caf6bba not found: ID does not exist" containerID="48436f50ba4dc1d50634248e25191e7dbcc333a65c182d498eded4710caf6bba"
Oct 30 08:06:30.593376 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:30.593315    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48436f50ba4dc1d50634248e25191e7dbcc333a65c182d498eded4710caf6bba"} err="failed to get container status \"48436f50ba4dc1d50634248e25191e7dbcc333a65c182d498eded4710caf6bba\": rpc error: code = NotFound desc = could not find container \"48436f50ba4dc1d50634248e25191e7dbcc333a65c182d498eded4710caf6bba\": container with ID starting with 48436f50ba4dc1d50634248e25191e7dbcc333a65c182d498eded4710caf6bba not found: ID does not exist"
Oct 30 08:06:30.651259 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:30.651226    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-bd8589-7c4b96684-wqlnp"]
Oct 30 08:06:30.664485 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:30.664441    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-bd8589-7c4b96684-wqlnp"]
Oct 30 08:06:31.501972 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:31.501928    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61643ced-f0bd-472a-b42b-abb02994320c" path="/var/lib/kubelet/pods/61643ced-f0bd-472a-b42b-abb02994320c/volumes"
Oct 30 08:06:31.502519 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:31.502491    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0988047-1b97-49f4-96c4-a5da4a25d114" path="/var/lib/kubelet/pods/e0988047-1b97-49f4-96c4-a5da4a25d114/volumes"
Oct 30 08:06:31.857624 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:06:31.857578    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 08:06:33.763535 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:33.763449    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ace8fb-5bc7f544dc-nbdvs" podUID="cce353ad-244f-40c3-95e8-9dc84e054f89" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:06:34.812024 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:34.811991    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-b5998f-7bff7846d-254sj"]
Oct 30 08:06:39.812791 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:39.812732    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-b5998f-7bff7846d-254sj" podUID="1ccf1d22-7e51-4883-86b0-3309d4ffc899" containerName="payload-processing" containerID="cri-o://dcfacb6809f42e60f210e518aa6646b95cc467799fec9e0fa5145d4523d52981" gracePeriod=25
Oct 30 08:06:42.507771 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:42.507743    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ace8fb-5bc7f544dc-nbdvs"
Oct 30 08:06:42.601677 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:42.601639    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cce353ad-244f-40c3-95e8-9dc84e054f89-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cce353ad-244f-40c3-95e8-9dc84e054f89-plugins-config-volume\") pod \"cce353ad-244f-40c3-95e8-9dc84e054f89\" (UID: \"cce353ad-244f-40c3-95e8-9dc84e054f89\") "
Oct 30 08:06:42.601890 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:42.601698    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cce353ad-244f-40c3-95e8-9dc84e054f89-kube-api-access-wpcj4\" (UniqueName: \"kubernetes.io/projected/cce353ad-244f-40c3-95e8-9dc84e054f89-kube-api-access-wpcj4\") pod \"cce353ad-244f-40c3-95e8-9dc84e054f89\" (UID: \"cce353ad-244f-40c3-95e8-9dc84e054f89\") "
Oct 30 08:06:42.602083 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:42.602059    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cce353ad-244f-40c3-95e8-9dc84e054f89-plugins-config-volume" pod "cce353ad-244f-40c3-95e8-9dc84e054f89" (UID: "cce353ad-244f-40c3-95e8-9dc84e054f89"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:06:42.602230 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:42.602210    6224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cce353ad-244f-40c3-95e8-9dc84e054f89-plugins-config-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:06:42.603909 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:42.603885    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cce353ad-244f-40c3-95e8-9dc84e054f89-kube-api-access-wpcj4" pod "cce353ad-244f-40c3-95e8-9dc84e054f89" (UID: "cce353ad-244f-40c3-95e8-9dc84e054f89"). InnerVolumeSpecName "kube-api-access-wpcj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:06:42.647427 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:42.647392    6224 generic.go:356] "Generic (PLEG): container finished" podID="cce353ad-244f-40c3-95e8-9dc84e054f89" containerID="0b6aafcc3e765018c05c237267e31cea7d47c5014aa6789cde4a90abc38844df" exitCode=0
Oct 30 08:06:42.647594 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:42.647433    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ace8fb-5bc7f544dc-nbdvs"
Oct 30 08:06:42.647594 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:42.647465    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ace8fb-5bc7f544dc-nbdvs" event={"ID":"cce353ad-244f-40c3-95e8-9dc84e054f89","Type":"ContainerDied","Data":"0b6aafcc3e765018c05c237267e31cea7d47c5014aa6789cde4a90abc38844df"}
Oct 30 08:06:42.647594 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:42.647522    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ace8fb-5bc7f544dc-nbdvs" event={"ID":"cce353ad-244f-40c3-95e8-9dc84e054f89","Type":"ContainerDied","Data":"c7e145dd053cd95c59733562e25fdd1a4d3a351d4fe148b017db9c172e85e2d4"}
Oct 30 08:06:42.647594 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:42.647547    6224 scope.go:122] "RemoveContainer" containerID="0b6aafcc3e765018c05c237267e31cea7d47c5014aa6789cde4a90abc38844df"
Oct 30 08:06:42.657199 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:42.657131    6224 scope.go:122] "RemoveContainer" containerID="0b6aafcc3e765018c05c237267e31cea7d47c5014aa6789cde4a90abc38844df"
Oct 30 08:06:42.657449 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:06:42.657428    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b6aafcc3e765018c05c237267e31cea7d47c5014aa6789cde4a90abc38844df\": container with ID starting with 0b6aafcc3e765018c05c237267e31cea7d47c5014aa6789cde4a90abc38844df not found: ID does not exist" containerID="0b6aafcc3e765018c05c237267e31cea7d47c5014aa6789cde4a90abc38844df"
Oct 30 08:06:42.657501 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:42.657458    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b6aafcc3e765018c05c237267e31cea7d47c5014aa6789cde4a90abc38844df"} err="failed to get container status \"0b6aafcc3e765018c05c237267e31cea7d47c5014aa6789cde4a90abc38844df\": rpc error: code = NotFound desc = could not find container \"0b6aafcc3e765018c05c237267e31cea7d47c5014aa6789cde4a90abc38844df\": container with ID starting with 0b6aafcc3e765018c05c237267e31cea7d47c5014aa6789cde4a90abc38844df not found: ID does not exist"
Oct 30 08:06:42.702949 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:42.702919    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpcj4\" (UniqueName: \"kubernetes.io/projected/cce353ad-244f-40c3-95e8-9dc84e054f89-kube-api-access-wpcj4\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:06:42.704497 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:42.704463    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-ace8fb-5bc7f544dc-nbdvs"]
Oct 30 08:06:42.708703 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:42.708677    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-ace8fb-5bc7f544dc-nbdvs"]
Oct 30 08:06:42.819329 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:42.819285    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b5998f-57685c68cb-xts98"]
Oct 30 08:06:42.819947 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:42.819890    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b5998f-57685c68cb-xts98" podUID="afe4bac2-032c-41ce-90a8-34d3619eebb1" containerName="maas-api" containerID="cri-o://0bdfc61a316f1e327e490ae86907ede7a04590ad0870e18d50e2c8bccc4281a6" gracePeriod=30
Oct 30 08:06:43.062159 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:43.062134    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b5998f-57685c68cb-xts98"
Oct 30 08:06:43.210025 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:43.209926    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afe4bac2-032c-41ce-90a8-34d3619eebb1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afe4bac2-032c-41ce-90a8-34d3619eebb1-maas-api-tls\") pod \"afe4bac2-032c-41ce-90a8-34d3619eebb1\" (UID: \"afe4bac2-032c-41ce-90a8-34d3619eebb1\") "
Oct 30 08:06:43.210189 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:43.210030    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afe4bac2-032c-41ce-90a8-34d3619eebb1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afe4bac2-032c-41ce-90a8-34d3619eebb1-openshift-service-ca\") pod \"afe4bac2-032c-41ce-90a8-34d3619eebb1\" (UID: \"afe4bac2-032c-41ce-90a8-34d3619eebb1\") "
Oct 30 08:06:43.210238 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:43.210197    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afe4bac2-032c-41ce-90a8-34d3619eebb1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/afe4bac2-032c-41ce-90a8-34d3619eebb1-metrics-certs\") pod \"afe4bac2-032c-41ce-90a8-34d3619eebb1\" (UID: \"afe4bac2-032c-41ce-90a8-34d3619eebb1\") "
Oct 30 08:06:43.210336 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:43.210318    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afe4bac2-032c-41ce-90a8-34d3619eebb1-kube-api-access-xqfft\" (UniqueName: \"kubernetes.io/projected/afe4bac2-032c-41ce-90a8-34d3619eebb1-kube-api-access-xqfft\") pod \"afe4bac2-032c-41ce-90a8-34d3619eebb1\" (UID: \"afe4bac2-032c-41ce-90a8-34d3619eebb1\") "
Oct 30 08:06:43.210376 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:43.210331    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afe4bac2-032c-41ce-90a8-34d3619eebb1-openshift-service-ca" pod "afe4bac2-032c-41ce-90a8-34d3619eebb1" (UID: "afe4bac2-032c-41ce-90a8-34d3619eebb1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:06:43.210609 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:43.210591    6224 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afe4bac2-032c-41ce-90a8-34d3619eebb1-openshift-service-ca\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:06:43.212257 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:43.212224    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afe4bac2-032c-41ce-90a8-34d3619eebb1-maas-api-tls" pod "afe4bac2-032c-41ce-90a8-34d3619eebb1" (UID: "afe4bac2-032c-41ce-90a8-34d3619eebb1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:06:43.212257 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:43.212234    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afe4bac2-032c-41ce-90a8-34d3619eebb1-kube-api-access-xqfft" pod "afe4bac2-032c-41ce-90a8-34d3619eebb1" (UID: "afe4bac2-032c-41ce-90a8-34d3619eebb1"). InnerVolumeSpecName "kube-api-access-xqfft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:06:43.212405 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:43.212268    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afe4bac2-032c-41ce-90a8-34d3619eebb1-metrics-certs" pod "afe4bac2-032c-41ce-90a8-34d3619eebb1" (UID: "afe4bac2-032c-41ce-90a8-34d3619eebb1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:06:43.311824 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:43.311776    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqfft\" (UniqueName: \"kubernetes.io/projected/afe4bac2-032c-41ce-90a8-34d3619eebb1-kube-api-access-xqfft\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:06:43.311975 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:43.311848    6224 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afe4bac2-032c-41ce-90a8-34d3619eebb1-maas-api-tls\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:06:43.311975 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:43.311867    6224 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afe4bac2-032c-41ce-90a8-34d3619eebb1-metrics-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:06:43.502691 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:43.502607    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cce353ad-244f-40c3-95e8-9dc84e054f89" path="/var/lib/kubelet/pods/cce353ad-244f-40c3-95e8-9dc84e054f89/volumes"
Oct 30 08:06:43.654565 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:43.654527    6224 generic.go:356] "Generic (PLEG): container finished" podID="afe4bac2-032c-41ce-90a8-34d3619eebb1" containerID="0bdfc61a316f1e327e490ae86907ede7a04590ad0870e18d50e2c8bccc4281a6" exitCode=0
Oct 30 08:06:43.655023 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:43.654588    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b5998f-57685c68cb-xts98"
Oct 30 08:06:43.655023 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:43.654614    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b5998f-57685c68cb-xts98" event={"ID":"afe4bac2-032c-41ce-90a8-34d3619eebb1","Type":"ContainerDied","Data":"0bdfc61a316f1e327e490ae86907ede7a04590ad0870e18d50e2c8bccc4281a6"}
Oct 30 08:06:43.655023 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:43.654667    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b5998f-57685c68cb-xts98" event={"ID":"afe4bac2-032c-41ce-90a8-34d3619eebb1","Type":"ContainerDied","Data":"4c09f7e3483e8c6599238713a635d5ace0188bb48b142793c8bb137eb7039907"}
Oct 30 08:06:43.655023 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:43.654693    6224 scope.go:122] "RemoveContainer" containerID="0bdfc61a316f1e327e490ae86907ede7a04590ad0870e18d50e2c8bccc4281a6"
Oct 30 08:06:43.664067 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:43.664044    6224 scope.go:122] "RemoveContainer" containerID="0bdfc61a316f1e327e490ae86907ede7a04590ad0870e18d50e2c8bccc4281a6"
Oct 30 08:06:43.664357 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:06:43.664333    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bdfc61a316f1e327e490ae86907ede7a04590ad0870e18d50e2c8bccc4281a6\": container with ID starting with 0bdfc61a316f1e327e490ae86907ede7a04590ad0870e18d50e2c8bccc4281a6 not found: ID does not exist" containerID="0bdfc61a316f1e327e490ae86907ede7a04590ad0870e18d50e2c8bccc4281a6"
Oct 30 08:06:43.664423 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:43.664366    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bdfc61a316f1e327e490ae86907ede7a04590ad0870e18d50e2c8bccc4281a6"} err="failed to get container status \"0bdfc61a316f1e327e490ae86907ede7a04590ad0870e18d50e2c8bccc4281a6\": rpc error: code = NotFound desc = could not find container \"0bdfc61a316f1e327e490ae86907ede7a04590ad0870e18d50e2c8bccc4281a6\": container with ID starting with 0bdfc61a316f1e327e490ae86907ede7a04590ad0870e18d50e2c8bccc4281a6 not found: ID does not exist"
Oct 30 08:06:43.713168 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:43.713137    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b5998f-57685c68cb-xts98"]
Oct 30 08:06:43.727352 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:43.727320    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b5998f-57685c68cb-xts98"]
Oct 30 08:06:45.502025 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:45.501992    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afe4bac2-032c-41ce-90a8-34d3619eebb1" path="/var/lib/kubelet/pods/afe4bac2-032c-41ce-90a8-34d3619eebb1/volumes"
Oct 30 08:06:47.911459 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:47.911414    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 08:06:49.811706 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:49.811656    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-b5998f-7bff7846d-254sj" podUID="1ccf1d22-7e51-4883-86b0-3309d4ffc899" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:06:51.118991 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:51.118948    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz"]
Oct 30 08:06:51.119469 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:51.119160    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz" podUID="f15b9c95-390e-4258-a66e-0a03806392d9" containerName="istio-proxy" containerID="cri-o://bf2ba91f603ed61edac39a7b8fd589f07276ef239a4ae2a5340a0d663e4edccf" gracePeriod=30
Oct 30 08:06:51.837188 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:51.837145    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-06b26e-6g5fv"]
Oct 30 08:06:51.837606 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:51.837587    6224 cpu_manager.go:414] "containerMap: removing container" podUID="cce353ad-244f-40c3-95e8-9dc84e054f89" containerName="payload-pre-processing"
Oct 30 08:06:51.837700 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:51.837608    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cce353ad-244f-40c3-95e8-9dc84e054f89" containerName="payload-pre-processing"
Oct 30 08:06:51.837700 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:51.837623    6224 cpu_manager.go:414] "containerMap: removing container" podUID="61643ced-f0bd-472a-b42b-abb02994320c" containerName="istio-proxy"
Oct 30 08:06:51.837700 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:51.837633    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61643ced-f0bd-472a-b42b-abb02994320c" containerName="istio-proxy"
Oct 30 08:06:51.837700 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:51.837649    6224 cpu_manager.go:414] "containerMap: removing container" podUID="b78cab03-456c-4ea9-93bb-e5c84f7d72a6" containerName="payload-pre-processing"
Oct 30 08:06:51.837700 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:51.837657    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b78cab03-456c-4ea9-93bb-e5c84f7d72a6" containerName="payload-pre-processing"
Oct 30 08:06:51.837700 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:51.837671    6224 cpu_manager.go:414] "containerMap: removing container" podUID="9e6a7f63-a247-424b-a491-4d451f994062" containerName="revoke-keys"
Oct 30 08:06:51.837700 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:51.837679    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e6a7f63-a247-424b-a491-4d451f994062" containerName="revoke-keys"
Oct 30 08:06:51.838131 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:51.837720    6224 cpu_manager.go:414] "containerMap: removing container" podUID="afe4bac2-032c-41ce-90a8-34d3619eebb1" containerName="maas-api"
Oct 30 08:06:51.838131 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:51.837730    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afe4bac2-032c-41ce-90a8-34d3619eebb1" containerName="maas-api"
Oct 30 08:06:51.838131 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:51.837751    6224 cpu_manager.go:414] "containerMap: removing container" podUID="e0988047-1b97-49f4-96c4-a5da4a25d114" containerName="payload-processing"
Oct 30 08:06:51.838131 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:51.837758    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0988047-1b97-49f4-96c4-a5da4a25d114" containerName="payload-processing"
Oct 30 08:06:51.838131 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:51.837878    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="cce353ad-244f-40c3-95e8-9dc84e054f89" containerName="payload-pre-processing"
Oct 30 08:06:51.838131 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:51.837894    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="61643ced-f0bd-472a-b42b-abb02994320c" containerName="istio-proxy"
Oct 30 08:06:51.838131 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:51.837906    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="b78cab03-456c-4ea9-93bb-e5c84f7d72a6" containerName="payload-pre-processing"
Oct 30 08:06:51.838131 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:51.837918    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e6a7f63-a247-424b-a491-4d451f994062" containerName="revoke-keys"
Oct 30 08:06:51.838131 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:51.837934    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="afe4bac2-032c-41ce-90a8-34d3619eebb1" containerName="maas-api"
Oct 30 08:06:51.838131 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:51.837946    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0988047-1b97-49f4-96c4-a5da4a25d114" containerName="payload-processing"
Oct 30 08:06:51.842451 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:51.842430    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-06b26e-6g5fv"
Oct 30 08:06:51.860190 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:51.860161    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-06b26e-6g5fv"]
Oct 30 08:06:51.907046 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:51.907012    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 08:06:51.907241 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:51.907109    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 08:06:52.000931 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:52.000898    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f717f7f2-6cec-4152-b45a-82a04df95be1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-06b26e-6g5fv\" (UID: \"f717f7f2-6cec-4152-b45a-82a04df95be1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-06b26e-6g5fv"
Oct 30 08:06:52.001112 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:52.001039    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw7nv\" (UniqueName: \"kubernetes.io/projected/f717f7f2-6cec-4152-b45a-82a04df95be1-kube-api-access-fw7nv\") pod \"maas-api-revoke-keys-e2e-worker-w0-06b26e-6g5fv\" (UID: \"f717f7f2-6cec-4152-b45a-82a04df95be1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-06b26e-6g5fv"
Oct 30 08:06:52.001112 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:52.001104    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f717f7f2-6cec-4152-b45a-82a04df95be1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-06b26e-6g5fv\" (UID: \"f717f7f2-6cec-4152-b45a-82a04df95be1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-06b26e-6g5fv"
Oct 30 08:06:52.102613 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:52.102517    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw7nv\" (UniqueName: \"kubernetes.io/projected/f717f7f2-6cec-4152-b45a-82a04df95be1-kube-api-access-fw7nv\") pod \"maas-api-revoke-keys-e2e-worker-w0-06b26e-6g5fv\" (UID: \"f717f7f2-6cec-4152-b45a-82a04df95be1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-06b26e-6g5fv"
Oct 30 08:06:52.102613 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:52.102579    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f717f7f2-6cec-4152-b45a-82a04df95be1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-06b26e-6g5fv\" (UID: \"f717f7f2-6cec-4152-b45a-82a04df95be1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-06b26e-6g5fv"
Oct 30 08:06:52.102954 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:52.102628    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f717f7f2-6cec-4152-b45a-82a04df95be1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-06b26e-6g5fv\" (UID: \"f717f7f2-6cec-4152-b45a-82a04df95be1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-06b26e-6g5fv"
Oct 30 08:06:52.103341 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:52.103317    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f717f7f2-6cec-4152-b45a-82a04df95be1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-06b26e-6g5fv\" (UID: \"f717f7f2-6cec-4152-b45a-82a04df95be1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-06b26e-6g5fv"
Oct 30 08:06:52.105200 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:52.105166    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f717f7f2-6cec-4152-b45a-82a04df95be1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-06b26e-6g5fv\" (UID: \"f717f7f2-6cec-4152-b45a-82a04df95be1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-06b26e-6g5fv"
Oct 30 08:06:52.130021 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:52.129985    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw7nv\" (UniqueName: \"kubernetes.io/projected/f717f7f2-6cec-4152-b45a-82a04df95be1-kube-api-access-fw7nv\") pod \"maas-api-revoke-keys-e2e-worker-w0-06b26e-6g5fv\" (UID: \"f717f7f2-6cec-4152-b45a-82a04df95be1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-06b26e-6g5fv"
Oct 30 08:06:52.155579 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:52.155546    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-06b26e-6g5fv"
Oct 30 08:06:52.451166 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:52.451138    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-06b26e-6g5fv"]
Oct 30 08:06:52.451391 ip-10-0-128-125 kubenswrapper[6224]: W1030 08:06:52.451363    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf717f7f2_6cec_4152_b45a_82a04df95be1.slice/crio-d7eebb7b42ddbd28c9f19570915a8711d4fc2567a92a00c1292461fcfa8cdccc WatchSource:0}: Error finding container d7eebb7b42ddbd28c9f19570915a8711d4fc2567a92a00c1292461fcfa8cdccc: Status 404 returned error can't find the container with id d7eebb7b42ddbd28c9f19570915a8711d4fc2567a92a00c1292461fcfa8cdccc
Oct 30 08:06:52.709395 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:52.709308    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-06b26e-6g5fv" event={"ID":"f717f7f2-6cec-4152-b45a-82a04df95be1","Type":"ContainerStarted","Data":"d7eebb7b42ddbd28c9f19570915a8711d4fc2567a92a00c1292461fcfa8cdccc"}
Oct 30 08:06:53.715869 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:53.715778    6224 generic.go:356] "Generic (PLEG): container finished" podID="f717f7f2-6cec-4152-b45a-82a04df95be1" containerID="e14349c45f55fae734e4c1c34aa5c35cd41463d2048ed1f4aeb214e222041883" exitCode=0
Oct 30 08:06:53.716249 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:53.715862    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-06b26e-6g5fv" event={"ID":"f717f7f2-6cec-4152-b45a-82a04df95be1","Type":"ContainerDied","Data":"e14349c45f55fae734e4c1c34aa5c35cd41463d2048ed1f4aeb214e222041883"}
Oct 30 08:06:54.871666 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:54.871638    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-06b26e-6g5fv"
Oct 30 08:06:55.033868 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:55.033793    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f717f7f2-6cec-4152-b45a-82a04df95be1-kube-api-access-fw7nv\" (UniqueName: \"kubernetes.io/projected/f717f7f2-6cec-4152-b45a-82a04df95be1-kube-api-access-fw7nv\") pod \"f717f7f2-6cec-4152-b45a-82a04df95be1\" (UID: \"f717f7f2-6cec-4152-b45a-82a04df95be1\") "
Oct 30 08:06:55.034362 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:55.033900    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f717f7f2-6cec-4152-b45a-82a04df95be1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f717f7f2-6cec-4152-b45a-82a04df95be1-maas-api-xks-ca\") pod \"f717f7f2-6cec-4152-b45a-82a04df95be1\" (UID: \"f717f7f2-6cec-4152-b45a-82a04df95be1\") "
Oct 30 08:06:55.034362 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:55.033954    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f717f7f2-6cec-4152-b45a-82a04df95be1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f717f7f2-6cec-4152-b45a-82a04df95be1-maas-api-openshift-ca\") pod \"f717f7f2-6cec-4152-b45a-82a04df95be1\" (UID: \"f717f7f2-6cec-4152-b45a-82a04df95be1\") "
Oct 30 08:06:55.034362 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:55.034321    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f717f7f2-6cec-4152-b45a-82a04df95be1-maas-api-openshift-ca" pod "f717f7f2-6cec-4152-b45a-82a04df95be1" (UID: "f717f7f2-6cec-4152-b45a-82a04df95be1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:06:55.035894 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:55.035871    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f717f7f2-6cec-4152-b45a-82a04df95be1-maas-api-xks-ca" pod "f717f7f2-6cec-4152-b45a-82a04df95be1" (UID: "f717f7f2-6cec-4152-b45a-82a04df95be1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:06:55.035989 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:55.035957    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f717f7f2-6cec-4152-b45a-82a04df95be1-kube-api-access-fw7nv" pod "f717f7f2-6cec-4152-b45a-82a04df95be1" (UID: "f717f7f2-6cec-4152-b45a-82a04df95be1"). InnerVolumeSpecName "kube-api-access-fw7nv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:06:55.135298 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:55.135256    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw7nv\" (UniqueName: \"kubernetes.io/projected/f717f7f2-6cec-4152-b45a-82a04df95be1-kube-api-access-fw7nv\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:06:55.135298 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:55.135292    6224 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f717f7f2-6cec-4152-b45a-82a04df95be1-maas-api-xks-ca\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:06:55.135298 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:55.135302    6224 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f717f7f2-6cec-4152-b45a-82a04df95be1-maas-api-openshift-ca\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:06:55.731055 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:55.731018    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-06b26e-6g5fv" event={"ID":"f717f7f2-6cec-4152-b45a-82a04df95be1","Type":"ContainerDied","Data":"d7eebb7b42ddbd28c9f19570915a8711d4fc2567a92a00c1292461fcfa8cdccc"}
Oct 30 08:06:55.731238 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:55.731061    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-06b26e-6g5fv"
Oct 30 08:06:55.731238 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:55.731069    6224 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7eebb7b42ddbd28c9f19570915a8711d4fc2567a92a00c1292461fcfa8cdccc"
Oct 30 08:06:56.373425 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:56.373392    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz"
Oct 30 08:06:56.446255 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:56.446167    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f15b9c95-390e-4258-a66e-0a03806392d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f15b9c95-390e-4258-a66e-0a03806392d9-credential-socket\") pod \"f15b9c95-390e-4258-a66e-0a03806392d9\" (UID: \"f15b9c95-390e-4258-a66e-0a03806392d9\") "
Oct 30 08:06:56.446255 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:56.446204    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f15b9c95-390e-4258-a66e-0a03806392d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f15b9c95-390e-4258-a66e-0a03806392d9-istio-podinfo\") pod \"f15b9c95-390e-4258-a66e-0a03806392d9\" (UID: \"f15b9c95-390e-4258-a66e-0a03806392d9\") "
Oct 30 08:06:56.446255 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:56.446224    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f15b9c95-390e-4258-a66e-0a03806392d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f15b9c95-390e-4258-a66e-0a03806392d9-workload-socket\") pod \"f15b9c95-390e-4258-a66e-0a03806392d9\" (UID: \"f15b9c95-390e-4258-a66e-0a03806392d9\") "
Oct 30 08:06:56.446255 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:56.446246    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f15b9c95-390e-4258-a66e-0a03806392d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f15b9c95-390e-4258-a66e-0a03806392d9-workload-certs\") pod \"f15b9c95-390e-4258-a66e-0a03806392d9\" (UID: \"f15b9c95-390e-4258-a66e-0a03806392d9\") "
Oct 30 08:06:56.446575 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:56.446353    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f15b9c95-390e-4258-a66e-0a03806392d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f15b9c95-390e-4258-a66e-0a03806392d9-istio-envoy\") pod \"f15b9c95-390e-4258-a66e-0a03806392d9\" (UID: \"f15b9c95-390e-4258-a66e-0a03806392d9\") "
Oct 30 08:06:56.446575 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:56.446394    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f15b9c95-390e-4258-a66e-0a03806392d9-kube-api-access-q67lh\" (UniqueName: \"kubernetes.io/projected/f15b9c95-390e-4258-a66e-0a03806392d9-kube-api-access-q67lh\") pod \"f15b9c95-390e-4258-a66e-0a03806392d9\" (UID: \"f15b9c95-390e-4258-a66e-0a03806392d9\") "
Oct 30 08:06:56.446575 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:56.446430    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f15b9c95-390e-4258-a66e-0a03806392d9-istio-token\" (UniqueName: \"kubernetes.io/projected/f15b9c95-390e-4258-a66e-0a03806392d9-istio-token\") pod \"f15b9c95-390e-4258-a66e-0a03806392d9\" (UID: \"f15b9c95-390e-4258-a66e-0a03806392d9\") "
Oct 30 08:06:56.446575 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:56.446471    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f15b9c95-390e-4258-a66e-0a03806392d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f15b9c95-390e-4258-a66e-0a03806392d9-istiod-ca-cert\") pod \"f15b9c95-390e-4258-a66e-0a03806392d9\" (UID: \"f15b9c95-390e-4258-a66e-0a03806392d9\") "
Oct 30 08:06:56.446575 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:56.446528    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f15b9c95-390e-4258-a66e-0a03806392d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f15b9c95-390e-4258-a66e-0a03806392d9-istio-data\") pod \"f15b9c95-390e-4258-a66e-0a03806392d9\" (UID: \"f15b9c95-390e-4258-a66e-0a03806392d9\") "
Oct 30 08:06:56.446575 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:56.446530    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f15b9c95-390e-4258-a66e-0a03806392d9-credential-socket" pod "f15b9c95-390e-4258-a66e-0a03806392d9" (UID: "f15b9c95-390e-4258-a66e-0a03806392d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:06:56.446575 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:56.446527    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f15b9c95-390e-4258-a66e-0a03806392d9-workload-socket" pod "f15b9c95-390e-4258-a66e-0a03806392d9" (UID: "f15b9c95-390e-4258-a66e-0a03806392d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:06:56.446895 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:56.446574    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f15b9c95-390e-4258-a66e-0a03806392d9-workload-certs" pod "f15b9c95-390e-4258-a66e-0a03806392d9" (UID: "f15b9c95-390e-4258-a66e-0a03806392d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:06:56.446895 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:56.446853    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f15b9c95-390e-4258-a66e-0a03806392d9-istio-data" pod "f15b9c95-390e-4258-a66e-0a03806392d9" (UID: "f15b9c95-390e-4258-a66e-0a03806392d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:06:56.446895 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:56.446874    6224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f15b9c95-390e-4258-a66e-0a03806392d9-credential-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:06:56.446895 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:56.446891    6224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f15b9c95-390e-4258-a66e-0a03806392d9-workload-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:06:56.447073 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:56.446902    6224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f15b9c95-390e-4258-a66e-0a03806392d9-workload-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:06:56.447073 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:56.446887    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f15b9c95-390e-4258-a66e-0a03806392d9-istiod-ca-cert" pod "f15b9c95-390e-4258-a66e-0a03806392d9" (UID: "f15b9c95-390e-4258-a66e-0a03806392d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:06:56.448710 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:56.448679    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f15b9c95-390e-4258-a66e-0a03806392d9-istio-token" pod "f15b9c95-390e-4258-a66e-0a03806392d9" (UID: "f15b9c95-390e-4258-a66e-0a03806392d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:06:56.448988 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:56.448949    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f15b9c95-390e-4258-a66e-0a03806392d9-kube-api-access-q67lh" pod "f15b9c95-390e-4258-a66e-0a03806392d9" (UID: "f15b9c95-390e-4258-a66e-0a03806392d9"). InnerVolumeSpecName "kube-api-access-q67lh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:06:56.449163 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:56.449134    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f15b9c95-390e-4258-a66e-0a03806392d9-istio-podinfo" pod "f15b9c95-390e-4258-a66e-0a03806392d9" (UID: "f15b9c95-390e-4258-a66e-0a03806392d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 08:06:56.449257 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:56.449214    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f15b9c95-390e-4258-a66e-0a03806392d9-istio-envoy" pod "f15b9c95-390e-4258-a66e-0a03806392d9" (UID: "f15b9c95-390e-4258-a66e-0a03806392d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:06:56.547497 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:56.547458    6224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f15b9c95-390e-4258-a66e-0a03806392d9-istio-podinfo\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:06:56.547497 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:56.547491    6224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f15b9c95-390e-4258-a66e-0a03806392d9-istio-envoy\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:06:56.547497 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:56.547501    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q67lh\" (UniqueName: \"kubernetes.io/projected/f15b9c95-390e-4258-a66e-0a03806392d9-kube-api-access-q67lh\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:06:56.547753 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:56.547512    6224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f15b9c95-390e-4258-a66e-0a03806392d9-istio-token\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:06:56.547753 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:56.547521    6224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f15b9c95-390e-4258-a66e-0a03806392d9-istiod-ca-cert\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:06:56.547753 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:56.547528    6224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f15b9c95-390e-4258-a66e-0a03806392d9-istio-data\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:06:56.573170 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:56.573135    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-06b26e/distinct-w0-06b26e-kserve-545cf57898-2fhhd"]
Oct 30 08:06:56.589620 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:56.589591    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-06b26e/premium-w0-06b26e-kserve-c9dbd9879-b4jlk"]
Oct 30 08:06:56.738609 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:56.738522    6224 generic.go:356] "Generic (PLEG): container finished" podID="f15b9c95-390e-4258-a66e-0a03806392d9" containerID="bf2ba91f603ed61edac39a7b8fd589f07276ef239a4ae2a5340a0d663e4edccf" exitCode=0
Oct 30 08:06:56.738609 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:56.738578    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz"
Oct 30 08:06:56.738791 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:56.738604    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz" event={"ID":"f15b9c95-390e-4258-a66e-0a03806392d9","Type":"ContainerDied","Data":"bf2ba91f603ed61edac39a7b8fd589f07276ef239a4ae2a5340a0d663e4edccf"}
Oct 30 08:06:56.738791 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:56.738658    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz" event={"ID":"f15b9c95-390e-4258-a66e-0a03806392d9","Type":"ContainerDied","Data":"837b2ec2c6615aa7bbda9aec1f57b1112844aa1945adf45896469be6eff55a53"}
Oct 30 08:06:56.738791 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:56.738674    6224 scope.go:122] "RemoveContainer" containerID="bf2ba91f603ed61edac39a7b8fd589f07276ef239a4ae2a5340a0d663e4edccf"
Oct 30 08:06:56.749244 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:56.749225    6224 scope.go:122] "RemoveContainer" containerID="bf2ba91f603ed61edac39a7b8fd589f07276ef239a4ae2a5340a0d663e4edccf"
Oct 30 08:06:56.749562 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:06:56.749541    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf2ba91f603ed61edac39a7b8fd589f07276ef239a4ae2a5340a0d663e4edccf\": container with ID starting with bf2ba91f603ed61edac39a7b8fd589f07276ef239a4ae2a5340a0d663e4edccf not found: ID does not exist" containerID="bf2ba91f603ed61edac39a7b8fd589f07276ef239a4ae2a5340a0d663e4edccf"
Oct 30 08:06:56.749653 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:56.749570    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf2ba91f603ed61edac39a7b8fd589f07276ef239a4ae2a5340a0d663e4edccf"} err="failed to get container status \"bf2ba91f603ed61edac39a7b8fd589f07276ef239a4ae2a5340a0d663e4edccf\": rpc error: code = NotFound desc = could not find container \"bf2ba91f603ed61edac39a7b8fd589f07276ef239a4ae2a5340a0d663e4edccf\": container with ID starting with bf2ba91f603ed61edac39a7b8fd589f07276ef239a4ae2a5340a0d663e4edccf not found: ID does not exist"
Oct 30 08:06:56.801287 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:56.801252    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz"]
Oct 30 08:06:56.806837 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:56.806793    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-b5998f-openshift-default-d57ddfcc9-llfmz"]
Oct 30 08:06:57.025692 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:57.025614    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f5ceca-q52n4"]
Oct 30 08:06:57.026027 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:57.026013    6224 cpu_manager.go:414] "containerMap: removing container" podUID="f15b9c95-390e-4258-a66e-0a03806392d9" containerName="istio-proxy"
Oct 30 08:06:57.026083 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:57.026029    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f15b9c95-390e-4258-a66e-0a03806392d9" containerName="istio-proxy"
Oct 30 08:06:57.026083 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:57.026058    6224 cpu_manager.go:414] "containerMap: removing container" podUID="f717f7f2-6cec-4152-b45a-82a04df95be1" containerName="revoke-keys"
Oct 30 08:06:57.026083 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:57.026063    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f717f7f2-6cec-4152-b45a-82a04df95be1" containerName="revoke-keys"
Oct 30 08:06:57.026192 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:57.026129    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="f717f7f2-6cec-4152-b45a-82a04df95be1" containerName="revoke-keys"
Oct 30 08:06:57.026192 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:57.026142    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="f15b9c95-390e-4258-a66e-0a03806392d9" containerName="istio-proxy"
Oct 30 08:06:57.030490 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:57.030471    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f5ceca-q52n4"
Oct 30 08:06:57.049134 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:57.049094    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f5ceca-q52n4"]
Oct 30 08:06:57.051662 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:57.051628    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4de1ddb-4b85-416d-9425-010831c97883-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-f5ceca-q52n4\" (UID: \"a4de1ddb-4b85-416d-9425-010831c97883\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f5ceca-q52n4"
Oct 30 08:06:57.051791 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:57.051688    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jctk\" (UniqueName: \"kubernetes.io/projected/a4de1ddb-4b85-416d-9425-010831c97883-kube-api-access-8jctk\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-f5ceca-q52n4\" (UID: \"a4de1ddb-4b85-416d-9425-010831c97883\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f5ceca-q52n4"
Oct 30 08:06:57.051862 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:57.051825    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4de1ddb-4b85-416d-9425-010831c97883-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-f5ceca-q52n4\" (UID: \"a4de1ddb-4b85-416d-9425-010831c97883\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f5ceca-q52n4"
Oct 30 08:06:57.135844 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:57.135790    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 08:06:57.136081 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:57.136061    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 08:06:57.152578 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:57.152545    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4de1ddb-4b85-416d-9425-010831c97883-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-f5ceca-q52n4\" (UID: \"a4de1ddb-4b85-416d-9425-010831c97883\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f5ceca-q52n4"
Oct 30 08:06:57.152745 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:57.152619    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4de1ddb-4b85-416d-9425-010831c97883-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-f5ceca-q52n4\" (UID: \"a4de1ddb-4b85-416d-9425-010831c97883\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f5ceca-q52n4"
Oct 30 08:06:57.152745 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:57.152650    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jctk\" (UniqueName: \"kubernetes.io/projected/a4de1ddb-4b85-416d-9425-010831c97883-kube-api-access-8jctk\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-f5ceca-q52n4\" (UID: \"a4de1ddb-4b85-416d-9425-010831c97883\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f5ceca-q52n4"
Oct 30 08:06:57.153301 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:57.153281    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4de1ddb-4b85-416d-9425-010831c97883-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-f5ceca-q52n4\" (UID: \"a4de1ddb-4b85-416d-9425-010831c97883\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f5ceca-q52n4"
Oct 30 08:06:57.155108 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:57.155085    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4de1ddb-4b85-416d-9425-010831c97883-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-f5ceca-q52n4\" (UID: \"a4de1ddb-4b85-416d-9425-010831c97883\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f5ceca-q52n4"
Oct 30 08:06:57.178259 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:57.178228    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jctk\" (UniqueName: \"kubernetes.io/projected/a4de1ddb-4b85-416d-9425-010831c97883-kube-api-access-8jctk\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-f5ceca-q52n4\" (UID: \"a4de1ddb-4b85-416d-9425-010831c97883\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f5ceca-q52n4"
Oct 30 08:06:57.344031 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:57.343989    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f5ceca-q52n4"
Oct 30 08:06:57.502130 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:57.502090    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f15b9c95-390e-4258-a66e-0a03806392d9" path="/var/lib/kubelet/pods/f15b9c95-390e-4258-a66e-0a03806392d9/volumes"
Oct 30 08:06:57.641551 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:57.641508    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f5ceca-q52n4"]
Oct 30 08:06:57.642065 ip-10-0-128-125 kubenswrapper[6224]: W1030 08:06:57.642040    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4de1ddb_4b85_416d_9425_010831c97883.slice/crio-1ffdf645bbde3ca0942f47238e6e2a24d9833daa6c8ef698f4191093452906ff WatchSource:0}: Error finding container 1ffdf645bbde3ca0942f47238e6e2a24d9833daa6c8ef698f4191093452906ff: Status 404 returned error can't find the container with id 1ffdf645bbde3ca0942f47238e6e2a24d9833daa6c8ef698f4191093452906ff
Oct 30 08:06:57.746303 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:57.746271    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f5ceca-q52n4" event={"ID":"a4de1ddb-4b85-416d-9425-010831c97883","Type":"ContainerStarted","Data":"1ffdf645bbde3ca0942f47238e6e2a24d9833daa6c8ef698f4191093452906ff"}
Oct 30 08:06:58.752874 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:58.752757    6224 generic.go:356] "Generic (PLEG): container finished" podID="a4de1ddb-4b85-416d-9425-010831c97883" containerID="bb77df88cbd714df1ee53e9b1d355d44cd68a3c737c61d5625afb9dbaffb6271" exitCode=0
Oct 30 08:06:58.752874 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:58.752849    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f5ceca-q52n4" event={"ID":"a4de1ddb-4b85-416d-9425-010831c97883","Type":"ContainerDied","Data":"bb77df88cbd714df1ee53e9b1d355d44cd68a3c737c61d5625afb9dbaffb6271"}
Oct 30 08:06:59.812681 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:59.812641    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-b5998f-7bff7846d-254sj" podUID="1ccf1d22-7e51-4883-86b0-3309d4ffc899" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:06:59.937846 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:59.937817    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f5ceca-q52n4"
Oct 30 08:06:59.979667 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:59.979627    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4de1ddb-4b85-416d-9425-010831c97883-kube-api-access-8jctk\" (UniqueName: \"kubernetes.io/projected/a4de1ddb-4b85-416d-9425-010831c97883-kube-api-access-8jctk\") pod \"a4de1ddb-4b85-416d-9425-010831c97883\" (UID: \"a4de1ddb-4b85-416d-9425-010831c97883\") "
Oct 30 08:06:59.979667 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:59.979670    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4de1ddb-4b85-416d-9425-010831c97883-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4de1ddb-4b85-416d-9425-010831c97883-maas-api-openshift-ca\") pod \"a4de1ddb-4b85-416d-9425-010831c97883\" (UID: \"a4de1ddb-4b85-416d-9425-010831c97883\") "
Oct 30 08:06:59.979933 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:59.979722    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4de1ddb-4b85-416d-9425-010831c97883-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4de1ddb-4b85-416d-9425-010831c97883-maas-api-xks-ca\") pod \"a4de1ddb-4b85-416d-9425-010831c97883\" (UID: \"a4de1ddb-4b85-416d-9425-010831c97883\") "
Oct 30 08:06:59.980200 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:59.980169    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4de1ddb-4b85-416d-9425-010831c97883-maas-api-openshift-ca" pod "a4de1ddb-4b85-416d-9425-010831c97883" (UID: "a4de1ddb-4b85-416d-9425-010831c97883"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:06:59.982074 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:59.982038    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4de1ddb-4b85-416d-9425-010831c97883-maas-api-xks-ca" pod "a4de1ddb-4b85-416d-9425-010831c97883" (UID: "a4de1ddb-4b85-416d-9425-010831c97883"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:06:59.982271 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:06:59.982236    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4de1ddb-4b85-416d-9425-010831c97883-kube-api-access-8jctk" pod "a4de1ddb-4b85-416d-9425-010831c97883" (UID: "a4de1ddb-4b85-416d-9425-010831c97883"). InnerVolumeSpecName "kube-api-access-8jctk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:07:00.048023 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:00.047992    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-b5998f-7bff7846d-254sj"
Oct 30 08:07:00.080290 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:00.080259    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ccf1d22-7e51-4883-86b0-3309d4ffc899-kube-api-access-5khl8\" (UniqueName: \"kubernetes.io/projected/1ccf1d22-7e51-4883-86b0-3309d4ffc899-kube-api-access-5khl8\") pod \"1ccf1d22-7e51-4883-86b0-3309d4ffc899\" (UID: \"1ccf1d22-7e51-4883-86b0-3309d4ffc899\") "
Oct 30 08:07:00.080474 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:00.080313    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ccf1d22-7e51-4883-86b0-3309d4ffc899-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ccf1d22-7e51-4883-86b0-3309d4ffc899-plugins-config-volume\") pod \"1ccf1d22-7e51-4883-86b0-3309d4ffc899\" (UID: \"1ccf1d22-7e51-4883-86b0-3309d4ffc899\") "
Oct 30 08:07:00.080548 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:00.080531    6224 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4de1ddb-4b85-416d-9425-010831c97883-maas-api-xks-ca\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:07:00.080548 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:00.080543    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jctk\" (UniqueName: \"kubernetes.io/projected/a4de1ddb-4b85-416d-9425-010831c97883-kube-api-access-8jctk\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:07:00.080669 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:00.080555    6224 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4de1ddb-4b85-416d-9425-010831c97883-maas-api-openshift-ca\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:07:00.080726 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:00.080706    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ccf1d22-7e51-4883-86b0-3309d4ffc899-plugins-config-volume" pod "1ccf1d22-7e51-4883-86b0-3309d4ffc899" (UID: "1ccf1d22-7e51-4883-86b0-3309d4ffc899"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:07:00.082441 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:00.082420    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ccf1d22-7e51-4883-86b0-3309d4ffc899-kube-api-access-5khl8" pod "1ccf1d22-7e51-4883-86b0-3309d4ffc899" (UID: "1ccf1d22-7e51-4883-86b0-3309d4ffc899"). InnerVolumeSpecName "kube-api-access-5khl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:07:00.181524 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:00.181486    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5khl8\" (UniqueName: \"kubernetes.io/projected/1ccf1d22-7e51-4883-86b0-3309d4ffc899-kube-api-access-5khl8\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:07:00.181695 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:00.181528    6224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ccf1d22-7e51-4883-86b0-3309d4ffc899-plugins-config-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:07:00.772384 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:00.772349    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f5ceca-q52n4"
Oct 30 08:07:00.772604 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:00.772386    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f5ceca-q52n4" event={"ID":"a4de1ddb-4b85-416d-9425-010831c97883","Type":"ContainerDied","Data":"1ffdf645bbde3ca0942f47238e6e2a24d9833daa6c8ef698f4191093452906ff"}
Oct 30 08:07:00.772604 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:00.772467    6224 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ffdf645bbde3ca0942f47238e6e2a24d9833daa6c8ef698f4191093452906ff"
Oct 30 08:07:00.773795 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:00.773772    6224 generic.go:356] "Generic (PLEG): container finished" podID="1ccf1d22-7e51-4883-86b0-3309d4ffc899" containerID="dcfacb6809f42e60f210e518aa6646b95cc467799fec9e0fa5145d4523d52981" exitCode=0
Oct 30 08:07:00.773892 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:00.773850    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-b5998f-7bff7846d-254sj" event={"ID":"1ccf1d22-7e51-4883-86b0-3309d4ffc899","Type":"ContainerDied","Data":"dcfacb6809f42e60f210e518aa6646b95cc467799fec9e0fa5145d4523d52981"}
Oct 30 08:07:00.773946 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:00.773902    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-b5998f-7bff7846d-254sj" event={"ID":"1ccf1d22-7e51-4883-86b0-3309d4ffc899","Type":"ContainerDied","Data":"5af74ec41d8fbad291d4d1f99c99f0d9b62770060afeacd108ba5d3c13015fb7"}
Oct 30 08:07:00.773946 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:00.773923    6224 scope.go:122] "RemoveContainer" containerID="dcfacb6809f42e60f210e518aa6646b95cc467799fec9e0fa5145d4523d52981"
Oct 30 08:07:00.773946 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:00.773863    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-b5998f-7bff7846d-254sj"
Oct 30 08:07:00.783508 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:00.783486    6224 scope.go:122] "RemoveContainer" containerID="dcfacb6809f42e60f210e518aa6646b95cc467799fec9e0fa5145d4523d52981"
Oct 30 08:07:00.783783 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:07:00.783760    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcfacb6809f42e60f210e518aa6646b95cc467799fec9e0fa5145d4523d52981\": container with ID starting with dcfacb6809f42e60f210e518aa6646b95cc467799fec9e0fa5145d4523d52981 not found: ID does not exist" containerID="dcfacb6809f42e60f210e518aa6646b95cc467799fec9e0fa5145d4523d52981"
Oct 30 08:07:00.783888 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:00.783789    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcfacb6809f42e60f210e518aa6646b95cc467799fec9e0fa5145d4523d52981"} err="failed to get container status \"dcfacb6809f42e60f210e518aa6646b95cc467799fec9e0fa5145d4523d52981\": rpc error: code = NotFound desc = could not find container \"dcfacb6809f42e60f210e518aa6646b95cc467799fec9e0fa5145d4523d52981\": container with ID starting with dcfacb6809f42e60f210e518aa6646b95cc467799fec9e0fa5145d4523d52981 not found: ID does not exist"
Oct 30 08:07:00.849570 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:00.849529    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-b5998f-7bff7846d-254sj"]
Oct 30 08:07:00.864637 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:00.864604    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-b5998f-7bff7846d-254sj"]
Oct 30 08:07:01.502127 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:01.502092    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ccf1d22-7e51-4883-86b0-3309d4ffc899" path="/var/lib/kubelet/pods/1ccf1d22-7e51-4883-86b0-3309d4ffc899/volumes"
Oct 30 08:07:01.855872 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:07:01.855833    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 08:07:02.769790 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:02.769751    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 08:07:07.502098 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:07.502067    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-f5ceca-5f57ff8b46-28rhc"]
Oct 30 08:07:08.059371 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:08.059340    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-914f19bf-csp6q"]
Oct 30 08:07:08.059727 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:08.059714    6224 cpu_manager.go:414] "containerMap: removing container" podUID="a4de1ddb-4b85-416d-9425-010831c97883" containerName="revoke-keys"
Oct 30 08:07:08.059774 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:08.059729    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4de1ddb-4b85-416d-9425-010831c97883" containerName="revoke-keys"
Oct 30 08:07:08.059774 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:08.059741    6224 cpu_manager.go:414] "containerMap: removing container" podUID="1ccf1d22-7e51-4883-86b0-3309d4ffc899" containerName="payload-processing"
Oct 30 08:07:08.059774 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:08.059747    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ccf1d22-7e51-4883-86b0-3309d4ffc899" containerName="payload-processing"
Oct 30 08:07:08.059910 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:08.059842    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ccf1d22-7e51-4883-86b0-3309d4ffc899" containerName="payload-processing"
Oct 30 08:07:08.059910 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:08.059859    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4de1ddb-4b85-416d-9425-010831c97883" containerName="revoke-keys"
Oct 30 08:07:08.064137 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:08.064117    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-914f19bf-csp6q"
Oct 30 08:07:08.082899 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:08.082870    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-914f19bf-csp6q"]
Oct 30 08:07:08.113387 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:08.113355    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 08:07:08.113546 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:08.113432    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b9pb7\""
Oct 30 08:07:08.156248 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:08.156199    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5be612e-951b-4688-895c-d226c6ef7b22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-914f19bf-csp6q\" (UID: \"f5be612e-951b-4688-895c-d226c6ef7b22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-914f19bf-csp6q"
Oct 30 08:07:08.156430 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:08.156331    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8t4d\" (UniqueName: \"kubernetes.io/projected/f5be612e-951b-4688-895c-d226c6ef7b22-kube-api-access-c8t4d\") pod \"maas-api-revoke-sub-914f19bf-csp6q\" (UID: \"f5be612e-951b-4688-895c-d226c6ef7b22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-914f19bf-csp6q"
Oct 30 08:07:08.156430 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:08.156369    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5be612e-951b-4688-895c-d226c6ef7b22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-914f19bf-csp6q\" (UID: \"f5be612e-951b-4688-895c-d226c6ef7b22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-914f19bf-csp6q"
Oct 30 08:07:08.257936 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:08.257893    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8t4d\" (UniqueName: \"kubernetes.io/projected/f5be612e-951b-4688-895c-d226c6ef7b22-kube-api-access-c8t4d\") pod \"maas-api-revoke-sub-914f19bf-csp6q\" (UID: \"f5be612e-951b-4688-895c-d226c6ef7b22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-914f19bf-csp6q"
Oct 30 08:07:08.258152 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:08.257949    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5be612e-951b-4688-895c-d226c6ef7b22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-914f19bf-csp6q\" (UID: \"f5be612e-951b-4688-895c-d226c6ef7b22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-914f19bf-csp6q"
Oct 30 08:07:08.258152 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:08.258042    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5be612e-951b-4688-895c-d226c6ef7b22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-914f19bf-csp6q\" (UID: \"f5be612e-951b-4688-895c-d226c6ef7b22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-914f19bf-csp6q"
Oct 30 08:07:08.258757 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:08.258731    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5be612e-951b-4688-895c-d226c6ef7b22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-914f19bf-csp6q\" (UID: \"f5be612e-951b-4688-895c-d226c6ef7b22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-914f19bf-csp6q"
Oct 30 08:07:08.260394 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:08.260373    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5be612e-951b-4688-895c-d226c6ef7b22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-914f19bf-csp6q\" (UID: \"f5be612e-951b-4688-895c-d226c6ef7b22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-914f19bf-csp6q"
Oct 30 08:07:08.334517 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:08.334432    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8t4d\" (UniqueName: \"kubernetes.io/projected/f5be612e-951b-4688-895c-d226c6ef7b22-kube-api-access-c8t4d\") pod \"maas-api-revoke-sub-914f19bf-csp6q\" (UID: \"f5be612e-951b-4688-895c-d226c6ef7b22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-914f19bf-csp6q"
Oct 30 08:07:08.376638 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:08.376603    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-914f19bf-csp6q"
Oct 30 08:07:08.684183 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:08.684148    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-914f19bf-csp6q"]
Oct 30 08:07:08.684572 ip-10-0-128-125 kubenswrapper[6224]: W1030 08:07:08.684488    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5be612e_951b_4688_895c_d226c6ef7b22.slice/crio-601199f876acc41989f2bf516d651bd8396425cb49a7275d5f3f897a7e76f99f WatchSource:0}: Error finding container 601199f876acc41989f2bf516d651bd8396425cb49a7275d5f3f897a7e76f99f: Status 404 returned error can't find the container with id 601199f876acc41989f2bf516d651bd8396425cb49a7275d5f3f897a7e76f99f
Oct 30 08:07:08.819099 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:08.819059    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-914f19bf-csp6q" event={"ID":"f5be612e-951b-4688-895c-d226c6ef7b22","Type":"ContainerStarted","Data":"601199f876acc41989f2bf516d651bd8396425cb49a7275d5f3f897a7e76f99f"}
Oct 30 08:07:09.827155 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:09.827114    6224 generic.go:356] "Generic (PLEG): container finished" podID="f5be612e-951b-4688-895c-d226c6ef7b22" containerID="a15f1f99ac8f2eefc44ca03ae46c143c00d1b8ff92d2e27a73e05d28de861d8d" exitCode=0
Oct 30 08:07:09.827580 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:09.827186    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-914f19bf-csp6q" event={"ID":"f5be612e-951b-4688-895c-d226c6ef7b22","Type":"ContainerDied","Data":"a15f1f99ac8f2eefc44ca03ae46c143c00d1b8ff92d2e27a73e05d28de861d8d"}
Oct 30 08:07:10.968795 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:10.968762    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-914f19bf-csp6q"
Oct 30 08:07:11.085417 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:11.085336    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5be612e-951b-4688-895c-d226c6ef7b22-kube-api-access-c8t4d\" (UniqueName: \"kubernetes.io/projected/f5be612e-951b-4688-895c-d226c6ef7b22-kube-api-access-c8t4d\") pod \"f5be612e-951b-4688-895c-d226c6ef7b22\" (UID: \"f5be612e-951b-4688-895c-d226c6ef7b22\") "
Oct 30 08:07:11.085417 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:11.085373    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5be612e-951b-4688-895c-d226c6ef7b22-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5be612e-951b-4688-895c-d226c6ef7b22-maas-api-openshift-ca\") pod \"f5be612e-951b-4688-895c-d226c6ef7b22\" (UID: \"f5be612e-951b-4688-895c-d226c6ef7b22\") "
Oct 30 08:07:11.085605 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:11.085432    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5be612e-951b-4688-895c-d226c6ef7b22-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5be612e-951b-4688-895c-d226c6ef7b22-maas-api-xks-ca\") pod \"f5be612e-951b-4688-895c-d226c6ef7b22\" (UID: \"f5be612e-951b-4688-895c-d226c6ef7b22\") "
Oct 30 08:07:11.085841 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:11.085801    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5be612e-951b-4688-895c-d226c6ef7b22-maas-api-openshift-ca" pod "f5be612e-951b-4688-895c-d226c6ef7b22" (UID: "f5be612e-951b-4688-895c-d226c6ef7b22"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:07:11.087367 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:11.087345    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5be612e-951b-4688-895c-d226c6ef7b22-kube-api-access-c8t4d" pod "f5be612e-951b-4688-895c-d226c6ef7b22" (UID: "f5be612e-951b-4688-895c-d226c6ef7b22"). InnerVolumeSpecName "kube-api-access-c8t4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:07:11.087456 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:11.087399    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5be612e-951b-4688-895c-d226c6ef7b22-maas-api-xks-ca" pod "f5be612e-951b-4688-895c-d226c6ef7b22" (UID: "f5be612e-951b-4688-895c-d226c6ef7b22"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:07:11.187054 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:11.187014    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8t4d\" (UniqueName: \"kubernetes.io/projected/f5be612e-951b-4688-895c-d226c6ef7b22-kube-api-access-c8t4d\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:07:11.187054 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:11.187047    6224 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5be612e-951b-4688-895c-d226c6ef7b22-maas-api-openshift-ca\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:07:11.187054 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:11.187060    6224 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5be612e-951b-4688-895c-d226c6ef7b22-maas-api-xks-ca\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:07:11.593068 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:11.593024    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-06b26e/distinct-w0-06b26e-kserve-545cf57898-2fhhd" podUID="71294bfa-ac2c-40b7-b889-0527daa07f63" containerName="main" containerID="cri-o://baca22d2e895843d07e8828e9c7a0a2643a15d069661c99636a1e08a4e6374d2" gracePeriod=45
Oct 30 08:07:11.609477 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:11.609439    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-06b26e/premium-w0-06b26e-kserve-c9dbd9879-b4jlk" podUID="1230dcae-e1cd-460d-86fc-bd40c8427b5f" containerName="main" containerID="cri-o://8ce4769417f2f3ff210967dee957c14f7b29519f85b0cf88f3df971dbb0aaaeb" gracePeriod=45
Oct 30 08:07:11.631793 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:11.631755    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-06b26e/premium-w0-06b26e-kserve-c9dbd9879-b4jlk" podUID="1230dcae-e1cd-460d-86fc-bd40c8427b5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": dial tcp 10.134.0.49:8000: connect: connection refused"
Oct 30 08:07:11.838949 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:11.838919    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-06b26e/distinct-w0-06b26e-kserve-545cf57898-2fhhd"
Oct 30 08:07:11.841009 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:11.840980    6224 generic.go:356] "Generic (PLEG): container finished" podID="71294bfa-ac2c-40b7-b889-0527daa07f63" containerID="baca22d2e895843d07e8828e9c7a0a2643a15d069661c99636a1e08a4e6374d2" exitCode=0
Oct 30 08:07:11.841166 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:11.841092    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-06b26e/distinct-w0-06b26e-kserve-545cf57898-2fhhd"
Oct 30 08:07:11.841211 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:11.841198    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-06b26e/distinct-w0-06b26e-kserve-545cf57898-2fhhd" event={"ID":"71294bfa-ac2c-40b7-b889-0527daa07f63","Type":"ContainerDied","Data":"baca22d2e895843d07e8828e9c7a0a2643a15d069661c99636a1e08a4e6374d2"}
Oct 30 08:07:11.841255 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:11.841226    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-06b26e/distinct-w0-06b26e-kserve-545cf57898-2fhhd" event={"ID":"71294bfa-ac2c-40b7-b889-0527daa07f63","Type":"ContainerDied","Data":"228559ab6cb1ca0c3a65955f2efeda3ab0ba508cb9792f9f38ef0ebe23c8804f"}
Oct 30 08:07:11.841255 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:11.841242    6224 scope.go:122] "RemoveContainer" containerID="baca22d2e895843d07e8828e9c7a0a2643a15d069661c99636a1e08a4e6374d2"
Oct 30 08:07:11.842873 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:11.842855    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-914f19bf-csp6q"
Oct 30 08:07:11.843003 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:11.842896    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-914f19bf-csp6q" event={"ID":"f5be612e-951b-4688-895c-d226c6ef7b22","Type":"ContainerDied","Data":"601199f876acc41989f2bf516d651bd8396425cb49a7275d5f3f897a7e76f99f"}
Oct 30 08:07:11.843003 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:11.842945    6224 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="601199f876acc41989f2bf516d651bd8396425cb49a7275d5f3f897a7e76f99f"
Oct 30 08:07:11.844853 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:11.844733    6224 generic.go:356] "Generic (PLEG): container finished" podID="1230dcae-e1cd-460d-86fc-bd40c8427b5f" containerID="8ce4769417f2f3ff210967dee957c14f7b29519f85b0cf88f3df971dbb0aaaeb" exitCode=0
Oct 30 08:07:11.844853 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:11.844776    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-06b26e/premium-w0-06b26e-kserve-c9dbd9879-b4jlk" event={"ID":"1230dcae-e1cd-460d-86fc-bd40c8427b5f","Type":"ContainerDied","Data":"8ce4769417f2f3ff210967dee957c14f7b29519f85b0cf88f3df971dbb0aaaeb"}
Oct 30 08:07:11.850565 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:11.850545    6224 scope.go:122] "RemoveContainer" containerID="baca22d2e895843d07e8828e9c7a0a2643a15d069661c99636a1e08a4e6374d2"
Oct 30 08:07:11.850871 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:07:11.850847    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baca22d2e895843d07e8828e9c7a0a2643a15d069661c99636a1e08a4e6374d2\": container with ID starting with baca22d2e895843d07e8828e9c7a0a2643a15d069661c99636a1e08a4e6374d2 not found: ID does not exist" containerID="baca22d2e895843d07e8828e9c7a0a2643a15d069661c99636a1e08a4e6374d2"
Oct 30 08:07:11.850971 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:11.850880    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baca22d2e895843d07e8828e9c7a0a2643a15d069661c99636a1e08a4e6374d2"} err="failed to get container status \"baca22d2e895843d07e8828e9c7a0a2643a15d069661c99636a1e08a4e6374d2\": rpc error: code = NotFound desc = could not find container \"baca22d2e895843d07e8828e9c7a0a2643a15d069661c99636a1e08a4e6374d2\": container with ID starting with baca22d2e895843d07e8828e9c7a0a2643a15d069661c99636a1e08a4e6374d2 not found: ID does not exist"
Oct 30 08:07:11.874317 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:11.874293    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-06b26e/premium-w0-06b26e-kserve-c9dbd9879-b4jlk"
Oct 30 08:07:11.894888 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:11.894860    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71294bfa-ac2c-40b7-b889-0527daa07f63-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71294bfa-ac2c-40b7-b889-0527daa07f63-model-cache\") pod \"71294bfa-ac2c-40b7-b889-0527daa07f63\" (UID: \"71294bfa-ac2c-40b7-b889-0527daa07f63\") "
Oct 30 08:07:11.895071 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:11.894951    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71294bfa-ac2c-40b7-b889-0527daa07f63-home\" (UniqueName: \"kubernetes.io/empty-dir/71294bfa-ac2c-40b7-b889-0527daa07f63-home\") pod \"71294bfa-ac2c-40b7-b889-0527daa07f63\" (UID: \"71294bfa-ac2c-40b7-b889-0527daa07f63\") "
Oct 30 08:07:11.895071 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:11.894984    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71294bfa-ac2c-40b7-b889-0527daa07f63-dshm\" (UniqueName: \"kubernetes.io/empty-dir/71294bfa-ac2c-40b7-b889-0527daa07f63-dshm\") pod \"71294bfa-ac2c-40b7-b889-0527daa07f63\" (UID: \"71294bfa-ac2c-40b7-b889-0527daa07f63\") "
Oct 30 08:07:11.895201 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:11.895083    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71294bfa-ac2c-40b7-b889-0527daa07f63-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71294bfa-ac2c-40b7-b889-0527daa07f63-tmp-dir\") pod \"71294bfa-ac2c-40b7-b889-0527daa07f63\" (UID: \"71294bfa-ac2c-40b7-b889-0527daa07f63\") "
Oct 30 08:07:11.895201 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:11.895116    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71294bfa-ac2c-40b7-b889-0527daa07f63-tls-certs\" (UniqueName: \"kubernetes.io/secret/71294bfa-ac2c-40b7-b889-0527daa07f63-tls-certs\") pod \"71294bfa-ac2c-40b7-b889-0527daa07f63\" (UID: \"71294bfa-ac2c-40b7-b889-0527daa07f63\") "
Oct 30 08:07:11.895201 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:11.895165    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71294bfa-ac2c-40b7-b889-0527daa07f63-kube-api-access-dscrd\" (UniqueName: \"kubernetes.io/projected/71294bfa-ac2c-40b7-b889-0527daa07f63-kube-api-access-dscrd\") pod \"71294bfa-ac2c-40b7-b889-0527daa07f63\" (UID: \"71294bfa-ac2c-40b7-b889-0527daa07f63\") "
Oct 30 08:07:11.895844 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:11.895426    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71294bfa-ac2c-40b7-b889-0527daa07f63-model-cache" pod "71294bfa-ac2c-40b7-b889-0527daa07f63" (UID: "71294bfa-ac2c-40b7-b889-0527daa07f63"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:07:11.895844 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:11.895512    6224 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71294bfa-ac2c-40b7-b889-0527daa07f63-model-cache\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:07:11.895844 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:11.895577    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71294bfa-ac2c-40b7-b889-0527daa07f63-home" pod "71294bfa-ac2c-40b7-b889-0527daa07f63" (UID: "71294bfa-ac2c-40b7-b889-0527daa07f63"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:07:11.895844 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:11.895677    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71294bfa-ac2c-40b7-b889-0527daa07f63-tmp-dir" pod "71294bfa-ac2c-40b7-b889-0527daa07f63" (UID: "71294bfa-ac2c-40b7-b889-0527daa07f63"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:07:11.898401 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:11.898374    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71294bfa-ac2c-40b7-b889-0527daa07f63-tls-certs" pod "71294bfa-ac2c-40b7-b889-0527daa07f63" (UID: "71294bfa-ac2c-40b7-b889-0527daa07f63"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:07:11.898508 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:11.898481    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71294bfa-ac2c-40b7-b889-0527daa07f63-kube-api-access-dscrd" pod "71294bfa-ac2c-40b7-b889-0527daa07f63" (UID: "71294bfa-ac2c-40b7-b889-0527daa07f63"). InnerVolumeSpecName "kube-api-access-dscrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:07:11.900393 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:11.900370    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71294bfa-ac2c-40b7-b889-0527daa07f63-dshm" pod "71294bfa-ac2c-40b7-b889-0527daa07f63" (UID: "71294bfa-ac2c-40b7-b889-0527daa07f63"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:07:11.996386 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:11.996344    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1230dcae-e1cd-460d-86fc-bd40c8427b5f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1230dcae-e1cd-460d-86fc-bd40c8427b5f-model-cache\") pod \"1230dcae-e1cd-460d-86fc-bd40c8427b5f\" (UID: \"1230dcae-e1cd-460d-86fc-bd40c8427b5f\") "
Oct 30 08:07:11.996903 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:11.996428    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1230dcae-e1cd-460d-86fc-bd40c8427b5f-tls-certs\" (UniqueName: \"kubernetes.io/secret/1230dcae-e1cd-460d-86fc-bd40c8427b5f-tls-certs\") pod \"1230dcae-e1cd-460d-86fc-bd40c8427b5f\" (UID: \"1230dcae-e1cd-460d-86fc-bd40c8427b5f\") "
Oct 30 08:07:11.996903 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:11.996462    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1230dcae-e1cd-460d-86fc-bd40c8427b5f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1230dcae-e1cd-460d-86fc-bd40c8427b5f-dshm\") pod \"1230dcae-e1cd-460d-86fc-bd40c8427b5f\" (UID: \"1230dcae-e1cd-460d-86fc-bd40c8427b5f\") "
Oct 30 08:07:11.996903 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:11.996561    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1230dcae-e1cd-460d-86fc-bd40c8427b5f-home\" (UniqueName: \"kubernetes.io/empty-dir/1230dcae-e1cd-460d-86fc-bd40c8427b5f-home\") pod \"1230dcae-e1cd-460d-86fc-bd40c8427b5f\" (UID: \"1230dcae-e1cd-460d-86fc-bd40c8427b5f\") "
Oct 30 08:07:11.996903 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:11.996599    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1230dcae-e1cd-460d-86fc-bd40c8427b5f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1230dcae-e1cd-460d-86fc-bd40c8427b5f-tmp-dir\") pod \"1230dcae-e1cd-460d-86fc-bd40c8427b5f\" (UID: \"1230dcae-e1cd-460d-86fc-bd40c8427b5f\") "
Oct 30 08:07:11.996903 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:11.996639    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1230dcae-e1cd-460d-86fc-bd40c8427b5f-kube-api-access-szbqn\" (UniqueName: \"kubernetes.io/projected/1230dcae-e1cd-460d-86fc-bd40c8427b5f-kube-api-access-szbqn\") pod \"1230dcae-e1cd-460d-86fc-bd40c8427b5f\" (UID: \"1230dcae-e1cd-460d-86fc-bd40c8427b5f\") "
Oct 30 08:07:11.996903 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:11.996744    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1230dcae-e1cd-460d-86fc-bd40c8427b5f-model-cache" pod "1230dcae-e1cd-460d-86fc-bd40c8427b5f" (UID: "1230dcae-e1cd-460d-86fc-bd40c8427b5f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:07:11.997280 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:11.996987    6224 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71294bfa-ac2c-40b7-b889-0527daa07f63-home\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:07:11.997280 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:11.997009    6224 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71294bfa-ac2c-40b7-b889-0527daa07f63-dshm\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:07:11.997280 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:11.997025    6224 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71294bfa-ac2c-40b7-b889-0527daa07f63-tmp-dir\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:07:11.997280 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:11.997040    6224 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71294bfa-ac2c-40b7-b889-0527daa07f63-tls-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:07:11.997280 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:11.997054    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dscrd\" (UniqueName: \"kubernetes.io/projected/71294bfa-ac2c-40b7-b889-0527daa07f63-kube-api-access-dscrd\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:07:11.997280 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:11.997068    6224 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1230dcae-e1cd-460d-86fc-bd40c8427b5f-model-cache\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:07:11.997606 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:11.997377    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1230dcae-e1cd-460d-86fc-bd40c8427b5f-home" pod "1230dcae-e1cd-460d-86fc-bd40c8427b5f" (UID: "1230dcae-e1cd-460d-86fc-bd40c8427b5f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:07:11.997675 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:11.997631    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1230dcae-e1cd-460d-86fc-bd40c8427b5f-tmp-dir" pod "1230dcae-e1cd-460d-86fc-bd40c8427b5f" (UID: "1230dcae-e1cd-460d-86fc-bd40c8427b5f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:07:11.999459 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:11.999429    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1230dcae-e1cd-460d-86fc-bd40c8427b5f-dshm" pod "1230dcae-e1cd-460d-86fc-bd40c8427b5f" (UID: "1230dcae-e1cd-460d-86fc-bd40c8427b5f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:07:11.999577 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:11.999499    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1230dcae-e1cd-460d-86fc-bd40c8427b5f-tls-certs" pod "1230dcae-e1cd-460d-86fc-bd40c8427b5f" (UID: "1230dcae-e1cd-460d-86fc-bd40c8427b5f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:07:11.999714 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:11.999699    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1230dcae-e1cd-460d-86fc-bd40c8427b5f-kube-api-access-szbqn" pod "1230dcae-e1cd-460d-86fc-bd40c8427b5f" (UID: "1230dcae-e1cd-460d-86fc-bd40c8427b5f"). InnerVolumeSpecName "kube-api-access-szbqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:07:12.098433 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:12.098392    6224 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1230dcae-e1cd-460d-86fc-bd40c8427b5f-tmp-dir\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:07:12.098433 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:12.098432    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szbqn\" (UniqueName: \"kubernetes.io/projected/1230dcae-e1cd-460d-86fc-bd40c8427b5f-kube-api-access-szbqn\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:07:12.098680 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:12.098447    6224 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1230dcae-e1cd-460d-86fc-bd40c8427b5f-tls-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:07:12.098680 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:12.098471    6224 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1230dcae-e1cd-460d-86fc-bd40c8427b5f-dshm\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:07:12.098680 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:12.098484    6224 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1230dcae-e1cd-460d-86fc-bd40c8427b5f-home\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:07:12.190930 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:12.190889    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-06b26e/distinct-w0-06b26e-kserve-545cf57898-2fhhd"]
Oct 30 08:07:12.195331 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:12.195303    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-06b26e/distinct-w0-06b26e-kserve-545cf57898-2fhhd"]
Oct 30 08:07:12.503141 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:12.503040    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f5ceca-5f57ff8b46-28rhc" podUID="671ab50b-8475-463c-8815-767e495feaa9" containerName="payload-processing" containerID="cri-o://f7fa1c951f077cf238ea5cadcae2815dddf74de59b376504e87d545da522246e" gracePeriod=25
Oct 30 08:07:12.853104 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:12.853066    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-06b26e/premium-w0-06b26e-kserve-c9dbd9879-b4jlk" event={"ID":"1230dcae-e1cd-460d-86fc-bd40c8427b5f","Type":"ContainerDied","Data":"fac0494352b179db837f55dfce53f4b9fa52a542c7b88aed208003e081bc6c29"}
Oct 30 08:07:12.853321 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:12.853114    6224 scope.go:122] "RemoveContainer" containerID="8ce4769417f2f3ff210967dee957c14f7b29519f85b0cf88f3df971dbb0aaaeb"
Oct 30 08:07:12.853321 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:12.853077    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-06b26e/premium-w0-06b26e-kserve-c9dbd9879-b4jlk"
Oct 30 08:07:12.898198 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:12.898162    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f5ceca-q52n4"]
Oct 30 08:07:12.903942 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:12.903912    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f5ceca-q52n4"]
Oct 30 08:07:12.912981 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:12.912953    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-06b26e/premium-w0-06b26e-kserve-c9dbd9879-b4jlk"]
Oct 30 08:07:12.917849 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:12.917822    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-06b26e/premium-w0-06b26e-kserve-c9dbd9879-b4jlk"]
Oct 30 08:07:13.500925 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:13.500890    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1230dcae-e1cd-460d-86fc-bd40c8427b5f" path="/var/lib/kubelet/pods/1230dcae-e1cd-460d-86fc-bd40c8427b5f/volumes"
Oct 30 08:07:13.501340 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:13.501318    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71294bfa-ac2c-40b7-b889-0527daa07f63" path="/var/lib/kubelet/pods/71294bfa-ac2c-40b7-b889-0527daa07f63/volumes"
Oct 30 08:07:13.501691 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:13.501677    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4de1ddb-4b85-416d-9425-010831c97883" path="/var/lib/kubelet/pods/a4de1ddb-4b85-416d-9425-010831c97883/volumes"
Oct 30 08:07:17.530563 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:17.530522    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f5ceca-5f57ff8b46-28rhc" podUID="671ab50b-8475-463c-8815-767e495feaa9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:07:19.405512 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:19.405473    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-06b26e-6g5fv"]
Oct 30 08:07:19.420897 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:19.420857    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-06b26e-6g5fv"]
Oct 30 08:07:19.502239 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:19.502203    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f717f7f2-6cec-4152-b45a-82a04df95be1" path="/var/lib/kubelet/pods/f717f7f2-6cec-4152-b45a-82a04df95be1/volumes"
Oct 30 08:07:19.867250 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:19.867215    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 08:07:20.878680 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:20.878640    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc"]
Oct 30 08:07:20.879238 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:20.878888    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc" podUID="50bc25e6-c53f-4d16-b1fd-738cdd8c6a10" containerName="istio-proxy" containerID="cri-o://f8a5146a122aefa7f66c946baa4d92150b599ed06fd877587ef627bb17b887cc" gracePeriod=30
Oct 30 08:07:23.051912 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:23.051869    6224 kubelet.go:1689] "Image garbage collection succeeded"
Oct 30 08:07:24.397167 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:24.397118    6224 patch_prober.go:29] interesting pod/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 08:07:24.397572 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:24.397187    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc" podUID="50bc25e6-c53f-4d16-b1fd-738cdd8c6a10" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 08:07:26.128001 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:26.127973    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc"
Oct 30 08:07:26.229313 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:26.229222    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50bc25e6-c53f-4d16-b1fd-738cdd8c6a10-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50bc25e6-c53f-4d16-b1fd-738cdd8c6a10-istio-data\") pod \"50bc25e6-c53f-4d16-b1fd-738cdd8c6a10\" (UID: \"50bc25e6-c53f-4d16-b1fd-738cdd8c6a10\") "
Oct 30 08:07:26.229313 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:26.229297    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50bc25e6-c53f-4d16-b1fd-738cdd8c6a10-kube-api-access-spknf\" (UniqueName: \"kubernetes.io/projected/50bc25e6-c53f-4d16-b1fd-738cdd8c6a10-kube-api-access-spknf\") pod \"50bc25e6-c53f-4d16-b1fd-738cdd8c6a10\" (UID: \"50bc25e6-c53f-4d16-b1fd-738cdd8c6a10\") "
Oct 30 08:07:26.229559 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:26.229327    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50bc25e6-c53f-4d16-b1fd-738cdd8c6a10-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50bc25e6-c53f-4d16-b1fd-738cdd8c6a10-credential-socket\") pod \"50bc25e6-c53f-4d16-b1fd-738cdd8c6a10\" (UID: \"50bc25e6-c53f-4d16-b1fd-738cdd8c6a10\") "
Oct 30 08:07:26.229559 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:26.229363    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50bc25e6-c53f-4d16-b1fd-738cdd8c6a10-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50bc25e6-c53f-4d16-b1fd-738cdd8c6a10-istio-envoy\") pod \"50bc25e6-c53f-4d16-b1fd-738cdd8c6a10\" (UID: \"50bc25e6-c53f-4d16-b1fd-738cdd8c6a10\") "
Oct 30 08:07:26.229559 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:26.229435    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50bc25e6-c53f-4d16-b1fd-738cdd8c6a10-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50bc25e6-c53f-4d16-b1fd-738cdd8c6a10-workload-socket\") pod \"50bc25e6-c53f-4d16-b1fd-738cdd8c6a10\" (UID: \"50bc25e6-c53f-4d16-b1fd-738cdd8c6a10\") "
Oct 30 08:07:26.229559 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:26.229477    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/50bc25e6-c53f-4d16-b1fd-738cdd8c6a10-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50bc25e6-c53f-4d16-b1fd-738cdd8c6a10-istio-podinfo\") pod \"50bc25e6-c53f-4d16-b1fd-738cdd8c6a10\" (UID: \"50bc25e6-c53f-4d16-b1fd-738cdd8c6a10\") "
Oct 30 08:07:26.229559 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:26.229514    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50bc25e6-c53f-4d16-b1fd-738cdd8c6a10-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50bc25e6-c53f-4d16-b1fd-738cdd8c6a10-workload-certs\") pod \"50bc25e6-c53f-4d16-b1fd-738cdd8c6a10\" (UID: \"50bc25e6-c53f-4d16-b1fd-738cdd8c6a10\") "
Oct 30 08:07:26.229872 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:26.229558    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50bc25e6-c53f-4d16-b1fd-738cdd8c6a10-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50bc25e6-c53f-4d16-b1fd-738cdd8c6a10-istiod-ca-cert\") pod \"50bc25e6-c53f-4d16-b1fd-738cdd8c6a10\" (UID: \"50bc25e6-c53f-4d16-b1fd-738cdd8c6a10\") "
Oct 30 08:07:26.229872 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:26.229600    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50bc25e6-c53f-4d16-b1fd-738cdd8c6a10-istio-token\" (UniqueName: \"kubernetes.io/projected/50bc25e6-c53f-4d16-b1fd-738cdd8c6a10-istio-token\") pod \"50bc25e6-c53f-4d16-b1fd-738cdd8c6a10\" (UID: \"50bc25e6-c53f-4d16-b1fd-738cdd8c6a10\") "
Oct 30 08:07:26.229872 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:26.229600    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50bc25e6-c53f-4d16-b1fd-738cdd8c6a10-credential-socket" pod "50bc25e6-c53f-4d16-b1fd-738cdd8c6a10" (UID: "50bc25e6-c53f-4d16-b1fd-738cdd8c6a10"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:07:26.229872 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:26.229800    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50bc25e6-c53f-4d16-b1fd-738cdd8c6a10-workload-socket" pod "50bc25e6-c53f-4d16-b1fd-738cdd8c6a10" (UID: "50bc25e6-c53f-4d16-b1fd-738cdd8c6a10"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:07:26.229872 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:26.229596    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50bc25e6-c53f-4d16-b1fd-738cdd8c6a10-istio-data" pod "50bc25e6-c53f-4d16-b1fd-738cdd8c6a10" (UID: "50bc25e6-c53f-4d16-b1fd-738cdd8c6a10"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:07:26.230154 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:26.229910    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50bc25e6-c53f-4d16-b1fd-738cdd8c6a10-istiod-ca-cert" pod "50bc25e6-c53f-4d16-b1fd-738cdd8c6a10" (UID: "50bc25e6-c53f-4d16-b1fd-738cdd8c6a10"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:07:26.230154 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:26.229963    6224 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50bc25e6-c53f-4d16-b1fd-738cdd8c6a10-credential-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:07:26.230154 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:26.229979    6224 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50bc25e6-c53f-4d16-b1fd-738cdd8c6a10-workload-socket\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:07:26.230154 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:26.229993    6224 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50bc25e6-c53f-4d16-b1fd-738cdd8c6a10-istio-data\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:07:26.230154 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:26.230028    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50bc25e6-c53f-4d16-b1fd-738cdd8c6a10-workload-certs" pod "50bc25e6-c53f-4d16-b1fd-738cdd8c6a10" (UID: "50bc25e6-c53f-4d16-b1fd-738cdd8c6a10"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:07:26.231698 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:26.231672    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50bc25e6-c53f-4d16-b1fd-738cdd8c6a10-istio-envoy" pod "50bc25e6-c53f-4d16-b1fd-738cdd8c6a10" (UID: "50bc25e6-c53f-4d16-b1fd-738cdd8c6a10"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:07:26.231845 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:26.231758    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50bc25e6-c53f-4d16-b1fd-738cdd8c6a10-istio-token" pod "50bc25e6-c53f-4d16-b1fd-738cdd8c6a10" (UID: "50bc25e6-c53f-4d16-b1fd-738cdd8c6a10"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:07:26.231845 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:26.231785    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50bc25e6-c53f-4d16-b1fd-738cdd8c6a10-kube-api-access-spknf" pod "50bc25e6-c53f-4d16-b1fd-738cdd8c6a10" (UID: "50bc25e6-c53f-4d16-b1fd-738cdd8c6a10"). InnerVolumeSpecName "kube-api-access-spknf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:07:26.231938 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:26.231905    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50bc25e6-c53f-4d16-b1fd-738cdd8c6a10-istio-podinfo" pod "50bc25e6-c53f-4d16-b1fd-738cdd8c6a10" (UID: "50bc25e6-c53f-4d16-b1fd-738cdd8c6a10"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 08:07:26.331395 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:26.331357    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spknf\" (UniqueName: \"kubernetes.io/projected/50bc25e6-c53f-4d16-b1fd-738cdd8c6a10-kube-api-access-spknf\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:07:26.331395 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:26.331390    6224 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50bc25e6-c53f-4d16-b1fd-738cdd8c6a10-istio-envoy\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:07:26.331395 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:26.331400    6224 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50bc25e6-c53f-4d16-b1fd-738cdd8c6a10-istio-podinfo\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:07:26.331652 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:26.331408    6224 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50bc25e6-c53f-4d16-b1fd-738cdd8c6a10-workload-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:07:26.331652 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:26.331417    6224 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50bc25e6-c53f-4d16-b1fd-738cdd8c6a10-istiod-ca-cert\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:07:26.331652 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:26.331425    6224 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50bc25e6-c53f-4d16-b1fd-738cdd8c6a10-istio-token\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:07:26.343231 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:26.343199    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2eda2c/distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw"]
Oct 30 08:07:26.368598 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:26.368571    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2eda2c/simulator-w3-2eda2c-kserve-56776b9849-9bqzm"]
Oct 30 08:07:26.947253 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:26.947210    6224 generic.go:356] "Generic (PLEG): container finished" podID="50bc25e6-c53f-4d16-b1fd-738cdd8c6a10" containerID="f8a5146a122aefa7f66c946baa4d92150b599ed06fd877587ef627bb17b887cc" exitCode=0
Oct 30 08:07:26.947465 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:26.947299    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc"
Oct 30 08:07:26.947465 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:26.947301    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc" event={"ID":"50bc25e6-c53f-4d16-b1fd-738cdd8c6a10","Type":"ContainerDied","Data":"f8a5146a122aefa7f66c946baa4d92150b599ed06fd877587ef627bb17b887cc"}
Oct 30 08:07:26.947465 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:26.947450    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc" event={"ID":"50bc25e6-c53f-4d16-b1fd-738cdd8c6a10","Type":"ContainerDied","Data":"e7414fe7f209e309195d94257281958b9dda7d1d861f1788b7eddb0870148c7c"}
Oct 30 08:07:26.947645 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:26.947481    6224 scope.go:122] "RemoveContainer" containerID="f8a5146a122aefa7f66c946baa4d92150b599ed06fd877587ef627bb17b887cc"
Oct 30 08:07:26.958675 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:26.958650    6224 scope.go:122] "RemoveContainer" containerID="30703ecef0c1c81dea8fc3b3d293f83ed29e7952b657931054ebd57c0632aeb3"
Oct 30 08:07:26.967513 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:26.967494    6224 scope.go:122] "RemoveContainer" containerID="f8a5146a122aefa7f66c946baa4d92150b599ed06fd877587ef627bb17b887cc"
Oct 30 08:07:26.967829 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:07:26.967788    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8a5146a122aefa7f66c946baa4d92150b599ed06fd877587ef627bb17b887cc\": container with ID starting with f8a5146a122aefa7f66c946baa4d92150b599ed06fd877587ef627bb17b887cc not found: ID does not exist" containerID="f8a5146a122aefa7f66c946baa4d92150b599ed06fd877587ef627bb17b887cc"
Oct 30 08:07:26.967880 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:26.967841    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8a5146a122aefa7f66c946baa4d92150b599ed06fd877587ef627bb17b887cc"} err="failed to get container status \"f8a5146a122aefa7f66c946baa4d92150b599ed06fd877587ef627bb17b887cc\": rpc error: code = NotFound desc = could not find container \"f8a5146a122aefa7f66c946baa4d92150b599ed06fd877587ef627bb17b887cc\": container with ID starting with f8a5146a122aefa7f66c946baa4d92150b599ed06fd877587ef627bb17b887cc not found: ID does not exist"
Oct 30 08:07:26.967880 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:26.967861    6224 scope.go:122] "RemoveContainer" containerID="30703ecef0c1c81dea8fc3b3d293f83ed29e7952b657931054ebd57c0632aeb3"
Oct 30 08:07:26.968086 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:07:26.968067    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30703ecef0c1c81dea8fc3b3d293f83ed29e7952b657931054ebd57c0632aeb3\": container with ID starting with 30703ecef0c1c81dea8fc3b3d293f83ed29e7952b657931054ebd57c0632aeb3 not found: ID does not exist" containerID="30703ecef0c1c81dea8fc3b3d293f83ed29e7952b657931054ebd57c0632aeb3"
Oct 30 08:07:26.968135 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:26.968092    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30703ecef0c1c81dea8fc3b3d293f83ed29e7952b657931054ebd57c0632aeb3"} err="failed to get container status \"30703ecef0c1c81dea8fc3b3d293f83ed29e7952b657931054ebd57c0632aeb3\": rpc error: code = NotFound desc = could not find container \"30703ecef0c1c81dea8fc3b3d293f83ed29e7952b657931054ebd57c0632aeb3\": container with ID starting with 30703ecef0c1c81dea8fc3b3d293f83ed29e7952b657931054ebd57c0632aeb3 not found: ID does not exist"
Oct 30 08:07:27.024658 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:27.024619    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc"]
Oct 30 08:07:27.033823 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:27.033774    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f5ceca-openshift-default-55bc4cc78f-gz7kc"]
Oct 30 08:07:27.503004 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:27.502966    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50bc25e6-c53f-4d16-b1fd-738cdd8c6a10" path="/var/lib/kubelet/pods/50bc25e6-c53f-4d16-b1fd-738cdd8c6a10/volumes"
Oct 30 08:07:27.531312 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:27.531267    6224 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f5ceca-5f57ff8b46-28rhc" podUID="671ab50b-8475-463c-8815-767e495feaa9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 08:07:31.844086 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:31.844052    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 08:07:31.857315 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:07:31.857280    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 08:07:32.755547 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:32.755521    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f5ceca-5f57ff8b46-28rhc"
Oct 30 08:07:32.891178 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:32.891141    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/671ab50b-8475-463c-8815-767e495feaa9-kube-api-access-x2hwj\" (UniqueName: \"kubernetes.io/projected/671ab50b-8475-463c-8815-767e495feaa9-kube-api-access-x2hwj\") pod \"671ab50b-8475-463c-8815-767e495feaa9\" (UID: \"671ab50b-8475-463c-8815-767e495feaa9\") "
Oct 30 08:07:32.891644 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:32.891204    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/671ab50b-8475-463c-8815-767e495feaa9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/671ab50b-8475-463c-8815-767e495feaa9-plugins-config-volume\") pod \"671ab50b-8475-463c-8815-767e495feaa9\" (UID: \"671ab50b-8475-463c-8815-767e495feaa9\") "
Oct 30 08:07:32.891644 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:32.891582    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/671ab50b-8475-463c-8815-767e495feaa9-plugins-config-volume" pod "671ab50b-8475-463c-8815-767e495feaa9" (UID: "671ab50b-8475-463c-8815-767e495feaa9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 08:07:32.893332 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:32.893309    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/671ab50b-8475-463c-8815-767e495feaa9-kube-api-access-x2hwj" pod "671ab50b-8475-463c-8815-767e495feaa9" (UID: "671ab50b-8475-463c-8815-767e495feaa9"). InnerVolumeSpecName "kube-api-access-x2hwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:07:32.979325 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:32.979288    6224 generic.go:356] "Generic (PLEG): container finished" podID="671ab50b-8475-463c-8815-767e495feaa9" containerID="f7fa1c951f077cf238ea5cadcae2815dddf74de59b376504e87d545da522246e" exitCode=0
Oct 30 08:07:32.979487 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:32.979331    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f5ceca-5f57ff8b46-28rhc"
Oct 30 08:07:32.979487 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:32.979370    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f5ceca-5f57ff8b46-28rhc" event={"ID":"671ab50b-8475-463c-8815-767e495feaa9","Type":"ContainerDied","Data":"f7fa1c951f077cf238ea5cadcae2815dddf74de59b376504e87d545da522246e"}
Oct 30 08:07:32.979487 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:32.979428    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f5ceca-5f57ff8b46-28rhc" event={"ID":"671ab50b-8475-463c-8815-767e495feaa9","Type":"ContainerDied","Data":"a282392afa187a06e3c8a27b05fa693a90b45b8a891c486587a8d73440f14086"}
Oct 30 08:07:32.979487 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:32.979452    6224 scope.go:122] "RemoveContainer" containerID="f7fa1c951f077cf238ea5cadcae2815dddf74de59b376504e87d545da522246e"
Oct 30 08:07:32.988627 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:32.988610    6224 scope.go:122] "RemoveContainer" containerID="f7fa1c951f077cf238ea5cadcae2815dddf74de59b376504e87d545da522246e"
Oct 30 08:07:32.988885 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:07:32.988867    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7fa1c951f077cf238ea5cadcae2815dddf74de59b376504e87d545da522246e\": container with ID starting with f7fa1c951f077cf238ea5cadcae2815dddf74de59b376504e87d545da522246e not found: ID does not exist" containerID="f7fa1c951f077cf238ea5cadcae2815dddf74de59b376504e87d545da522246e"
Oct 30 08:07:32.988934 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:32.988893    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7fa1c951f077cf238ea5cadcae2815dddf74de59b376504e87d545da522246e"} err="failed to get container status \"f7fa1c951f077cf238ea5cadcae2815dddf74de59b376504e87d545da522246e\": rpc error: code = NotFound desc = could not find container \"f7fa1c951f077cf238ea5cadcae2815dddf74de59b376504e87d545da522246e\": container with ID starting with f7fa1c951f077cf238ea5cadcae2815dddf74de59b376504e87d545da522246e not found: ID does not exist"
Oct 30 08:07:32.992373 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:32.992355    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2hwj\" (UniqueName: \"kubernetes.io/projected/671ab50b-8475-463c-8815-767e495feaa9-kube-api-access-x2hwj\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:07:32.992430 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:32.992376    6224 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/671ab50b-8475-463c-8815-767e495feaa9-plugins-config-volume\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:07:33.038848 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:33.038783    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-f5ceca-5f57ff8b46-28rhc"]
Oct 30 08:07:33.042549 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:33.042521    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-f5ceca-5f57ff8b46-28rhc"]
Oct 30 08:07:33.502269 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:33.502233    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="671ab50b-8475-463c-8815-767e495feaa9" path="/var/lib/kubelet/pods/671ab50b-8475-463c-8815-767e495feaa9/volumes"
Oct 30 08:07:41.363259 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:41.363213    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw" podUID="0597696e-26b8-4ef8-8697-64d49b2b9398" containerName="main" containerID="cri-o://1e1e6c1f02aac62165ac73f472cc11e8a2f238417bdd3b38f2a5d72aac48d5aa" gracePeriod=45
Oct 30 08:07:41.387900 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:41.387857    6224 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-2eda2c/simulator-w3-2eda2c-kserve-56776b9849-9bqzm" podUID="482e6b7c-2231-423c-b54f-a54e56f16ab7" containerName="main" containerID="cri-o://32924d950abe45c0e04c671daea038e121ef91ddef69154d07678df7f821c1ca" gracePeriod=45
Oct 30 08:07:41.644949 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:41.644910    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2eda2c/simulator-w3-2eda2c-kserve-56776b9849-9bqzm"
Oct 30 08:07:41.647775 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:41.647754    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw"
Oct 30 08:07:41.671731 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:41.671694    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/482e6b7c-2231-423c-b54f-a54e56f16ab7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/482e6b7c-2231-423c-b54f-a54e56f16ab7-tmp-dir\") pod \"482e6b7c-2231-423c-b54f-a54e56f16ab7\" (UID: \"482e6b7c-2231-423c-b54f-a54e56f16ab7\") "
Oct 30 08:07:41.671909 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:41.671738    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/482e6b7c-2231-423c-b54f-a54e56f16ab7-kube-api-access-99qq6\" (UniqueName: \"kubernetes.io/projected/482e6b7c-2231-423c-b54f-a54e56f16ab7-kube-api-access-99qq6\") pod \"482e6b7c-2231-423c-b54f-a54e56f16ab7\" (UID: \"482e6b7c-2231-423c-b54f-a54e56f16ab7\") "
Oct 30 08:07:41.671909 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:41.671759    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0597696e-26b8-4ef8-8697-64d49b2b9398-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0597696e-26b8-4ef8-8697-64d49b2b9398-model-cache\") pod \"0597696e-26b8-4ef8-8697-64d49b2b9398\" (UID: \"0597696e-26b8-4ef8-8697-64d49b2b9398\") "
Oct 30 08:07:41.671909 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:41.671787    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0597696e-26b8-4ef8-8697-64d49b2b9398-home\" (UniqueName: \"kubernetes.io/empty-dir/0597696e-26b8-4ef8-8697-64d49b2b9398-home\") pod \"0597696e-26b8-4ef8-8697-64d49b2b9398\" (UID: \"0597696e-26b8-4ef8-8697-64d49b2b9398\") "
Oct 30 08:07:41.671909 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:41.671845    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0597696e-26b8-4ef8-8697-64d49b2b9398-kube-api-access-9f8gl\" (UniqueName: \"kubernetes.io/projected/0597696e-26b8-4ef8-8697-64d49b2b9398-kube-api-access-9f8gl\") pod \"0597696e-26b8-4ef8-8697-64d49b2b9398\" (UID: \"0597696e-26b8-4ef8-8697-64d49b2b9398\") "
Oct 30 08:07:41.671909 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:41.671879    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0597696e-26b8-4ef8-8697-64d49b2b9398-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0597696e-26b8-4ef8-8697-64d49b2b9398-dshm\") pod \"0597696e-26b8-4ef8-8697-64d49b2b9398\" (UID: \"0597696e-26b8-4ef8-8697-64d49b2b9398\") "
Oct 30 08:07:41.672275 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:41.671914    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/482e6b7c-2231-423c-b54f-a54e56f16ab7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/482e6b7c-2231-423c-b54f-a54e56f16ab7-dshm\") pod \"482e6b7c-2231-423c-b54f-a54e56f16ab7\" (UID: \"482e6b7c-2231-423c-b54f-a54e56f16ab7\") "
Oct 30 08:07:41.672275 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:41.671973    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/482e6b7c-2231-423c-b54f-a54e56f16ab7-home\" (UniqueName: \"kubernetes.io/empty-dir/482e6b7c-2231-423c-b54f-a54e56f16ab7-home\") pod \"482e6b7c-2231-423c-b54f-a54e56f16ab7\" (UID: \"482e6b7c-2231-423c-b54f-a54e56f16ab7\") "
Oct 30 08:07:41.672275 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:41.672017    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/482e6b7c-2231-423c-b54f-a54e56f16ab7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/482e6b7c-2231-423c-b54f-a54e56f16ab7-model-cache\") pod \"482e6b7c-2231-423c-b54f-a54e56f16ab7\" (UID: \"482e6b7c-2231-423c-b54f-a54e56f16ab7\") "
Oct 30 08:07:41.672275 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:41.672054    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0597696e-26b8-4ef8-8697-64d49b2b9398-tls-certs\" (UniqueName: \"kubernetes.io/secret/0597696e-26b8-4ef8-8697-64d49b2b9398-tls-certs\") pod \"0597696e-26b8-4ef8-8697-64d49b2b9398\" (UID: \"0597696e-26b8-4ef8-8697-64d49b2b9398\") "
Oct 30 08:07:41.672275 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:41.672064    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0597696e-26b8-4ef8-8697-64d49b2b9398-model-cache" pod "0597696e-26b8-4ef8-8697-64d49b2b9398" (UID: "0597696e-26b8-4ef8-8697-64d49b2b9398"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:07:41.672275 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:41.672076    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0597696e-26b8-4ef8-8697-64d49b2b9398-home" pod "0597696e-26b8-4ef8-8697-64d49b2b9398" (UID: "0597696e-26b8-4ef8-8697-64d49b2b9398"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:07:41.672275 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:41.672067    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/482e6b7c-2231-423c-b54f-a54e56f16ab7-tmp-dir" pod "482e6b7c-2231-423c-b54f-a54e56f16ab7" (UID: "482e6b7c-2231-423c-b54f-a54e56f16ab7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:07:41.672275 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:41.672088    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0597696e-26b8-4ef8-8697-64d49b2b9398-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0597696e-26b8-4ef8-8697-64d49b2b9398-tmp-dir\") pod \"0597696e-26b8-4ef8-8697-64d49b2b9398\" (UID: \"0597696e-26b8-4ef8-8697-64d49b2b9398\") "
Oct 30 08:07:41.672275 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:41.672152    6224 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/482e6b7c-2231-423c-b54f-a54e56f16ab7-tls-certs\" (UniqueName: \"kubernetes.io/secret/482e6b7c-2231-423c-b54f-a54e56f16ab7-tls-certs\") pod \"482e6b7c-2231-423c-b54f-a54e56f16ab7\" (UID: \"482e6b7c-2231-423c-b54f-a54e56f16ab7\") "
Oct 30 08:07:41.672737 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:41.672278    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0597696e-26b8-4ef8-8697-64d49b2b9398-tmp-dir" pod "0597696e-26b8-4ef8-8697-64d49b2b9398" (UID: "0597696e-26b8-4ef8-8697-64d49b2b9398"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:07:41.672737 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:41.672290    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/482e6b7c-2231-423c-b54f-a54e56f16ab7-home" pod "482e6b7c-2231-423c-b54f-a54e56f16ab7" (UID: "482e6b7c-2231-423c-b54f-a54e56f16ab7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:07:41.672737 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:41.672301    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/482e6b7c-2231-423c-b54f-a54e56f16ab7-model-cache" pod "482e6b7c-2231-423c-b54f-a54e56f16ab7" (UID: "482e6b7c-2231-423c-b54f-a54e56f16ab7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:07:41.672737 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:41.672476    6224 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0597696e-26b8-4ef8-8697-64d49b2b9398-tmp-dir\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:07:41.672737 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:41.672499    6224 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/482e6b7c-2231-423c-b54f-a54e56f16ab7-tmp-dir\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:07:41.672737 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:41.672513    6224 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0597696e-26b8-4ef8-8697-64d49b2b9398-model-cache\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:07:41.672737 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:41.672527    6224 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0597696e-26b8-4ef8-8697-64d49b2b9398-home\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:07:41.672737 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:41.672541    6224 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/482e6b7c-2231-423c-b54f-a54e56f16ab7-home\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:07:41.672737 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:41.672553    6224 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/482e6b7c-2231-423c-b54f-a54e56f16ab7-model-cache\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:07:41.674574 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:41.674551    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/482e6b7c-2231-423c-b54f-a54e56f16ab7-dshm" pod "482e6b7c-2231-423c-b54f-a54e56f16ab7" (UID: "482e6b7c-2231-423c-b54f-a54e56f16ab7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:07:41.674689 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:41.674615    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0597696e-26b8-4ef8-8697-64d49b2b9398-tls-certs" pod "0597696e-26b8-4ef8-8697-64d49b2b9398" (UID: "0597696e-26b8-4ef8-8697-64d49b2b9398"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:07:41.674689 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:41.674634    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/482e6b7c-2231-423c-b54f-a54e56f16ab7-kube-api-access-99qq6" pod "482e6b7c-2231-423c-b54f-a54e56f16ab7" (UID: "482e6b7c-2231-423c-b54f-a54e56f16ab7"). InnerVolumeSpecName "kube-api-access-99qq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:07:41.674689 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:41.674645    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0597696e-26b8-4ef8-8697-64d49b2b9398-kube-api-access-9f8gl" pod "0597696e-26b8-4ef8-8697-64d49b2b9398" (UID: "0597696e-26b8-4ef8-8697-64d49b2b9398"). InnerVolumeSpecName "kube-api-access-9f8gl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 08:07:41.674689 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:41.674667    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/482e6b7c-2231-423c-b54f-a54e56f16ab7-tls-certs" pod "482e6b7c-2231-423c-b54f-a54e56f16ab7" (UID: "482e6b7c-2231-423c-b54f-a54e56f16ab7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 08:07:41.674689 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:41.674678    6224 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0597696e-26b8-4ef8-8697-64d49b2b9398-dshm" pod "0597696e-26b8-4ef8-8697-64d49b2b9398" (UID: "0597696e-26b8-4ef8-8697-64d49b2b9398"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 08:07:41.773098 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:41.773056    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9f8gl\" (UniqueName: \"kubernetes.io/projected/0597696e-26b8-4ef8-8697-64d49b2b9398-kube-api-access-9f8gl\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:07:41.773098 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:41.773098    6224 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0597696e-26b8-4ef8-8697-64d49b2b9398-dshm\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:07:41.773378 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:41.773115    6224 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/482e6b7c-2231-423c-b54f-a54e56f16ab7-dshm\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:07:41.773378 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:41.773134    6224 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0597696e-26b8-4ef8-8697-64d49b2b9398-tls-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:07:41.773378 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:41.773150    6224 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/482e6b7c-2231-423c-b54f-a54e56f16ab7-tls-certs\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:07:41.773378 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:41.773162    6224 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99qq6\" (UniqueName: \"kubernetes.io/projected/482e6b7c-2231-423c-b54f-a54e56f16ab7-kube-api-access-99qq6\") on node \"ip-10-0-128-125.ec2.internal\" DevicePath \"\""
Oct 30 08:07:42.029360 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:42.029269    6224 generic.go:356] "Generic (PLEG): container finished" podID="0597696e-26b8-4ef8-8697-64d49b2b9398" containerID="1e1e6c1f02aac62165ac73f472cc11e8a2f238417bdd3b38f2a5d72aac48d5aa" exitCode=0
Oct 30 08:07:42.029360 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:42.029317    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw"
Oct 30 08:07:42.029582 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:42.029352    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw" event={"ID":"0597696e-26b8-4ef8-8697-64d49b2b9398","Type":"ContainerDied","Data":"1e1e6c1f02aac62165ac73f472cc11e8a2f238417bdd3b38f2a5d72aac48d5aa"}
Oct 30 08:07:42.029582 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:42.029400    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2eda2c/distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw" event={"ID":"0597696e-26b8-4ef8-8697-64d49b2b9398","Type":"ContainerDied","Data":"cc16343a5fee439198c92efcb77fb41d9b058ffcf78fc347091fe748fcb3cc3b"}
Oct 30 08:07:42.029582 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:42.029422    6224 scope.go:122] "RemoveContainer" containerID="1e1e6c1f02aac62165ac73f472cc11e8a2f238417bdd3b38f2a5d72aac48d5aa"
Oct 30 08:07:42.030736 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:42.030717    6224 generic.go:356] "Generic (PLEG): container finished" podID="482e6b7c-2231-423c-b54f-a54e56f16ab7" containerID="32924d950abe45c0e04c671daea038e121ef91ddef69154d07678df7f821c1ca" exitCode=0
Oct 30 08:07:42.030852 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:42.030767    6224 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2eda2c/simulator-w3-2eda2c-kserve-56776b9849-9bqzm"
Oct 30 08:07:42.030852 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:42.030828    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2eda2c/simulator-w3-2eda2c-kserve-56776b9849-9bqzm" event={"ID":"482e6b7c-2231-423c-b54f-a54e56f16ab7","Type":"ContainerDied","Data":"32924d950abe45c0e04c671daea038e121ef91ddef69154d07678df7f821c1ca"}
Oct 30 08:07:42.030990 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:42.030868    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2eda2c/simulator-w3-2eda2c-kserve-56776b9849-9bqzm" event={"ID":"482e6b7c-2231-423c-b54f-a54e56f16ab7","Type":"ContainerDied","Data":"c4da7976c80abf5b67e3a42e73c90b4fa01584653045c71d34c7fb91df65b426"}
Oct 30 08:07:42.038942 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:42.038719    6224 scope.go:122] "RemoveContainer" containerID="1e1e6c1f02aac62165ac73f472cc11e8a2f238417bdd3b38f2a5d72aac48d5aa"
Oct 30 08:07:42.039139 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:07:42.039115    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e1e6c1f02aac62165ac73f472cc11e8a2f238417bdd3b38f2a5d72aac48d5aa\": container with ID starting with 1e1e6c1f02aac62165ac73f472cc11e8a2f238417bdd3b38f2a5d72aac48d5aa not found: ID does not exist" containerID="1e1e6c1f02aac62165ac73f472cc11e8a2f238417bdd3b38f2a5d72aac48d5aa"
Oct 30 08:07:42.039239 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:42.039145    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e1e6c1f02aac62165ac73f472cc11e8a2f238417bdd3b38f2a5d72aac48d5aa"} err="failed to get container status \"1e1e6c1f02aac62165ac73f472cc11e8a2f238417bdd3b38f2a5d72aac48d5aa\": rpc error: code = NotFound desc = could not find container \"1e1e6c1f02aac62165ac73f472cc11e8a2f238417bdd3b38f2a5d72aac48d5aa\": container with ID starting with 1e1e6c1f02aac62165ac73f472cc11e8a2f238417bdd3b38f2a5d72aac48d5aa not found: ID does not exist"
Oct 30 08:07:42.039239 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:42.039163    6224 scope.go:122] "RemoveContainer" containerID="32924d950abe45c0e04c671daea038e121ef91ddef69154d07678df7f821c1ca"
Oct 30 08:07:42.047579 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:42.047558    6224 scope.go:122] "RemoveContainer" containerID="32924d950abe45c0e04c671daea038e121ef91ddef69154d07678df7f821c1ca"
Oct 30 08:07:42.048486 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:07:42.048465    6224 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32924d950abe45c0e04c671daea038e121ef91ddef69154d07678df7f821c1ca\": container with ID starting with 32924d950abe45c0e04c671daea038e121ef91ddef69154d07678df7f821c1ca not found: ID does not exist" containerID="32924d950abe45c0e04c671daea038e121ef91ddef69154d07678df7f821c1ca"
Oct 30 08:07:42.048561 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:42.048493    6224 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32924d950abe45c0e04c671daea038e121ef91ddef69154d07678df7f821c1ca"} err="failed to get container status \"32924d950abe45c0e04c671daea038e121ef91ddef69154d07678df7f821c1ca\": rpc error: code = NotFound desc = could not find container \"32924d950abe45c0e04c671daea038e121ef91ddef69154d07678df7f821c1ca\": container with ID starting with 32924d950abe45c0e04c671daea038e121ef91ddef69154d07678df7f821c1ca not found: ID does not exist"
Oct 30 08:07:42.094772 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:42.094732    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2eda2c/distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw"]
Oct 30 08:07:42.098556 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:42.098527    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-2eda2c/distinct-2-w3-2eda2c-kserve-677d6c85c6-4dnhw"]
Oct 30 08:07:42.147670 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:42.147635    6224 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2eda2c/simulator-w3-2eda2c-kserve-56776b9849-9bqzm"]
Oct 30 08:07:42.157469 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:42.157435    6224 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-2eda2c/simulator-w3-2eda2c-kserve-56776b9849-9bqzm"]
Oct 30 08:07:43.500928 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:43.500896    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0597696e-26b8-4ef8-8697-64d49b2b9398" path="/var/lib/kubelet/pods/0597696e-26b8-4ef8-8697-64d49b2b9398/volumes"
Oct 30 08:07:43.501352 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:43.501306    6224 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="482e6b7c-2231-423c-b54f-a54e56f16ab7" path="/var/lib/kubelet/pods/482e6b7c-2231-423c-b54f-a54e56f16ab7/volumes"
Oct 30 08:07:49.057572 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:49.057534    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 08:07:59.986282 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:07:59.986251    6224 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-gwz2f_c84bb85f-b5d5-4934-b67f-0589d63fc861/manager/0.log"
Oct 30 08:08:00.166005 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:00.165971    6224 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-zbc8w_e44c3a31-a231-4a77-903b-ad4afbecaae0/kuadrant-console-plugin/0.log"
Oct 30 08:08:00.745053 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:00.745019    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9m92k"]
Oct 30 08:08:01.856068 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:08:01.856030    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 08:08:02.329930 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:02.329891    6224 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-9m92k_e1113d78-e48d-48b2-aab4-c3f859f69eae/limitador/0.log"
Oct 30 08:08:02.548634 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:02.548599    6224 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-8stfv_1270d903-7234-4ffb-825c-b4c71a4e9b86/manager/0.log"
Oct 30 08:08:04.020547 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:04.020508    6224 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-44xkr_0f8fa62e-5595-4c17-b186-f604f06b17ed/manager/0.log"
Oct 30 08:08:17.545064 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:17.545021    6224 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-44xkr_0f8fa62e-5595-4c17-b186-f604f06b17ed/manager/0.log"
Oct 30 08:08:19.628894 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:19.628848    6224 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5bddd4c9fd-sjdgf_567cf906-1440-49f5-876b-692dab3f4fed/maas-api/0.log"
Oct 30 08:08:21.787391 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:21.787358    6224 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-914f19bf-csp6q_f5be612e-951b-4688-895c-d226c6ef7b22/revoke-keys/0.log"
Oct 30 08:08:24.496796 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:24.496764    6224 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-gwz2f_c84bb85f-b5d5-4934-b67f-0589d63fc861/manager/0.log"
Oct 30 08:08:24.690788 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:24.690754    6224 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-zbc8w_e44c3a31-a231-4a77-903b-ad4afbecaae0/kuadrant-console-plugin/0.log"
Oct 30 08:08:26.935083 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:26.935050    6224 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-9m92k_e1113d78-e48d-48b2-aab4-c3f859f69eae/limitador/0.log"
Oct 30 08:08:27.141082 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:27.141020    6224 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-8stfv_1270d903-7234-4ffb-825c-b4c71a4e9b86/manager/0.log"
Oct 30 08:08:28.573536 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:28.573503    6224 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54599bd898-89xfk_504b6d22-335a-4ba3-b724-da2c4880543d/kube-auth-proxy/0.log"
Oct 30 08:08:31.856291 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:08:31.856256    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 08:08:43.751823 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:43.751764    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-png44/must-gather-k5fv9"]
Oct 30 08:08:43.752313 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:43.752245    6224 cpu_manager.go:414] "containerMap: removing container" podUID="1230dcae-e1cd-460d-86fc-bd40c8427b5f" containerName="main"
Oct 30 08:08:43.752313 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:43.752264    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1230dcae-e1cd-460d-86fc-bd40c8427b5f" containerName="main"
Oct 30 08:08:43.752313 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:43.752279    6224 cpu_manager.go:414] "containerMap: removing container" podUID="671ab50b-8475-463c-8815-767e495feaa9" containerName="payload-processing"
Oct 30 08:08:43.752313 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:43.752286    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="671ab50b-8475-463c-8815-767e495feaa9" containerName="payload-processing"
Oct 30 08:08:43.752313 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:43.752297    6224 cpu_manager.go:414] "containerMap: removing container" podUID="0597696e-26b8-4ef8-8697-64d49b2b9398" containerName="main"
Oct 30 08:08:43.752313 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:43.752305    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0597696e-26b8-4ef8-8697-64d49b2b9398" containerName="main"
Oct 30 08:08:43.752313 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:43.752316    6224 cpu_manager.go:414] "containerMap: removing container" podUID="50bc25e6-c53f-4d16-b1fd-738cdd8c6a10" containerName="istio-proxy"
Oct 30 08:08:43.752713 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:43.752323    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50bc25e6-c53f-4d16-b1fd-738cdd8c6a10" containerName="istio-proxy"
Oct 30 08:08:43.752713 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:43.752341    6224 cpu_manager.go:414] "containerMap: removing container" podUID="50bc25e6-c53f-4d16-b1fd-738cdd8c6a10" containerName="istio-proxy"
Oct 30 08:08:43.752713 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:43.752348    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50bc25e6-c53f-4d16-b1fd-738cdd8c6a10" containerName="istio-proxy"
Oct 30 08:08:43.752713 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:43.752382    6224 cpu_manager.go:414] "containerMap: removing container" podUID="482e6b7c-2231-423c-b54f-a54e56f16ab7" containerName="main"
Oct 30 08:08:43.752713 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:43.752389    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="482e6b7c-2231-423c-b54f-a54e56f16ab7" containerName="main"
Oct 30 08:08:43.752713 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:43.752398    6224 cpu_manager.go:414] "containerMap: removing container" podUID="f5be612e-951b-4688-895c-d226c6ef7b22" containerName="revoke-keys"
Oct 30 08:08:43.752713 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:43.752405    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5be612e-951b-4688-895c-d226c6ef7b22" containerName="revoke-keys"
Oct 30 08:08:43.752713 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:43.752419    6224 cpu_manager.go:414] "containerMap: removing container" podUID="71294bfa-ac2c-40b7-b889-0527daa07f63" containerName="main"
Oct 30 08:08:43.752713 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:43.752427    6224 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71294bfa-ac2c-40b7-b889-0527daa07f63" containerName="main"
Oct 30 08:08:43.752713 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:43.752514    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="50bc25e6-c53f-4d16-b1fd-738cdd8c6a10" containerName="istio-proxy"
Oct 30 08:08:43.752713 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:43.752531    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="482e6b7c-2231-423c-b54f-a54e56f16ab7" containerName="main"
Oct 30 08:08:43.752713 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:43.752540    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5be612e-951b-4688-895c-d226c6ef7b22" containerName="revoke-keys"
Oct 30 08:08:43.752713 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:43.752550    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="71294bfa-ac2c-40b7-b889-0527daa07f63" containerName="main"
Oct 30 08:08:43.752713 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:43.752562    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="1230dcae-e1cd-460d-86fc-bd40c8427b5f" containerName="main"
Oct 30 08:08:43.752713 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:43.752571    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="671ab50b-8475-463c-8815-767e495feaa9" containerName="payload-processing"
Oct 30 08:08:43.752713 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:43.752578    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="0597696e-26b8-4ef8-8697-64d49b2b9398" containerName="main"
Oct 30 08:08:43.753572 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:43.752740    6224 memory_manager.go:365] "RemoveStaleState removing state" podUID="50bc25e6-c53f-4d16-b1fd-738cdd8c6a10" containerName="istio-proxy"
Oct 30 08:08:43.756144 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:43.756122    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-png44/must-gather-k5fv9"
Oct 30 08:08:43.788102 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:43.788058    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-png44\"/\"openshift-service-ca.crt\""
Oct 30 08:08:43.788330 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:43.788167    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-png44\"/\"kube-root-ca.crt\""
Oct 30 08:08:43.788456 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:43.788413    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-png44\"/\"default-dockercfg-hh7kk\""
Oct 30 08:08:43.790410 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:43.790350    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-png44/must-gather-k5fv9"]
Oct 30 08:08:43.877644 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:43.877569    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/70bb1463-2c78-4b67-b5d9-402ceae23c33-must-gather-output\") pod \"must-gather-k5fv9\" (UID: \"70bb1463-2c78-4b67-b5d9-402ceae23c33\") " pod="openshift-must-gather-png44/must-gather-k5fv9"
Oct 30 08:08:43.877644 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:43.877607    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4p98\" (UniqueName: \"kubernetes.io/projected/70bb1463-2c78-4b67-b5d9-402ceae23c33-kube-api-access-l4p98\") pod \"must-gather-k5fv9\" (UID: \"70bb1463-2c78-4b67-b5d9-402ceae23c33\") " pod="openshift-must-gather-png44/must-gather-k5fv9"
Oct 30 08:08:43.979076 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:43.979037    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/70bb1463-2c78-4b67-b5d9-402ceae23c33-must-gather-output\") pod \"must-gather-k5fv9\" (UID: \"70bb1463-2c78-4b67-b5d9-402ceae23c33\") " pod="openshift-must-gather-png44/must-gather-k5fv9"
Oct 30 08:08:43.979076 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:43.979083    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4p98\" (UniqueName: \"kubernetes.io/projected/70bb1463-2c78-4b67-b5d9-402ceae23c33-kube-api-access-l4p98\") pod \"must-gather-k5fv9\" (UID: \"70bb1463-2c78-4b67-b5d9-402ceae23c33\") " pod="openshift-must-gather-png44/must-gather-k5fv9"
Oct 30 08:08:43.979412 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:43.979392    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/70bb1463-2c78-4b67-b5d9-402ceae23c33-must-gather-output\") pod \"must-gather-k5fv9\" (UID: \"70bb1463-2c78-4b67-b5d9-402ceae23c33\") " pod="openshift-must-gather-png44/must-gather-k5fv9"
Oct 30 08:08:44.033478 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:44.033442    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4p98\" (UniqueName: \"kubernetes.io/projected/70bb1463-2c78-4b67-b5d9-402ceae23c33-kube-api-access-l4p98\") pod \"must-gather-k5fv9\" (UID: \"70bb1463-2c78-4b67-b5d9-402ceae23c33\") " pod="openshift-must-gather-png44/must-gather-k5fv9"
Oct 30 08:08:44.070005 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:44.069975    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-png44/must-gather-k5fv9"
Oct 30 08:08:44.404798 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:44.404776    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-png44/must-gather-k5fv9"]
Oct 30 08:08:44.405983 ip-10-0-128-125 kubenswrapper[6224]: W1030 08:08:44.405958    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod70bb1463_2c78_4b67_b5d9_402ceae23c33.slice/crio-592ec8d6db8d9ff68fd1c32108af651fd8462a3640e91691d5c83181c994ef73 WatchSource:0}: Error finding container 592ec8d6db8d9ff68fd1c32108af651fd8462a3640e91691d5c83181c994ef73: Status 404 returned error can't find the container with id 592ec8d6db8d9ff68fd1c32108af651fd8462a3640e91691d5c83181c994ef73
Oct 30 08:08:44.408186 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:44.408170    6224 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 08:08:45.349383 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:45.349343    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-png44/must-gather-k5fv9" event={"ID":"70bb1463-2c78-4b67-b5d9-402ceae23c33","Type":"ContainerStarted","Data":"592ec8d6db8d9ff68fd1c32108af651fd8462a3640e91691d5c83181c994ef73"}
Oct 30 08:08:48.949513 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:48.949477    6224 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-qktbg_9092e2e8-ad23-426c-b6f1-14e1a2b9221e/global-pull-secret-syncer/0.log"
Oct 30 08:08:49.874909 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:49.874823    6224 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-wv8tb_31519c22-050e-435c-aec3-7be0b68ff313/konnectivity-agent/0.log"
Oct 30 08:08:50.001575 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:50.001538    6224 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-125.ec2.internal_4ba2b683794afdaba9ca4ed884bfac10/haproxy/0.log"
Oct 30 08:08:50.379360 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:50.379316    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-png44/must-gather-k5fv9" event={"ID":"70bb1463-2c78-4b67-b5d9-402ceae23c33","Type":"ContainerStarted","Data":"e348f7a595e7d3d6ba4aac0b49f23f82a8414665b1fce2aa78e6b227c707eeac"}
Oct 30 08:08:50.379360 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:50.379365    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-png44/must-gather-k5fv9" event={"ID":"70bb1463-2c78-4b67-b5d9-402ceae23c33","Type":"ContainerStarted","Data":"878d81d3c7ae26f0b96df8f7c72ad0187833ceda0399dd8cd9ae590116291318"}
Oct 30 08:08:50.424404 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:50.424348    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-png44/must-gather-k5fv9" podStartSLOduration=2.266425008 podStartE2EDuration="7.424335144s" totalImagesPullingTime="5.157910136s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:08:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:08:50.423175188 +0000 UTC m=+4289.032582158" watchObservedRunningTime="2026-10-30 08:08:50.424335144 +0000 UTC m=+4289.033742092"
Oct 30 08:08:53.005063 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:53.005001    6224 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-44xkr_0f8fa62e-5595-4c17-b186-f604f06b17ed/manager/0.log"
Oct 30 08:08:53.052692 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:53.052659    6224 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-44xkr_0f8fa62e-5595-4c17-b186-f604f06b17ed/manager/0.log"
Oct 30 08:08:59.343235 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:59.343197    6224 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-h4tpf/perf-node-gather-daemonset-j8bck"]
Oct 30 08:08:59.347781 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:59.347763    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-h4tpf/perf-node-gather-daemonset-j8bck"
Oct 30 08:08:59.366015 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:59.365988    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-h4tpf\"/\"openshift-service-ca.crt\""
Oct 30 08:08:59.366256 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:59.366231    6224 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-h4tpf\"/\"kube-root-ca.crt\""
Oct 30 08:08:59.366379 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:59.366338    6224 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-h4tpf\"/\"default-dockercfg-x222n\""
Oct 30 08:08:59.368036 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:59.368014    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-h4tpf/perf-node-gather-daemonset-j8bck"]
Oct 30 08:08:59.427345 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:59.427315    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/01be6602-625e-43b2-a75d-6607ae5895d3-lib-modules\") pod \"perf-node-gather-daemonset-j8bck\" (UID: \"01be6602-625e-43b2-a75d-6607ae5895d3\") " pod="openshift-must-gather-h4tpf/perf-node-gather-daemonset-j8bck"
Oct 30 08:08:59.427345 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:59.427345    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/01be6602-625e-43b2-a75d-6607ae5895d3-sys\") pod \"perf-node-gather-daemonset-j8bck\" (UID: \"01be6602-625e-43b2-a75d-6607ae5895d3\") " pod="openshift-must-gather-h4tpf/perf-node-gather-daemonset-j8bck"
Oct 30 08:08:59.427559 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:59.427364    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/01be6602-625e-43b2-a75d-6607ae5895d3-podres\") pod \"perf-node-gather-daemonset-j8bck\" (UID: \"01be6602-625e-43b2-a75d-6607ae5895d3\") " pod="openshift-must-gather-h4tpf/perf-node-gather-daemonset-j8bck"
Oct 30 08:08:59.427559 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:59.427503    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/01be6602-625e-43b2-a75d-6607ae5895d3-proc\") pod \"perf-node-gather-daemonset-j8bck\" (UID: \"01be6602-625e-43b2-a75d-6607ae5895d3\") " pod="openshift-must-gather-h4tpf/perf-node-gather-daemonset-j8bck"
Oct 30 08:08:59.427559 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:59.427551    6224 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgww4\" (UniqueName: \"kubernetes.io/projected/01be6602-625e-43b2-a75d-6607ae5895d3-kube-api-access-vgww4\") pod \"perf-node-gather-daemonset-j8bck\" (UID: \"01be6602-625e-43b2-a75d-6607ae5895d3\") " pod="openshift-must-gather-h4tpf/perf-node-gather-daemonset-j8bck"
Oct 30 08:08:59.528316 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:59.528278    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgww4\" (UniqueName: \"kubernetes.io/projected/01be6602-625e-43b2-a75d-6607ae5895d3-kube-api-access-vgww4\") pod \"perf-node-gather-daemonset-j8bck\" (UID: \"01be6602-625e-43b2-a75d-6607ae5895d3\") " pod="openshift-must-gather-h4tpf/perf-node-gather-daemonset-j8bck"
Oct 30 08:08:59.528525 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:59.528325    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/01be6602-625e-43b2-a75d-6607ae5895d3-lib-modules\") pod \"perf-node-gather-daemonset-j8bck\" (UID: \"01be6602-625e-43b2-a75d-6607ae5895d3\") " pod="openshift-must-gather-h4tpf/perf-node-gather-daemonset-j8bck"
Oct 30 08:08:59.528525 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:59.528350    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/01be6602-625e-43b2-a75d-6607ae5895d3-sys\") pod \"perf-node-gather-daemonset-j8bck\" (UID: \"01be6602-625e-43b2-a75d-6607ae5895d3\") " pod="openshift-must-gather-h4tpf/perf-node-gather-daemonset-j8bck"
Oct 30 08:08:59.528525 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:59.528371    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/01be6602-625e-43b2-a75d-6607ae5895d3-podres\") pod \"perf-node-gather-daemonset-j8bck\" (UID: \"01be6602-625e-43b2-a75d-6607ae5895d3\") " pod="openshift-must-gather-h4tpf/perf-node-gather-daemonset-j8bck"
Oct 30 08:08:59.528525 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:59.528439    6224 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/01be6602-625e-43b2-a75d-6607ae5895d3-proc\") pod \"perf-node-gather-daemonset-j8bck\" (UID: \"01be6602-625e-43b2-a75d-6607ae5895d3\") " pod="openshift-must-gather-h4tpf/perf-node-gather-daemonset-j8bck"
Oct 30 08:08:59.528525 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:59.528480    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/01be6602-625e-43b2-a75d-6607ae5895d3-lib-modules\") pod \"perf-node-gather-daemonset-j8bck\" (UID: \"01be6602-625e-43b2-a75d-6607ae5895d3\") " pod="openshift-must-gather-h4tpf/perf-node-gather-daemonset-j8bck"
Oct 30 08:08:59.528525 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:59.528480    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/01be6602-625e-43b2-a75d-6607ae5895d3-sys\") pod \"perf-node-gather-daemonset-j8bck\" (UID: \"01be6602-625e-43b2-a75d-6607ae5895d3\") " pod="openshift-must-gather-h4tpf/perf-node-gather-daemonset-j8bck"
Oct 30 08:08:59.528757 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:59.528529    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/01be6602-625e-43b2-a75d-6607ae5895d3-podres\") pod \"perf-node-gather-daemonset-j8bck\" (UID: \"01be6602-625e-43b2-a75d-6607ae5895d3\") " pod="openshift-must-gather-h4tpf/perf-node-gather-daemonset-j8bck"
Oct 30 08:08:59.528757 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:59.528530    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/01be6602-625e-43b2-a75d-6607ae5895d3-proc\") pod \"perf-node-gather-daemonset-j8bck\" (UID: \"01be6602-625e-43b2-a75d-6607ae5895d3\") " pod="openshift-must-gather-h4tpf/perf-node-gather-daemonset-j8bck"
Oct 30 08:08:59.554060 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:59.554034    6224 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgww4\" (UniqueName: \"kubernetes.io/projected/01be6602-625e-43b2-a75d-6607ae5895d3-kube-api-access-vgww4\") pod \"perf-node-gather-daemonset-j8bck\" (UID: \"01be6602-625e-43b2-a75d-6607ae5895d3\") " pod="openshift-must-gather-h4tpf/perf-node-gather-daemonset-j8bck"
Oct 30 08:08:59.660288 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:59.660200    6224 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-h4tpf/perf-node-gather-daemonset-j8bck"
Oct 30 08:08:59.955860 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:08:59.955826    6224 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-h4tpf/perf-node-gather-daemonset-j8bck"]
Oct 30 08:08:59.956393 ip-10-0-128-125 kubenswrapper[6224]: W1030 08:08:59.956372    6224 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod01be6602_625e_43b2_a75d_6607ae5895d3.slice/crio-9b4077fe7d422527b548628a67158211212c3eec926a25dfebb34cdcf92ddf89 WatchSource:0}: Error finding container 9b4077fe7d422527b548628a67158211212c3eec926a25dfebb34cdcf92ddf89: Status 404 returned error can't find the container with id 9b4077fe7d422527b548628a67158211212c3eec926a25dfebb34cdcf92ddf89
Oct 30 08:09:00.434178 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:09:00.434138    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-h4tpf/perf-node-gather-daemonset-j8bck" event={"ID":"01be6602-625e-43b2-a75d-6607ae5895d3","Type":"ContainerStarted","Data":"c3e0e4d6084b7f046aa9f5ba681bce9b2028a04a8fc0c1403c26fb20934fe3fb"}
Oct 30 08:09:00.434934 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:09:00.434200    6224 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-h4tpf/perf-node-gather-daemonset-j8bck" event={"ID":"01be6602-625e-43b2-a75d-6607ae5895d3","Type":"ContainerStarted","Data":"9b4077fe7d422527b548628a67158211212c3eec926a25dfebb34cdcf92ddf89"}
Oct 30 08:09:00.434934 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:09:00.434228    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-h4tpf/perf-node-gather-daemonset-j8bck"
Oct 30 08:09:00.435121 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:09:00.434945    6224 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-gwz2f_c84bb85f-b5d5-4934-b67f-0589d63fc861/manager/0.log"
Oct 30 08:09:00.485011 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:09:00.484949    6224 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-h4tpf/perf-node-gather-daemonset-j8bck" podStartSLOduration=1.4849313290000001 podStartE2EDuration="1.484931329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 08:08:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:09:00.483580123 +0000 UTC m=+4299.092987110" watchObservedRunningTime="2026-10-30 08:09:00.484931329 +0000 UTC m=+4299.094338300"
Oct 30 08:09:00.589188 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:09:00.589122    6224 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-zbc8w_e44c3a31-a231-4a77-903b-ad4afbecaae0/kuadrant-console-plugin/0.log"
Oct 30 08:09:01.102557 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:09:01.102521    6224 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-44xkr_0f8fa62e-5595-4c17-b186-f604f06b17ed/manager/0.log"
Oct 30 08:09:01.857758 ip-10-0-128-125 kubenswrapper[6224]: E1030 08:09:01.857713    6224 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltdxc" containerName="extractor"
Oct 30 08:09:04.967512 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:09:04.967473    6224 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-44xkr_0f8fa62e-5595-4c17-b186-f604f06b17ed/manager/0.log"
Oct 30 08:09:06.457864 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:09:06.457834    6224 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-h4tpf/perf-node-gather-daemonset-j8bck"
Oct 30 08:09:07.514965 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:09:07.514934    6224 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-9m92k_e1113d78-e48d-48b2-aab4-c3f859f69eae/limitador/0.log"
Oct 30 08:09:07.691695 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:09:07.691656    6224 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-8stfv_1270d903-7234-4ffb-825c-b4c71a4e9b86/manager/0.log"
Oct 30 08:09:12.704954 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:09:12.704920    6224 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-769d4f7fd-5sbjs_c00279a4-113f-432d-8531-baa80c405f3a/metrics-server/0.log"
Oct 30 08:09:13.003583 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:09:13.003494    6224 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-44xkr_0f8fa62e-5595-4c17-b186-f604f06b17ed/manager/0.log"
Oct 30 08:09:13.006544 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:09:13.006514    6224 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n7zk9_f7666c14-ff3f-44a5-9cce-21ff2dfda0e2/node-exporter/0.log"
Oct 30 08:09:13.138079 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:09:13.138048    6224 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n7zk9_f7666c14-ff3f-44a5-9cce-21ff2dfda0e2/kube-rbac-proxy/0.log"
Oct 30 08:09:13.280428 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:09:13.280355    6224 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n7zk9_f7666c14-ff3f-44a5-9cce-21ff2dfda0e2/init-textfile/0.log"
Oct 30 08:09:17.160290 ip-10-0-128-125 kubenswrapper[6224]: I1030 08:09:17.160248    6224 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-44xkr_0f8fa62e-5595-4c17-b186-f604f06b17ed/manager/0.log"